ai-natural-language-processing

AI Natural Language Processing (NLP)

In this post we are going to explore Natural language processing technology in AI.

what is Natural Language Processing (NLP)?

ai-natural-language-processing-1.png

 

 

Specifically, you can use NLP to:

  • Classify documents. For instance, you can label documents as sensitive or spam.
  • Do subsequent processing or searches. You can use NLP output for these purposes.
  • Summarize text by identifying the entities that are present in the document.
  • Tag documents with keywords. For the keywords, NLP can use identified entities.
  • Do content-based search and retrieval. Tagging makes this functionality possible.
  • Summarize a document’s important topics. NLP can combine identified entities into topics.
  • Categorize documents for navigation. For this purpose, NLP uses detected topics.
  • Enumerate related documents based on a selected topic. For this purpose, NLP uses detected topics.
  • Score text for sentiment. By using this functionality, you can assess the positive or negative tone of a document.

Potential use cases

Business scenarios that can benefit from custom NLP include:

  • Document intelligence for handwritten or machine-created documents in finance, healthcare, retail, government, and other sectors.
  • Industry-agnostic NLP tasks for text processing, such as name entity recognition (NER), classification, summarization, and relation extraction. These tasks automate the process of retrieving, identifying, and analyzing document information like text and unstructured data. Examples of these tasks include risk stratification models, ontology classification, and retail summarizations.
  • Information retrieval and knowledge graph creation for semantic search. This functionality makes it possible to create medical knowledge graphs that support drug discovery and clinical trials.
  • Text translation for conversational AI systems in customer-facing applications across retail, finance, travel, and other industries.

Apache Spark as a customized NLP framework

Apache Spark is a parallel processing framework that supports in-memory processing to boost the performance of big-data analytic applications. Azure Synapse Analytics, Azure HDInsight, and Azure Databricks offer access to Spark and take advantage of its processing power.

For customized NLP workloads, Spark NLP serves as an efficient framework for processing a large amount of text. This open-source NLP library provides Python, Java, and Scala libraries that offer the full functionality of traditional NLP libraries such as spaCy, NLTK, Stanford CoreNLP, and Open NLP. Spark NLP also offers functionality such as spell checking, sentiment analysis, and document classification. Spark NLP improves on previous efforts by providing state-of-the-art accuracy, speed, and scalability.

For more information look to the whole article Microsoft learning Natural language processing technology

What is conversation AI

ai-natural-language-processing-2.png

 

Natural Language Processing and conversational AI in Azure

Azure AI Language is a managed service for developing natural language processing applications. Identify key terms and phrases, analyze sentiment, summarize text, and build conversational interfaces. Use Language to annotate, train, evaluate, and deploy customizable AI models with minimal machine-learning expertise.

ai-natural-language-processing-3.png

 

Building natural language solutions in Azure

Text Analyzing

ai-natural-language-processing-4.png

 

Speech recognition and synthesis

ai-natural-language-processing-5.png

 

Translation

ai-natural-language-processing-6.png

Conversational Language Undrestadning

ai-natural-language-processing-7.png

Question and Answering

 

Azure Bot service

ai-natural-language-processing-9.png

Analyze text with the Language Service

In this part we create language Service namely  Analyze text in azure Portal and run it in the Azure Cloud Shell

Create an Azure AI services resource in your Azure subscription as following:

  1. Go to Azure portal at https://portal.azure.com, signing in with your Microsoft account.
  2. Search Azure AI services then all AI Services is listed in the left side under All Azure AI Service
  3. Select Language  and press to + Create button to create Service
  4. Then you see Select additional Services and a number of default services are selected automatically as bellow:
  5. Press to Continue to create you Resources and fill in  Resource group, location, name and pricing tier (you should take a pricing group which is not free) then press to the Review + Create button.
  6. Press to the Create button to create the Service, takes a while, when it created press to the Resource group
  7. Then you have the following View:
  8. As you see you have EndPoint and Manage keys by pressing it you can see the kays
  9. Press to the Cloud Shell on the Menu and press to Create Storage then you have the following:

10. change Bash to Power shell by pressing to the Bash dropdown

11. In the command shell, enter the following command to download the sample application and save it to a folder called ai-900. with following command:

git clone https://github.com/MicrosoftLearning/AI-900-AIFundamentals ai-900

12. when The files are downloaded to a folder named ai-900. Now we want to see all of the files in your Cloud Shell storage and work with them. Type the following command into the shell:

code .

13. In the Files pane on the left, expand ai-900 and select analyze-text.ps1. This file contains some code that uses the Language service:

14. From the Azure Portal press to manage keys link and copy the key1 and paste to the Key value in the file as shown in  the above figure and copy the Endpoint from Portal and paste to the Endpoint  in the file above.

15. right click and press save and right click again and press to Quit.

16. In the Cloud Shell, enter the following command to run the code:

cd ai-900
 ./analyze-text.ps1 review1.txt

17. Then it can give you following  output about the picture:

PS /home/mehrdad/ai-900> ./analyze-text.ps1 review1.txt
***Detecting Language***                                                                                                
  - Language: English
  - Code: en
  - Score: 0.99


***Finding Key Phrases***
  - Key Phrases: 
     The Royal Hotel
     Good Hotel
     good service
     great location
     Buckingham Palace
     Westminster Abbey
     same group
     West coast
     Michelin Star
     taster menu
     enormous bathroom
     Clean rooms
     staff
     London
     UK
     stay
     courtyard
     restaurant
     part
     plenty
     fish
     kitchen
     lounge
     bedroom


***Analyzing Sentiment***
  - A positive sentiment based on these scores:
    - Positive: 0.92
    - Neutral: 0.08
    - Negative: 0.01


***Identifying known entities***
  - GOOD Music : https://en.wikipedia.org/wiki/GOOD_Music
  - Hotel : https://en.wikipedia.org/wiki/Hotel
  - The Royal Hotel : https://en.wikipedia.org/wiki/The_Royal_Hotel
  - London : https://en.wikipedia.org/wiki/London
  - Buckingham Palace : https://en.wikipedia.org/wiki/Buckingham_Palace
  - Westminster Abbey : https://en.wikipedia.org/wiki/Westminster_Abbey
  - India : https://en.wikipedia.org/wiki/India
  - West Coast Main Line : https://en.wikipedia.org/wiki/West_Coast_Main_Line
  - Michelin Guide : https://en.wikipedia.org/wiki/Michelin_Guide

18. Run another command:

./analyze-text.ps1 review2.txt

19. Output shall be:

***Detecting Language***                                                                                                
  - Language: English
  - Code: en
  - Score: 0.98


***Finding Key Phrases***
  - Key Phrases: 
     The Royal Hotel
     Tired hotel
     old hotel
     poor service
     United Kingdom
     room furnishings
     office rooms
     flight home
     British Museum
     London
     changing
     internet
     website
     1950


***Analyzing Sentiment***
  - A negative sentiment based on these scores:
    - Positive: 0
    - Neutral: 0.02
    - Negative: 0.97


***Identifying known entities***
  - The Royal Hotel : https://en.wikipedia.org/wiki/The_Royal_Hotel
  - London : https://en.wikipedia.org/wiki/London
  - British Museum : https://en.wikipedia.org/wiki/British_Museum
PS /home/mehrdad/ai-900>

Conclusion

In this post we have learned AI Natural Language Processing (NLP) and  in Azure Portal Created Language Service and tested it with Analyze-text which is a Azer AI Service in the  Cloud Shell to analyze Images.

In Next post I shall explore OPEN AI which is a developer platform for AI.

This post is part of  AI (Artificial Intelligence) step by step

Back to home page