The basics of NLP on Botnation
To properly configure your AI, you need to understand how a chatbot analyzes what users are typing. We based our NLP on the detection of isolated or associated keywords.
Definition of an Intent:
In the context of Artificial Intelligence, intent refers to the goal that the user has in mind when entering text.
If the user enters “I am looking to own a villa” his intent is “to buy a house”.
Definition of a Keyword:
It is a word that triggers a response from the chatbot. You need to select words that will be most frequently used by users to express an intention. Think about synonyms and sometimes conjugations.
EX: “to buy”, “purchase”, “to acquire
Definition of an Expression:
It is a combination of keywords that generally distinguish two similar intentions. When an Expression is used in an AI rule, the rule is only triggered if all the keywords in the expression are present. However, the order in which these words were entered by the user or if they are separated by other words does not influence the triggering of the rule.
The Expression “buy house” triggers a response even if the user entered “I am looking for a house to buy.”
Definition of an AI rule:
It is a set of keywords and Expressions corresponding to an Intent that triggers a response from the chatbot.
The chatbot does not understand a sentence in its entirety but it will recognize some words. It is by comparing these words with those configured in the rules of its AI that it will be able to associate the right answer.
The selection of the right Keywords and Expressions is therefore essential for a successful AI.
You need to look for the differentiating elements between rules and translate the intentions you want your chatbot to understand into relevant keywords.
Let’s imagine a chatbot specialized in real estate sales(and only sales).
We want the chatbot to react to the desire to buy a house, which we differentiate from the desire to buy an apartment. There are two intentions: the type of transaction (purchase) and the type of property (house).
Since it’s a sales-focused chatbot, we don’t need to detect the type of transaction. It is the type of property that is the differentiator.
Therefore, it is enough to make rules on synonyms of house and apartment.
“Buying House” Rule:
“Buying Apartment” Rule
Always type in the key words with the correct spelling. Botnation’s algorithms will handle typos, accents or plurals. However, when the user makes too many mistakes in the same word, the chatbot may not be able to understand and will return to the Default Response Sequence. The misunderstood words will then appear in the recommendation algorithm. Only these errors should be added to your AI rules. We will deal with this point later.
the keyword “purchase” will be recognized even when it is written:
But it will not be understood if it is written:
Don’t worry about case either, whether your keywords and expressions are in upper or lower case the algorithm will treat them the same.
Use an Expression only if one of the words is common to another rule.
Note that the AI triggers a rule only if all the words of an Expression are contained in the user’s sentence. So the more words in the Expression, the less likely the user is to use that combination.
Expressions of more than 2 words should be limited to strict necessity.
Do not include pronouns, conjunctions, prepositions, definite articles or contractions: the, the, his, the, I, you, he, she, to, from, and, or, nor, or, etc.
Now let’s imagine that our real estate chatbot also wants to offer property sales. The chatbot must react to the intention to buy or sell a house or an apartment. That’s four possibilities, so we need to create four AI rules.
And when you think about it in terms of keywords, you realize that there are going to be words common to several rules. You can’t use isolated keywords, you have to make associations to differentiate the intentions.
First, we need to imagine the different ways in which users might express purchase intent:
- I want to buy …
- I am considering the purchase of ….
- I would like to be the owner of …
And for the sales intention we have :
- I want to sell …
- I am considering the sale of …
- I sell…
When we isolate the differentiating elements we get :
- for purchase: buy, purchase, owner
- for sale: sell, sell, sell.
For the type of property we have :
- synonyms for house: villa, residence
- synonyms for apartment: studio.
This gives the following rules:
“Buying House” Rule
“Buying Apartment” Rule
“Selling House” Rule
“Selling Apartment” Rule
If in the same rule you have an isolated keyword that is found in an Expression, this Expression is useless because the rule is triggered anyway as soon as this keyword appears in the sentence.
If we have the following rule:
“Bank Account Problems” Rule
Since we have “bank” as an isolated keyword, the rule is triggered when the user enters any phrase containing this word. This includes sentences containing ” bankproblem”, ” bankaccount” and ” bankforbidden” and therefore makes these Expressions useless.
So in the end we have:
“Bank Account Problems” Rule
Between two conflicting rules, the AI will choose the one that has been used the most statistically and not the one that is potentially the most relevant (see the paragraph on Machine Learning).
AI optimization also involves using the fewest possible rules with the fewest possible Keywords and Expressions. The less there is, the more responsive the chatbot will be to find the right answer.