Method and system for automated intent mining, classification and disposition

Patent metadata

Field	Value
Publication number	US-11507750-B2
Application number	US-202016931007-A
Country	US
Kind code	B2
Filing date	Jul 16, 2020
Priority date	Mar 23, 2018
Publication date	Nov 22, 2022
Grant date	Nov 22, 2022

Abstract

An agent automation system includes a memory configured to store a corpus of utterances and a semantic mining framework and a processor configured to execute instructions of the semantic mining framework to cause the agent automation system to perform actions, wherein the actions include: detecting intents within the corpus of utterances; producing intent vectors for the intents within the corpus; calculating distances between the intent vectors; generating meaning clusters of intent vectors based on the distances; detecting stable ranges of cluster radius values for the meaning clusters; and generating an intent/entity model from the meaning clusters and the stable ranges of cluster radius values, wherein the agent automation system is configured to use the intent/entity model to classify intents in received natural language requests.

First claim

What is claimed is:

1. An agent automation system, comprising:
a memory configured to store a corpus of utterances and a semantic mining framework; and
at least a processor configured to execute instructions of the semantic mining framework to cause the agent automation system to perform actions comprising:
detecting intents within the corpus of utterances;
determining intent vectors for the intents of the corpus;
calculating distances between the intent vectors in a vector space;
detecting stable cluster radii based on the distances between the intent vectors in the vector space;
generating a cluster formation tree, wherein each level of the cluster formation tree includes a respective clustering of the intent vectors using one of the stable cluster radii; and
clustering the intent vectors into meaning clusters having a particular stable cluster radius.

2. The system of claim 1 , wherein the processor is configured to execute the instructions of the semantic mining framework to cause the agent automation system to perform actions comprising:
detecting the stable cluster radii by identifying substantially flat portions of a curve plotting number of meaning clusters as a function of cluster radius.

3. The system of claim 1 , wherein the processor is configured to execute the instructions of the semantic mining framework to cause the agent automation system to perform actions comprising:
performing one or more cluster cleaning steps and/or one or more cluster data augmentation steps on the meaning clusters based on a collection of rules stored in the memory.

4. The system of claim 1 , wherein the processor is configured to execute the instructions of the semantic mining framework to cause the agent automation system to perform actions comprising:
selecting a respective utterance represented by a particular intent vector of each of the meaning clusters as a sample utterance of each of the meaning clusters, wherein the particular intent vector is a highest prevalence intent vector of each of the meaning clusters.

5. The system of claim 4 , wherein the processor is configured to execute the instructions of the semantic mining framework to cause the agent automation system to perform actions comprising:
generating an intent/entity model based on the meaning clusters and the sample utterances, wherein the intent/entity model stores relationships between a representative intent of each of the meaning clusters and the sample utterances.

6. The system of claim 1 , wherein the processor is configured to execute the instructions of the semantic mining framework to cause the agent automation system to perform actions comprising:
presenting the cluster formation tree as a dendrogram on a display device, wherein the dendrogram provides a navigable schema of the respective clustering of the intent vectors at each of the levels of the cluster formation tree.

7. The system of claim 1 , wherein the processor is configured to execute the instructions of the semantic mining framework to cause the agent automation system to perform actions comprising:
receiving user input indicating the particular stable cluster radius and, in response, clustering the intent vectors into the meaning clusters having the particular stable cluster radius.

8. The system of claim 1 , wherein at least one intent vector of the intent vectors is associated with at least one corresponding entity as a parameter of the intent vector.

9. A method, comprising:
detecting intents within a corpus of utterances;
determining intent vectors for the intents of the corpus;
calculating distances between the intent vectors in a vector space;
detecting stable cluster radii based on the distances between the intent vectors in the vector space;
generating and presenting a cluster formation tree, wherein each level of the cluster formation tree includes a respective clustering of the intent vectors using one of the stable cluster radii;
receiving user input indicating a particular stable cluster radius; and
clustering the intent vectors into meaning clusters having the particular stable cluster radius.

10. The method of claim 9 , comprising:
selecting sample utterances from the corpus of utterances for each of the meaning clusters; and
generating an intent/entity model based on the meaning clusters and the sample utterances, wherein the intent/entity model stores relationships between a representative intent of each of the meaning clusters and the sample utterances.

11. The method of claim 10 , wherein selecting the sample utterances comprises:
determining a highest prevalence intent of each of the meaning clusters; and
selecting a respective utterance of the corpus of utterances that is represented by the highest prevalence intent in each of the meaning clusters as a respective sample utterance of each of the meaning clusters.

12. The method of claim 9 , comprising:
performing intent analytics to determine prevalence scores of the meaning clusters; and
identifying blind spots in a stored conversation model based on the prevalence scores of the meaning clusters of intent vectors.

13. The method of claim 9 , wherein detecting the stable cluster radii comprises:
detecting the stable cluster radii using agglomerative clustering, density based clustering, or a combination thereof.

14. A non-transitory, computer-readable medium storing instructions executable by a processor of a computing system, the instructions comprising instructions to:
detect intents within a corpus of utterances;
determine intent vectors for the intents of the corpus;
calculate distances between the intent vectors in a vector space;
detect stable cluster radii based on the distances between the intent vectors in the vector space;
cluster the intent vectors into meaning clusters having a particular stable cluster radius;
selecting sample utterances from the corpus of utterances for each of the meaning clusters; and
generating an intent/entity model based on the meaning clusters and the sample utterances, wherein the intent/entity model stores relationships between a representative intent of each of the meaning clusters and the sample utterances.

15. The medium of claim 14 , wherein the instructions comprise instructions to:
generate and present a cluster formation tree, wherein each level of the cluster formation tree includes a respective clustering of the intent vectors using one of the stable cluster radii; and
receive user input indicating the particular stable cluster radius.

16. The medium of claim 14 , wherein the instructions comprise instructions to:
determining a highest prevalence intent of each of the meaning clusters; and
selecting a respective utterance of the corpus of utterances that is represented by the highest prevalence intent in each of the meaning clusters as a sample utterance of each of the meaning clusters.

17. The medium of claim 14 , wherein the instructions to detect the stable cluster radii comprise instructions to:
determine cluster radius values at which a number of the meaning clusters formed does not substantially increase with increasing cluster radius values.

18. The medium of claim 14 , wherein the instructions comprise instructions to:
augment the intent/entity model by performing a rule-based re-expression of the sample utterances of the intent/entity model and removal of structurally similar sample utterances of the intent/entity model.

19. The medium of claim 18 , wherein the rule-based re-expression comprises an active-to-pas

Assignees

• Servicenow Inc

Inventors

• Sapugay Edwin
• Madamala Anil Kumar
• Naboka Maxim
• Sunkara Srinivas Satyasai
• Santos Lewis Savio Landry
• Subbarao Murali B

Classifications

• G06F2216/03

  Data mining · CPC title

• G06F40/247

  Thesauruses; Synonyms · CPC title

• G06F40/30 Primary

  Semantic analysis · CPC title

• G06N5/04

  Inference or reasoning models · CPC title

• G06N20/00

  Machine learning · CPC title