Transferring information across language understanding model domains

Patent metadata

Field	Value
Publication number	US-10191999-B2
Application number	US-201414266253-A
Country	US
Kind code	B2
Filing date	Apr 30, 2014
Priority date	Apr 30, 2014
Publication date	Jan 29, 2019
Grant date	Jan 29, 2019

Abstract

Aspects of the present invention provide a technique to validate the transfer of intents or entities between existing natural language model domains (hereafter “domain” or “NLU”) using click logs, a knowledge graph, or both. At least two different types of transfers are possible. Intents from a first domain may be transferred to a second domain. Alternatively or additionally, entities from the second domain may be transferred to an existing intent in the first domain. Either way, additional intent/entity pairs can be generated and validated. Before the new intent/entity pair is added to a domain, aspects of the present invention validate that the intent or entity is transferable between domains. Validation techniques that are consistent with aspects of the invention can use a knowledge graph, search query click logs, or both to validate a transfer of intents or entities from one domain to another.

First claim

The invention claimed is:

1. A computer-implemented method of expanding slot coverage for a domain-specific natural language understanding (“NLU”) system, the method comprising:
accessing a plurality of queries from labeled training data for a classifier used to recognize an intent within the domain-specific NLU, wherein the intent is associated with a slot of the domain-specific NLU;
identifying a plurality of entities that occur in the slot of the domain-specific NLU within the plurality of queries;
extracting, from a knowledge graph, a graph type for each of the plurality of entities to generate a plurality of candidate graph types for the slot of the domain-specific NLU, the candidate graph types comprising at least one compatible entity that is eligible for pairing with the recognized intent of the domain-specific NLU;
calculating a correlation score for each graph type in the plurality of candidate graph types for pairing the at least one compatible entity with the recognized intent of the domain-specific NLU;
assigning an individual graph type having the highest correlation score as the graph type the domain-specific NLU slot can accept;
expanding slot coverage for the domain-specific NLU by validating the pairing of the at least one compatible entity associated with the assigned individual graph type with the recognized intent of the domain-specific NLU; and
utilizing the validated pairing of the compatible entity and the recognized intent to interpret a natural language input of a query.

2. The method of claim 1 , wherein the method further comprises associating a new graph type with the domain-specific NLU slot upon determining the new graph type is a child within the knowledge graph of the individual graph type.

3. The method of claim 1 , wherein the method further comprises associating a new graph type with the domain-specific NLU slot upon determining the new graph type has an attribute within the knowledge graph of the individual graph type.

4. The method of claim 1 , wherein the correlation score is calculated using a Term Frequency/ Inverse document frequency (“TF/IDF”) equation that uses graph types as terms and slots as documents.

5. The method of claim 4 , wherein IDF is calculated to avoid down weighting a graph type that appears with many similar slots by determining a similarity between slots.

6. The method of claim 4 , wherein TF is calculated using a function that gives a weight that is inversely proportional to a number of graph types appearing with each entity in the knowledge graph.

7. The method of claim 1 , wherein the correlation score is calculated using a weighted summation.

8. The method of claim 1 , wherein the method further comprises transferring an entity list from a second domain-specific natural language understanding (“NLU”) system upon determining entities in the entity list are associated with a graph type that is compatible with the individual graph type.

9. One or more computer-storage media having computer-executable instructions embodied thereon that when executed by a computing device perform the method of expanding slot coverage for a domain-specific natural language understanding (“NLU”) system, the method comprising:
accessing a plurality of queries from labeled training data for a classifier used to recognize an intent within the domain-specific NLU, wherein the intent is associated with a slot of the domain-specific NLU;
identifying a plurality of entities that occur in the slot of the domain-specific NLU within the plurality of queries;
extracting, from a knowledge graph, a graph type for each of the plurality of entities to generate a plurality of candidate graph types for the slot of the domain-specific NLU, the candidate graph types comprising at least one compatible entity that is eligible for pairing with the recognized intent of the domain-specific NLU;
calculating a correlation score for each graph type in the plurality of candidate graph types for pairing the at least one compatible entity with the recognized intent of the domain-specific NLU;
assigning an individual graph type having the highest correlation score as the graph type the domain-specific NLU slot can accept;
expanding slot coverage for the domain-specific NLU by validating the pairing of the at least one compatible entity associated with the assigned individual graph type with the recognized intent of the domain-specific NLU; and
utilizing the validated pairing of the compatible entity and the recognized intent to interpret a natural language input of a query.

10. The method of claim 9 , the method further comprising associating a new graph type with the domain-specific NLU slot upon determining the new graph type is a child within the knowledge graph of the individual graph type.

11. The method of claim 9 , wherein the method further comprises associating a new graph type with the domain-specific NLU slot upon determining the new graph type has an attribute within the knowledge graph of the individual graph type.

12. The method of claim 9 , wherein the correlation score is calculated using a Term Frequency/Inverse document frequency (“TF/IDF”) equation that uses graph types as terms and slots as documents.

13. The method of claim 12 , wherein IDF is calculated to avoid down weighting a graph type that appears with many similar slots by determining a similarity between slots.

14. The method of claim 12 , wherein TF is calculated using a function that gives a weight that is inversely proportional to a number of graph types appearing with each entity in the knowledge graph.

15. The method of claim 9 , wherein the correlation score is calculated using a weighted summation.

16. The method of claim 9 , wherein the method further comprises transferring an entity list from a second domain-specific natural language understanding (“NLU”) system upon determining entities in the entity list are associated with a graph type that is compatible with the individual graph type.

17. A computer-implemented method of expanding slot coverage for a domain-specific natural language understanding (“NLU”) system, the method comprising:
accessing a plurality of queries from training data to recognize an intent and a plurality of domain-specific entities, wherein the recognized intent and the plurality of domain-specific entities are associated with a slot of the domain-specific NLU;
generating a plurality of candidate graph types for the slot of the domain-specific NLU by extracting, from a knowledge graph, a plurality of candidate graph types associated with each of the plurality of domain-specific entities, the candidate graph types comprising at least one compatible entity that is eligible for pairing with the recognized intent of the domain-specific NLU;
assigning at least one of the plurality of candidate graph types as an individual graph type that the domain-specific NLU can accept based on a correlation score for each of the plurality of candidate graph types associated with the domain-specific entity, the individual graph type having the highest correlation score;
expanding slot coverage of the domain-specific NLU by validating the pairing of the at least one compatible entity associated with the assigned graph type with the recognized intent of the domain-specific NLU; and
utilizing the validated pairing of the compatible entity and the recognized intent to interpret a natural language input of a query.

18. The method of claim 17 , wherein the method further comprises associating a new graph type with the domain-specific NLU slot upon de

Assignees

• Microsoft Corp
• Microsoft Technology Licensing Llc

Inventors

• Liu Xiaohu
• Elkahky Ali Mamdouh
• Sarikaya Ruhi
• Tur Gokhan
• Hakkani-Tur Dilek
• Heck Larry Paul

Classifications

• G06F40/30

  Semantic analysis · CPC title

• G06F40/40

  Processing or translation of natural language (natural language analysis G06F40/20; semantic analysis G06F40/30) · CPC title

• G10L15/1822

  Parsing for meaning understanding · CPC title

• G06F16/90332 Primary

  Natural language query formulation or dialogue systems · CPC title

• G10L15/063

  Training · CPC title