Weighting dictionary entities for language understanding models

Patent metadata

Field	Value
Publication number	US-9519870-B2
Application number	US-201414207986-A
Country	US
Kind code	B2
Filing date	Mar 13, 2014
Priority date	Mar 13, 2014
Publication date	Dec 13, 2016
Grant date	Dec 13, 2016

Abstract

A dictionary used by a spoken language understanding (SLU) system is improved by providing weightings for entities in the dictionary that represent the likelihood each entity belongs to an entity class represented by the dictionary. A classifier model may be trained using a seed list containing sample entities that belong in the entity class and a background entity list containing samples that do not belong in the entity class. Clicked URLs from search logs, search result URLs, and attributes from an entity graph may be used as features of the sample entities to train the classifier model. The classifier model may be used to weight entities from a candidate dictionary. The entity weightings are used to generate an improved dictionary for use in the SLU system.

First claim

What is claimed is:

1. One or more computer storage media storing computer-executable instructions that, when executed by one or more computing devices, cause the one or more computing devices to perform operations comprising:
accessing a seed list containing positive sample entities that belong to an entity class;
accessing a background entity list containing negative sample entities that do not belong to the entity class;
identifying clicked URLs from search click logs for at least a portion of the positive sample entities and negative sample entities;
identifying search result URLs for at least a portion of the positive sample entities and negative sample entities;
identifying attributes from an entity graph for at least a portion of the positive sample entities and negative sample entities;
training a classifier model using the clicked URLs, search result URLs, and attributes from the entity graph as features of the positive sample entities and negative sample entities; and
using the classifier model to weight entities in a candidate dictionary to provide weightings for the entities from the candidate dictionary.

2. The one or more computer storage media of claim 1 , wherein the seed list is generated based on information from at least one selected from the following: an existing entity graph and training data from a spoken language understanding system.

3. The one or more computer storage media of claim 1 , wherein the background entity list is generated based on information from at least one selected from the following: an existing entity graph and training data from a spoken language understanding system.

4. The one or more computer storage media of claim 1 , wherein URL search results are obtained for positive sample entities and negative sample entities for which no clicked URLs are available from the search click logs.

5. The one or more computer storage media of claim 1 , wherein the classifier model is trained using at least one selected from the following: logistic regression and support vector machines.

6. The one or more computer storage media of claim 1 , wherein the operations further comprise:
using the weightings to generate an improved dictionary that does not include a subset of entities that do not satisfy a weighting threshold; and
using the improved dictionary in a spoken language understanding system to process a user input.

7. The one or more computer storage media of claim 1 , wherein the operations further comprise:
creating a weighted dictionary using the weightings; and
employing the weighted dictionary in a spoken language understanding system to process a user input.

8. The one or more computer storage media of claim 1 , wherein the operations further comprise:
clustering entities based on the weightings; and
generating a set of clustered dictionaries, each clustered dictionary including a cluster of entities.

9. The one or more computer storage media of claim 8 , wherein the operations further comprise:
employing the set of clustered dictionaries in a spoken language understanding system to process a user input.

10. A computer-implemented method comprising:
accessing positive sample entities that belong to an entity class and negative sample entities that do not belong to the entity class;
identifying clicked URLs from search click logs for at least a portion of the positive sample entities and negative sample entities;
identifying search result URLs for at least a portion of the positive sample entities and negative sample entities;
identifying attributes from an entity graph for at least a portion of the positive sample entities and negative sample entities;
training, using a computing device, a classifier model using the clicked URLs, search result URLs, and attributes from the entity graph as features of the positive sample entities and negative sample entities; and
employing the classifier model to weight entities in a candidate dictionary to provide weightings for the entities from the candidate dictionary.

11. The method of claim 10 , wherein the positive sample entities and the negative sample entities are identified based on information from at least one selected from the following: an existing entity graph and training data from a spoken language understanding system.

12. The method of claim 10 , wherein URL search results are obtained for positive sample entities and negative sample entities for which no clicked URLs are available from the search click logs.

13. The method of claim 10 , wherein the method further comprises:
employing the weightings to generate at least one improved dictionary for use in a spoken language understanding system to process a user input.

14. The method of claim 10 , wherein the method further comprises:
clustering entities based on weighting; and
generating a set of clustered dictionaries, each clustered dictionary including a cluster of entities.

15. The method of claim 14 , wherein the method further comprises:
employing the set of clustered dictionaries in a spoken language understanding system to process a user input.

16. A computerized system comprising:
one or more processors; and
a plurality of components that include computer-executable instructions that are executed by the one or more processors, the components including:
a model building component that trains a classifier model for an entity class using positive sample entities that belong to the entity class and negative sample entities that do not belong to the entity class, the model building component also using clicked URLs, search result URLs, and attributes from an entity graph as features of the positive sample entities and negative sample entities to train the classifier model; and
a weighting component that employs the classifier model to weight entities in a candidate dictionary to provide weightings for the entities from the candidate dictionary.

17. The system of claim 16 , wherein the positive sample entities are from a seed list and the negative sample entities are from a background entity list, and wherein the seed list and background entity list are generated based on information from at least one selected from the following: an existing entity graph and training data from a spoken language understanding system.

18. The system of claim 16 , wherein the weightings for the entities from the candidate dictionary are used to provide at least one improved dictionary.

19. The computerized system of claim 16 , wherein the components further comprise:
a clustering component that clusters entities based on the weightings and generates a set of clustered dictionaries, each clustered dictionary including a cluster of entities.

Assignees

• Microsoft Corp
• Microsoft Technology Licensing Llc

Inventors

• Sarikaya Ruhi
• Liu Xiaohu

Classifications

• G06N99/005 Primary

  Physics · mapped topic

• G06N20/00 Primary

  Machine learning · CPC title

• G06F16/313 Primary

  Selection or weighting of terms for indexing · CPC title