Feature completion in computer-human interactive learning

Patent metadata

Field	Value
Publication number	US-9355088-B2
Application number	US-201314075701-A
Country	US
Kind code	B2
Filing date	Nov 8, 2013
Priority date	Jul 12, 2013
Publication date	May 31, 2016
Grant date	May 31, 2016

Abstract

A collection of data that is extremely large can be difficult to search and/or analyze. Relevance may be dramatically improved by automatically classifying queries and web pages in useful categories, and using these classification scores as relevance features. A thorough approach may require building a large number of classifiers, corresponding to the various types of information, activities, and products. Creation of classifiers and schematizers is provided on large data sets. Exercising the classifiers and schematizers on hundreds of millions of items may expose value that is inherent to the data by adding usable meta-data. Some aspects include active labeling exploration, automatic regularization and cold start, scaling with the number of items and the number of classifiers, active featuring, and segmentation and schematization.

First claim

The invention claimed is:

1. One or more hardware computer-storage media having embodied thereon computer-usable instructions that, when executed, facilitate a method of feature completion for machine learning, the method comprising:
storing a first set of data items, wherein each data item includes a text stream of words;
accessing a dictionary, wherein the dictionary includes a list of words that define a concept usable as an input feature for training a machine-learning model to score data items with a probability of being a positive example or a negative example of a particular class of data item;
providing a feature that is already trained to determine a probability that a word at a given word position corresponds semantically to the concept defined by the words in the dictionary; and
training the machine-learning model with the dictionary as an input feature, wherein the training includes
A) for the given word position in a text stream within a data item, utilizing the provided feature to calculate a first probability that the word at the given word position corresponds semantically to the concept defined by the words in the dictionary,
B) examining a context of the given word position, wherein the context includes a number of words preceding the given word position and a number of words following the given word position, and wherein the context does not include the word at the given word position,
C) calculating a second probability that the word at the given word position corresponds semantically to the concept defined by the words in the dictionary, based on a function of the words in the context of the given word position, wherein calculating the second probability comprises one or more of:
1) determining whether any words from a given list appear at a center of a window of text around the given word position in which center words in the window of text have been removed,
2) determining a presence or absence of a verb in the window,
3) determining a presence or absence of a noun followed by an adjective, or
4) determining a number of occurrences of a given word in the window, and

D) modifying the function to adjust the calculated second probability, based on the calculated first probability.

2. The media of claim 1 , wherein modifying the function to adjust the calculated second probability includes
A) modifying the function to increase the calculated second probability when the word at the given word position is in the dictionary, and
B) modifying the function to decrease the calculated second probability when the word at the given word position is not in the dictionary.

3. The media of claim 1 , wherein the machine-learning model includes at least one of a classifier and a schematizer.

4. The media of claim 1 , wherein the context is a sliding window that includes a number of words immediately preceding the given word position and a number of words immediately following the given word position.

5. The media of claim 1 , wherein the calculated first probability is an estimate of the first probability.

6. A method of feature completion for machine learning, comprising:
storing a first set of data items, wherein each data item includes a text stream of words;
accessing a dictionary, wherein the dictionary includes a list of words that define a concept usable as an input feature for training a machine-learning model to score data items with a probability of being a positive example or a negative example of a particular class of data item;
providing a feature that is already trained to determine a probability that a word at a given word position corresponds semantically to the concept defined by the words in the dictionary; and
training the machine-learning model with the dictionary as an input feature, wherein the training includes
A) for the given word position in a text stream within a data item, utilizing the provided feature to calculate a first probability that the word at the given word position corresponds semantically to the concept defined by the words in the dictionary,
B) examining a context of the given word position, wherein the context includes a number of words preceding the given word position and a number of words following the given word position, and wherein the context does not include the word at the given word position,
C) calculating a second probability that the word at the given word position corresponds semantically to the concept defined by the words in the dictionary, based on a function of the words in the context of the given word position, wherein calculating the second probability comprises one or more of:
1) determining whether any words from a given list appear at a center of a window of text around the given word position in which center words in the window of text have been removed,
2) determining a presence or absence of a verb in the window,
3) determining a presence or absence of a noun followed by an adjective, or
4) determining a number of occurrences of a given word in the window, and

D) modifying the function to adjust the calculated second probability, based on the calculated first probability.

7. The method of claim 6 , wherein modifying the function to adjust the calculated second probability includes
A) modifying the function to increase the calculated second probability when the word at the given word position is in the dictionary, and
B) modifying the function to decrease the calculated second probability when the word at the given word position is not in the dictionary.

8. The method of claim 6 , wherein the machine-learning model includes at least one of a classifier and a schematizer.

9. The method of claim 6 , wherein the context is a sliding window that includes a number of words immediately preceding the given word position and a number of words immediately following the given word position.

10. The method of claim 6 , wherein the calculated first probability is an estimate of the first probability.

11. The method of claim 6 , wherein the feature is a regular expression operating over strings to predict semantically matching positions in text within a string at each considered position.

12. A system for feature completion for machine learning, comprising:
one or more computer-storage media configured to store a first set of data items, wherein each data item includes a text stream of words;
one or more computer-storage media configured to store a dictionary; and
one or more computing devices configured to
A) access the dictionary, wherein the dictionary includes a list of words that define a concept usable as an input feature for training a machine-learning model to score data items with a probability of being a positive example or a negative example of a particular class of data item;
B) utilize a feature that is already trained to determine a probability that a word at a given word position corresponds semantically to the concept defined by the words in the dictionary; and
C) train the machine-learning model with the dictionary as an input feature, wherein the training includes
1) for the given word position in a text stream within a data item, utilize the provided feature to calculate a first probability that the word at the given word position corresponds semantically to the concept defined by the words in the dictionary,
2) examine a context of the given word position, wherein the context includes a number of words preceding the given word position and a number of words following the given word position, and wherein the context does not include the word at the given word position,
3) calculate a second probability that the word at the given word position corresponds

Assignees

• Microsoft Corp
• Microsoft Technology Licensing Llc

Inventors

• Simard Patrice Y
• Chickering David Max
• Grangier David G
• Charles Denis X
• Bottou Leon
• Suarez Carlos Garcia Jurado

Classifications

• G06N7/01

  Probabilistic graphical models, e.g. probabilistic networks · CPC title

• G06F40/242 Primary

  Dictionaries · CPC title

• H04L1/0079

  Formats for control data (H04L1/16 takes precedence; training sequences H04L25/00 and H04L27/00) · CPC title

• G06F3/0482

  Interaction with lists of selectable items, e.g. menus · CPC title

• G06F16/951

  Indexing; Web crawling techniques · CPC title