Interactive concept editing in computer-human interactive learning

Patent metadata

Field	Value
Publication number	US-2024135098-A1
Application number	US-202318527104-A
Country	US
Kind code	A1
Filing date	Dec 1, 2023
Priority date	Jul 12, 2013
Publication date	Apr 25, 2024
Grant date	—

Abstract

A collection of data that is extremely large can be difficult to search and/or analyze. Relevance may be dramatically improved by automatically classifying queries and web pages in useful categories, and using these classification scores as relevance features. A thorough approach may require building a large number of classifiers, corresponding to the various types of information, activities, and products. Creation of classifiers and schematizers is provided on large data sets. Exercising the classifiers and schematizers on hundreds of millions of items may expose value that is inherent to the data by adding usable meta-data. Some aspects include active labeling exploration, automatic regularization and cold start, scaling with the number of items and the number of classifiers, active featuring, and segmentation and schematization.

First claim

1 . One or more hardware computer-storage devices having embodied thereon computer-usable instructions that, when executed by at least one processor, cause performance of operations associated with interactively generating a machine-learning feature, the operations comprising:
causing presentation of a user interface for generating a representation of a first concept by associating, to the first concept, a list of words that are specific examples of the first concept; providing, via the user interface, an input field that receives a first input comprising a plurality of characters of text that form at least one specific example of the first concept; generating a generalized concept determined at least in part from the first input; generating, based on a classifier, a set of words that are each a specific example of the generalized concept; providing, via the user interface, a first field that displays the set of words as suggested words that are selectable for inclusion as specific examples of the first concept; receiving a second input indicative of a selection of at least one suggested word of the suggested words; providing, via the user interface, a second field that displays (1) the list of words that are specific examples of the first concept and (2) the at least one suggested word of the second input; and saving (1) the list of words that are specific examples of the first concept and (2) the at least one suggested word of the second input as an updated representation of the first concept, wherein the classifier is trained based on a first machine learning feature indicating the updated representation that comprises (1) the list of words that are specific examples of the first concept and (2) the at least one suggested word of the second input.

2 . The one or more hardware computer-storage devices of claim 1 , wherein the set of words displayed in the first field as the suggested words are selectable for exclusion as non-examples of the first concept, wherein the classifier is further trained based on a second machine learning feature comprising at least one word of the suggested words selected for exclusion as non-examples of the first concept.

3 . The one or more hardware computer-storage devices of claim 2 , wherein the classifier is further trained based on a third machine learning feature comprising a computer-generated word, wherein the first machine learning feature has a higher weight than the third machine learning feature based on the first machine learning feature comprising (1) the list of words that are specific examples of the first concept and (2) the at least one suggested word of the second input.

4 . The one or more hardware computer-storage devices of claim 1 , the operations further comprising:
receiving a first word that is an example of the first concept; and presenting a list of suggested words that represent the generalized concept generated based at least on the first word.

5 . The one or more hardware computer-storage devices of claim 4 , the operations further comprising:
subsequent to presenting the list of suggested words, receiving a second word that is a second example of the first concept; refining the list of suggested words based at least on a combination of the first word and the second word; and presenting the refined list of suggested words that represent a refined generalized concept.

6 . The one or more hardware computer-storage devices of claim 4 , the operations further comprising:
subsequent to presenting the list of suggested words, receiving a second word that is designated as not being an example of the first concept; refining the list of suggested words based at least on a combination of the first word and the second word; and presenting the refined list of suggested words that represent a refined generalized concept.

7 . The one or more hardware computer-storage devices of claim 4 , the operations further comprising:
including the one or more of the suggested words associated with the second input in the input field that receives the first input that forms at least one specific examples of the first concept.

8 . The one or more hardware computer-storage devices of claim 1 , wherein each word in the list of words is assigned a respective weight that is scaled, based on training data and during generation of the list of words, by a function of frequency of a size of the list of words, wherein the scaled weights are related by a regularization constraint that adjusts the respective weights of the words that have less training data toward a value determined by the words that have more training data.

9 . The one or more hardware computer-storage devices of claim 8 , wherein the classifier comprises a neural network.

10 . A method of interactively generating an updated representation of a first concept for training a classifier, the method comprising:
presenting a user interface for generating a representation of the first concept by associating, to the first concept, a list of phrases that are specific examples of the first concept; providing, via the user interface, an input field that receives a first input comprising a plurality of characters of text that form at least one specific example of the first concept; generating a generalized concept determined at least in part from the first input; generating, based on the classifier, a set of phrases that are each a specific example of the generalized concept; providing, via the user interface, a first field that displays the set of phrases as suggested phrases that are selectable for inclusion as specific examples of the first concept; receiving a second input indicative of a selection of at least one suggested phrase of the suggested phrases; providing, via the user interface, a second field that displays (1) the list of phrases that are specific examples of the first concept and (2) the at least one suggested phrase of the second input; and saving (1) the list of phrases that are specific examples of the first concept and (2) the at least one suggested phrase of the second input as the updated representation of the first concept, wherein the classifier comprises a neural network that is trained based on a first machine learning feature indicating the updated representation that comprises (1) the list of phrases that are specific examples of the first concept and (2) the at least one suggested phrase of the second input.

11 . The method of claim 10 , wherein the set of phrases displayed in the first field as suggested phrases are selectable for exclusion as non-examples of the first concept, wherein the classifier is further trained based on a second machine learning feature comprising at least one phrase of the suggested phrases selected for exclusion as non-examples of the first concept.

12 . The method of claim 11 , wherein the classifier is further trained based on a third machine learning feature comprising a computer-generated phrase, wherein the first machine learning feature has a higher weight than the third machine learning feature based on the first machine learning feature comprising (1) the list of phrases that are specific examples of the first concept and (2) the at least one suggested phrase of the second input.

13 . The method of claim 10 , further comprising:
generating a refinement of the suggested phrases that represents a refinement of the generalized concept based on at least one or more additional input phrases; and presenting the refinement of the set of phrases on the user interface.

14 . The method of claim 13 , further comprising:
repeating steps of generating the refinem

Assignees

• Microsoft Technology Licensing Llc

Inventors

• Simard Patrice Y
• Grangier David G
• Bottou Leon
• Amershi Saleema A

Classifications

• G06F40/242 Primary

  Dictionaries · CPC title

• G06F3/0482

  Interaction with lists of selectable items, e.g. menus · CPC title

• G06F16/951

  Indexing; Web crawling techniques · CPC title

• G06F40/30

  Semantic analysis · CPC title

• G06N7/01

  Probabilistic graphical models, e.g. probabilistic networks · CPC title