Interactive segment extraction in computer-human interactive learning

Patent metadata

Field	Value
Publication number	US-9489373-B2
Application number	US-201314075713-A
Country	US
Kind code	B2
Filing date	Nov 8, 2013
Priority date	Jul 12, 2013
Publication date	Nov 8, 2016
Grant date	Nov 8, 2016

Abstract

A collection of data that is extremely large can be difficult to search and/or analyze. Relevance may be dramatically improved by automatically classifying queries and web pages in useful categories, and using these classification scores as relevance features. A thorough approach may require building a large number of classifiers, corresponding to the various types of information, activities, and products. Creation of classifiers and schematizers is provided on large data sets. Exercising the classifiers and schematizers on hundreds of millions of items may expose value that is inherent to the data by adding usable meta-data. Some aspects include active labeling exploration, automatic regularization and cold start, scaling with the number of items and the number of classifiers, active featuring, and segmentation and schematization.

First claim

The invention claimed is:

1. One or more hardware computer-readable media having embodied thereon computer-usable instructions that, when executed, facilitate a method of segment extraction by a user for a machine learning system, the method comprising:
storing a set of data items, wherein each data item includes a plurality of tokens;
providing a segment extractor that is trainable to identify a segment within a data item as an example of a concept, wherein the segment includes a group of tokens;
presenting on a user interface a concept hierarchy that represents the concept, wherein the concept hierarchy depicts a root node that corresponds to the concept and one or more child nodes that correspond to hierarchical sub-concepts that are constituent parts of the concept, wherein the child nodes depict respective labels that identify sub-concepts that correspond to the child nodes, wherein one or more of the child nodes are user-selectable for labeling tokens within the data item, and wherein selection of a child node within the concept hierarchy identifies the respective label that is utilized for labeling a token within the data item;
receiving a user selection of a child node that corresponds to a selected sub-concept in the concept hierarchy;
utilizing the segment extractor to select from the plurality of data items a first data item that is predicted to include an example of the concept associated with the concept hierarchy, wherein the example is represented by one or more of the tokens in the first data item;
displaying the first data item, wherein displaying the first data item includes presenting a first set of one or more pre-labels that identify a first set of one or more tokens as predicted positive examples of the selected sub-concept;
receiving a user selection of a first token in the displayed second data item that labels the first token as a positive or negative example of the selected sub-concept;
replacing the first set of one or more pre-labels with a second set of one or more pre-labels that identify a second set of one or more tokens as predicted positive examples of the selected sub-concept; and
based at least on the labeling of the first token as an example of the selected sub-concept, training the segment extractor.

2. The media of claim 1 , the method further comprising:
displaying a second data item from the plurality of data items, wherein the second data item is selected by means of a user-provided search query;
receiving a user selection of a second token in the displayed second data item that labels the second token as an example of the selected sub-concept; and
based at least on the labeling of the second token as an example of the selected sub-concept, training the segment extractor.

3. The media of claim 1 , wherein the segment extractor is trainable to identify tokens within the segment as examples of sub-concepts that correspond to constituent parts of the concept.

4. The media of claim 1 , wherein the data items are documents, and wherein the tokens are words.

5. The media of claim 1 , the method further comprising dividing at least one of the displayed first data item or the displayed second data item into sections and indicating a section that includes the example of the concept.

6. The media of claim 5 , wherein at least one of the selected first token or the selected second token is within the indicated section, and wherein tokens outside of the indicated section are not utilized for training the segment extractor.

7. The media of claim 6 , wherein the indicated section is resizable by the user.

8. The media of claim 7 , wherein when the user selects a token that is outside of the indicated section, the indicated section is resized to include the selected token.

9. The media of claim 1 , wherein the second set of one or more pre-labels is identified based on a constraint determined from the user input that labeled the one or more tokens as positive or negative examples of the selected sub-concept, and wherein replacing the first set of one or more pre-labels with the second set of one or more pre-labels occurs prior to said training the segment extractor based on the labeling of the second token.

10. A system for segment extraction by a user for a machine learning environment, the method comprising:
one or more processing devices configured to:
store a set of data items, wherein each data item includes a plurality of tokens;
operate as a segment extractor that is trainable to identify a segment within a data item as an example of a concept, wherein the segment includes a group of tokens;
present on a user interface a concept hierarchy that represents the concept, wherein the concept hierarchy depicts a root node that corresponds to the concept and one or more child nodes that correspond to hierarchical sub-concepts that are constituent parts of the concept, wherein the child nodes depict respective labels that identify sub-concepts that correspond to the child nodes, wherein one or more of the child nodes are user-selectable for labeling tokens within the data item, wherein selection of a child node within the concept hierarchy identifies the respective label that is utilized for labeling a token within the data item;
receive a user selection of a child node that corresponds to a selected sub-concept in the concept hierarchy;
utilize the segment extractor to select from the plurality of data items a first data item that is predicted to include an example of the concept associated with the concept hierarchy, wherein the example is represented by one or more of the tokens in the first data item;
display the first data item, wherein displaying the first data item includes presenting a first set of one or more pre-labels that identify a first set of one or more tokens as predicted positive examples of the selected sub-concept;
receive a user selection of a first token in the displayed second data item that labels the first token as a positive or negative example of the selected sub-concept;
replacing the first set of one or more pre-labels with a second set of one or more pre-labels that identify a second set of one or more tokens as predicted positive examples of the selected sub-concept; and
based at least on the labeling of the first token as an example of the selected sub-concept, train the segment extractor.

11. The system of claim 10 , wherein the one or more processing devices are further configured to:
display a second data item from the plurality of data items, wherein the second data item is selected by means of a user-provided search query;
receive a user selection of a second token in the displayed second data item that labels the second token as an example of the selected sub-concept; and
based at least on the labeling of the second token as an example of the selected sub-concept, training the segment extractor.

12. The system of claim 10 , wherein the segment extractor is trainable to identify tokens within the segment as examples of sub-concepts that correspond to constituent parts of the concept.

13. The system of claim 10 , wherein the data items are documents, and wherein the tokens are words.

14. The system of claim 10 , wherein the one or more processing devices are further configured to divide at least one of the displayed first data item or the displayed second data item into sections and indicate a section that includes the example of the concept.

15. The system of claim 14 , wherein at least one of the selected first token or the selected second token is within the indicated section, and wherein tokens outside of the indicated section ar

Assignees

• Microsoft Corp
• Microsoft Technology Licensing Llc

Inventors

• Simard Patrice Y
• Chickering David Max
• Grangier David G
• Charles Denis X
• Bottou Leon
• Amershi Saleema A
• Lakshmiratan Aparna
• Suarez Carlos Garcia Jurado

Classifications

• G06N7/01

  Probabilistic graphical models, e.g. probabilistic networks · CPC title

• G06F40/30

  Semantic analysis · CPC title

• G06F16/951

  Indexing; Web crawling techniques · CPC title

• H04L1/0072

  Error control for data other than payload data, e.g. control data · CPC title

• G06N20/00 Primary

  Machine learning · CPC title