System and method for large-scale multi-label learning using incomplete label assignments

Patent metadata

Field	Value
Publication number	US-2023222386-A1
Application number	US-202318175249-A
Country	US
Kind code	A1
Filing date	Feb 27, 2023
Priority date	Nov 17, 2014
Publication date	Jul 13, 2023
Grant date	—

Abstract

At least one label prediction model is trained, or learned, using training data that may comprise training instances that may be missing one or more labels. The at least one label prediction model may be used in identifying a content item's ground-truth label set comprising an indicator for each label in the label set indicating whether or not the label is applicable to the content item.

First claim

1 . A method comprising:
training, by a computing device, a multi-level label prediction model to predict an applicability, to a digital content item, of each label of a plurality of labels, the multi-level label prediction model comprising a first level and a second level, the first level being trained using training data missing a number of applicable labels, label prediction output of the trained first level including an applicability prediction for each of the number of applicable labels missing from the training data, the second level being trained using the label prediction output generated by the trained first level; generating, by the computing device, the digital content item's label prediction output using the trained multi-level label prediction model, the digital content item's labeling prediction output indicating an applicability, to the digital content item, of each label of the plurality of labels; and automatically generating, by the computing device, an annotation for the digital content item based on the digital content item's labeling prediction output.

2 . The method of claim 1 , comprising:
identifying, via the computing device, a set of digital content items using the digital content item's annotation.

3 . The method of claim 2 , each digital content item in the set of digital content items having a respective annotation generated using label prediction output from the trained multi-level label prediction model.

4 . The method of claim 2 , identifying the set of digital content items is responsive to a received search request.

5 . The method of claim 4 , the received search request comprising information indicating the digital content item.

6 . The method of claim 1 , the training data comprising a plurality of training instances corresponding to a plurality of digital content items, labeling data corresponding to at least one training instance is missing at least one applicable label from the plurality of labels.

7 . The method of claim 6 , a training instance, of the plurality of training instances, comprising associated labeling data and feature data.

8 . The method of claim 7 , further comprising:
analyzing, a digital content item of the plurality of digital content items, and based on the analysis, generating the feature data for the analyzed digital content item.

9 . The method of claim 1 , the number of levels of the multi-level label prediction model is empirically determined, a final level of the multi-level label prediction model being identified based on a determined convergence in the label prediction output generated by the final level and a potential next level.

10 . The method of claim 1 , the digital content item is a document and the annotation comprises a number of words contained in the document.

11 . The method of claim 1 , the digital content item comprises an image.

12 . A computer readable non-transitory storage medium tangibly encoded with computer-executable instructions that when executed by a processor associated with a computing device perform a method comprising:
training a multi-level label prediction model to predict an applicability, to a digital content item, of each label of a plurality of labels, the multi-level label prediction model comprising a first level and a second level, the first level being trained using training data missing a number of applicable labels, label prediction output of the trained first level including an applicability prediction for each of the number of applicable labels missing from the training data, the second level being trained using the label prediction output generated by the trained first level; generating the digital content item's label prediction output using the trained multi-level label prediction model, the digital content item's labeling prediction output indicating an applicability, to the digital content item, of each label of the plurality of labels; and automatically generating annotation for the digital content item based on the digital content item's labeling prediction output.

13 . The computer readable non-transitory storage medium of claim 12 , the method further comprising:
identifying a set of digital content items using the digital content item's annotation.

14 . The computer readable non-transitory storage medium of claim 13 , each digital content item in the set of digital content items having a respective annotation generated using label prediction output from the trained multi-level label prediction model.

15 . The computer readable non-transitory storage medium of claim 13 , identifying the set of digital content items is responsive to a received search request.

16 . The computer readable non-transitory storage medium of claim 15 , the received search request comprising information indicating the digital content item.

17 . The computer readable non-transitory storage medium of claim 12 , the training data comprising a plurality of training instances corresponding to a plurality of digital content items, labeling data corresponding to at least one training instance is missing at least one applicable label from the plurality of labels.

18 . The computer readable non-transitory storage medium of claim 17 , a training instance, of the plurality of training instances, comprising associated labeling data and feature data.

19 . The computer readable non-transitory storage medium of claim 18 , further comprising:
analyzing, a digital content item of the plurality of digital content items, and based on the analysis, generating the feature data for the analyzed digital content item.

20 . A system comprising:
a processor; a storage medium for tangibly storing thereon program logic for execution by the processor, the stored logic comprising:
training logic executed by the processor for training a multi-level label prediction model to predict an applicability, to a digital content item, of each label of a plurality of labels, the multi-level label prediction model comprising a first level and a second level, the first level being trained using training data missing a number of applicable labels, label prediction output of the trained first level including an applicability prediction for each of the number of applicable labels missing from the training data, the second level being trained using the label prediction output generated by the trained first level;
generating logic executed by the processor for generating the digital content item's label prediction output using the trained multi-level label prediction model, the digital content item's labeling prediction output indicating an applicability, to the digital content item, of each label of the plurality of labels; and
generating logic executed by the processor for automatically generating annotation for the digital content item based on the digital content item's labeling prediction output.

Assignees

• Yahoo Assets Llc

Inventors

• Li Jia
• Chang Yi
• Kong Xiangnan

Classifications

• G06N20/00 Primary

  Machine learning · CPC title