Targeted data acquisition for model training

Patent metadata

Field	Value
Publication number	US-11537915-B2
Application number	US-202016874531-A
Country	US
Kind code	B2
Filing date	May 14, 2020
Priority date	May 14, 2020
Publication date	Dec 27, 2022
Grant date	Dec 27, 2022

Abstract

Targeted acquisition of data for model training includes automatically generating metadata describing samples, of an initial dataset, in neighborhoods of an embedding space in which the samples are embedded. The samples described by the automatically generated metadata are classified by a classification model, and include both correctly classified samples in the neighborhoods and incorrectly classified samples in the neighborhoods. Additionally, attributes of one or more correctly classified samples of the collection of samples and one or more incorrectly classified samples of the collection of samples are identified, and queries are generated based on the identified attributes, the queries tailored, based on the attributes, to retrieve additional training data for training the classification model to more accurately classify samples and avoid incorrect sample classification.

First claim

What is claimed is:

1. A computer-implemented method comprising:
automatically generating metadata describing samples, of an initial dataset, in neighborhoods of an embedding space in which the samples are embedded, the samples described by the automatically generated metadata being classified by a classification model, and including both correctly classified samples in the neighborhoods and incorrectly classified samples in the neighborhoods;
identifying attributes of one or more correctly classified samples of the collection of samples and one or more incorrectly classified samples of the collection of samples; and
generating queries based on the identified attributes, the queries tailored, based on the attributes, to retrieve additional training data for training the classification model to more accurately classify samples and avoid incorrect sample classification.

2. The method of claim 1 , further comprising:
obtaining an embedding of a collection of samples, of the initial dataset, in the embedding space, the embedding presenting a respective classification of each sample of the collection of samples by the classification model; and
defining the neighborhoods of the embedding space, wherein each neighborhood of the neighborhoods comprises a respective at least one incorrectly classified sample of the collection of samples, embedded in the embedding space, that the classification model incorrectly classified and a respective at least one correctly classified sample of the collection of samples, embedded in the embedding space, that the classification model correctly classified, wherein the samples described by the automatically generated metadata are samples in the defined neighborhoods.

3. The method of claim 2 , wherein the defining the neighborhoods comprises using at least one radius to define a region of the embedding space centered around an incorrectly classified sample, wherein the region defines a neighborhood of the neighborhoods, and wherein any samples within the region are samples within that neighborhood.

4. The method of claim 1 , further comprising constructing, using the automatically generated metadata, knowledge graphs that inform the identified attributes of the one or more correctly classified samples of the collection of samples and the one or more incorrectly classified samples of the collection of samples.

5. The method of claim 4 , wherein the automatically generating the metadata comprises initially generating metadata for each neighborhood of the neighborhoods, wherein the constructing the knowledge graphs comprises constructing a respective knowledge graph for each neighborhood of the neighborhoods based on the metadata generated for that neighborhood, wherein the method further comprises combining attributes informed by at least some of the constructed knowledge graphs to construct an aggregate knowledge graph, and wherein the generating the queries generates at least one query of the generated queries from the aggregate knowledge graph.

6. The method of claim 1 , wherein:
the collection of samples comprises at least one selected from the group consisting of: images and text; and
the automatically generated metadata comprises at least one selected from the group consisting of: (i) caption information of the images and (ii) object, subject, and relationship information of the text.

7. The method of claim 1 , wherein the collection of samples comprises images, wherein at least some of the identified attributes comprise an object depicted in at least some of the images, wherein the automatically generated metadata comprises visibility flags indicating whether the object is visible in the images, and wherein the generating the queries uses the visibility flags in determining whether to tailor at least one query to include or omit results in which the object is visible.

8. The method of claim 1 , wherein the generated queries are implemented as crawl paths for crawling remote resources to retrieve the additional training data.

9. The method of claim 1 , wherein at least some of the generated queries are multimodal, in which a multimodal query queries for both text and image results.

10. The method of claim 1 , further comprising:
ranking the generated queries into a ranked list of queries; and
selecting, from the ranked list of queries, a query to issue, wherein the selecting applies a budget comprising one or more constraints on resources to execute the query, the resources comprising at least one selected from the group consisting of: computing cost, memory, time, and electrical power.

11. The method of claim 1 , further comprising:
issuing at least one query of the generated queries;
retrieving, in response to the issuing, additional samples on which to train the classification model; and
retraining the classification model using the additional samples.

12. The method of claim 11 , further comprising, based on the retraining:
iterating one or more times: (i) the automatically generating metadata, (ii) the identifying attributes, (iii) the generating queries, (iv) the issuing at least one query, (v) the retrieving additional samples, and (vi) the retraining the classification model; and
checking whether to halt the iterating by testing stability of the classification model and determining whether a threshold has been reached to avoid overfitting the classification model.

13. A computer system comprising:
a memory; and
a processor in communication with the memory, wherein the computer system is configured to perform a method comprising:
automatically generating metadata describing samples, of an initial dataset, in neighborhoods of an embedding space in which the samples are embedded, the samples described by the automatically generated metadata being classified by a classification model, and including both correctly classified samples in the neighborhoods and incorrectly classified samples in the neighborhoods;
identifying attributes of one or more correctly classified samples of the collection of samples and one or more incorrectly classified samples of the collection of samples; and
generating queries based on the identified attributes, the queries tailored, based on the attributes, to retrieve additional training data for training the classification model to more accurately classify samples and avoid incorrect sample classification.

14. The computer system of claim 13 , wherein the method further comprises:
obtaining an embedding of a collection of samples, of the initial dataset, in the embedding space, the embedding presenting a respective classification of each sample of the collection of samples by the classification model; and
defining the neighborhoods of the embedding space, wherein each neighborhood of the neighborhoods comprises a respective at least one incorrectly classified sample of the collection of samples, embedded in the embedding space, that the classification model incorrectly classified and a respective at least one correctly classified sample of the collection of samples, embedded in the embedding space, that the classification model correctly classified, wherein the samples described by the automatically generated metadata are samples in the defined neighborhoods.

15. The computer system of claim 13 , wherein the method further comprises constructing, using the automatically generated metadata, knowledge graphs that inform the identified attributes of the one or more correctly classified samples of the collection of samples and the one or more incorrectly classified samples of the collection of samples.

16. The computer sy

Assignees

• IBM

Inventors

• Kabra Namit
• Gupta Ritesh Kumar
• Ekambaram Vijay
• MARVANIYA Smitkumar Narotambhai

Classifications

• G06N5/04 Primary

  Inference or reasoning models · CPC title

• G06N5/022 Primary

  Knowledge engineering; Knowledge acquisition · CPC title

• G06N20/00

  Machine learning · CPC title

• G06N3/0475

  Generative networks · CPC title

• G06N3/0895

  Weakly supervised learning, e.g. semi-supervised or self-supervised learning · CPC title