Targeted crowd sourcing for metadata management across data sets

Patent metadata

Field	Value
Publication number	US-11604980-B2
Application number	US-201916419651-A
Country	US
Kind code	B2
Filing date	May 22, 2019
Priority date	May 22, 2019
Publication date	Mar 14, 2023
Grant date	Mar 14, 2023

Abstract

A system includes: a memory operable to store a predictive model; a first processor communicatively coupled to the memory, the first processor operable to execute the predictive model to perform operations including generating knowledge score metrics based on a set of attributes for individuals included in a specified population, where the knowledge score metrics quantify a prediction of a capability of an individual for performing metadata labeling; a second processor communicatively coupled to the memory and the first processor, the second processor is operable to perform operations including comparing the knowledge score metrics to a specified threshold, and identifying attributes of individuals from a specified population having knowledge score metrics exceeding the specified threshold as attributes of individuals capable of performing metadata labeling.

First claim

What is claimed is:

1. A method for generating seed input for an artificial intelligence system, the method comprising:
collecting information on a set of attributes for each individual in an initial set of individuals;
assessing a quality of metadata labels produced during a metadata labeling task performed by each individual in the initial set of individuals;
evaluating, by a processor of the artificial intelligence system, the set of attributes and the quality of metadata labels produced during the metadata labeling task;
based on results of the evaluating, associating quality metrics with the information on the set of attributes for each individual in the initial set of individuals;
comparing, by the processor of the artificial intelligence system, the quality metrics to a threshold;
in response to determining that a first quality metric of the quality metrics exceeds the threshold, identifying, by the processor of the artificial intelligence system, a subset of the set of attributes which is associated with the first quality metric as being attributes of an expert with respect to a type of data in a given data set; and
inputting the subset of the set of attributes to a neural network processor to seed the artificial intelligence system to identify experts.

2. The method of claim 1 , further comprising:
in response to determining that a second quality metric of the quality metrics does not exceed the threshold, inputting, to the neural network processor, a subset of the set of attributes which is associated with the second quality metric in a hold out set of training data for the artificial intelligence system.

3. The method of claim 1 , wherein the quality of the metadata labels produced during the metadata labeling task comprises an assessed degree of accuracy in defining an operational definition of elements of a given data set.

4. The method of claim 1 , wherein the set of attributes comprises one or more of: demographic data, education data, or data regarding a role within a company, and
wherein the set of attributes indicates a degree of familiarity with a given data set.

5. The method of claim 1 , wherein at least some of the metadata labels produced during the metadata labeling task identify a technical description of data in a given data set.

6. The method of claim 1 , wherein at least some of the metadata labels produced during the metadata labeling task identify an operational description of data in a given data set.

7. The method of claim 1 , further comprising identifying training data for a plurality of different machine learning models, wherein each machine learning model of the plurality of different machine learning models is trained based on training data identified for a different type of data set.

8. A non-transitory computer readable medium storing instructions which, when executed by a processor of an artificial intelligence system, cause the processor to perform operations, the operations comprising:
collecting information on a set of attributes for each individual in an initial set of individuals;
assessing a quality of metadata labels produced during a metadata labeling task performed by each individual in the initial set of individuals;
evaluating the set of attributes and the quality of metadata labels produced during the metadata labeling task;
based on results of the evaluating, associating quality metrics with the information on the set of attributes for each individual in the initial set of individuals;
comparing the quality metrics to a threshold;
in response to determining that a first quality metric of the quality metrics exceeds the threshold, identifying a subset of the set of attributes which is associated with the first quality metric as being attributes of an expert with respect to a type of data in a given data set; and
inputting the subset of the set of attributes to a neural network processor to seed the artificial intelligence system to identify experts.

9. The non-transitory computer readable medium of claim 8 , wherein the operations further comprise:
in response to determining that a second quality metric of the quality metrics does not exceed the threshold, inputting, to the neural network processor, a subset of the set of attributes which is associated with the second quality metric in a hold out set of training data for the artificial intelligence system.

10. The non-transitory computer readable medium of claim 8 , wherein the quality of the metadata labels produced during the metadata labeling task comprises an assessed degree of accuracy in defining an operational definition of elements of a given data set.

11. The non-transitory computer readable medium of claim 8 , wherein the set of attributes comprises one or more of: demographic data, education data, or data regarding a role within a company, and
wherein the set of attributes indicates a degree of familiarity with a given data set.

12. The non-transitory computer readable medium of claim 8 , wherein at least some of the metadata labels produced during the metadata labeling task identify a technical description of data in a given data set.

13. The non-transitory computer readable medium of claim 8 , wherein at least some of the metadata labels produced during the metadata labeling task identify an operational description of data in a given data set.

14. The non-transitory computer readable medium of claim 8 , wherein the operations further comprise:
identifying training data for a plurality of different machine learning models, wherein each machine learning model of the plurality of different machine learning models is trained based on training data identified for a different type of data set.

15. An artificial intelligence system comprising:
a processor; and
a non-transitory computer readable medium storing instructions which, when executed by the processor, cause the processor to perform operations, the operations comprising:
collecting information on a set of attributes for each individual in an initial set of individuals;
assessing a quality of metadata labels produced during a metadata labeling task performed by each individual in the initial set of individuals;
evaluating the set of attributes and the quality of metadata labels produced during the metadata labeling task;
based on results of the evaluating, associating quality metrics with the information on the set of attributes for each individual in the initial set of individuals;
comparing the quality metrics to a threshold;
in response to determining that a first quality metric of the quality metrics exceeds the threshold, identifying a subset of the set of attributes which is associated with the first quality metric as being attributes of an expert with respect to a type of data in a given data set; and
inputting the subset of the set of attributes to a neural network processor to seed the artificial intelligence system to identify experts.

16. The artificial intelligence system of claim 15 , wherein the operations further comprise:
in response to determining that a second quality metric of the quality metrics does not exceed the threshold, inputting, to the neural network processor, a subset of the set of attributes which is associated with the second quality metric in a hold out set of training data for the artificial intelligence system.

17. The artificial intelligence system of claim 15 , wherein the quality of the metadata labels produced during the metadata labeling task comprises an assessed degree of accuracy in defining an operational definition of elements of a

Assignees

• At & T Ip I Lp

Inventors

• Woods Jr Robert
• Austin Mark D

Classifications

• G06N3/0499

  Feedforward networks · CPC title

• G06N3/09

  Supervised learning · CPC title

• G06N3/082

  modifying the architecture, e.g. adding, deleting or silencing nodes or connections · CPC title

• G06N3/0442

  characterised by memory or gating, e.g. long short-term memory [LSTM] or gated recurrent units [GRU] · CPC title

• G06N3/08 Primary

  Learning methods · CPC title