Mixed intelligence data labeling system for machine learning

Patent metadata

Field	Value
Publication number	US-2020327374-A1
Application number	US-201916381843-A
Country	US
Kind code	A1
Filing date	Apr 11, 2019
Priority date	Apr 11, 2019
Publication date	Oct 15, 2020
Grant date	—

Abstract

A method of hybrid data labeling for machine learning, including receiving multiple unlabeled objects forming an unlabeled data set, pre-labeling the unlabeled data set by a machine learning system to output a pending label data pool, bifurcating the pending label data pool by the machine learning system into high and low confidence sets, dispatching the high confidence set to a machine labeler, dispatching the low confidence set to a human labeler, merging the label sets to return a pre-review label data pool, determining a difference between the pending label data pool and the pre-review label data pool, review labeling the data objects, if the determined difference of the data objects is greater than a predefined error threshold and storing the data objects to a reviewed pool if the determined difference of the data objects is less than and equal to the predefined error threshold.

First claim

What is claimed is:

1 . A method of hybrid data labeling for machine learning, comprising:
receiving a plurality of data objects that are unlabeled, wherein the unlabeled data objects form an unlabeled data set; pre-labeling the unlabeled data set by a machine learning system to output a pending label data pool; bifurcating the pending label data pool by the machine learning system into a high confidence set and a low confidence set; dispatching the high confidence set to a machine labeler to return a machine labeled set; dispatching the low confidence set to at least one human labeler to return a human defined label set; merging the machine labeled set and the human defined label set to return a pre-review label data pool; determining a difference between the pending label data pool and the pre-review label data pool; review labeling of the at least one of the plurality of data objects, if the determined difference of at least one of the plurality of data objects is greater than a predefined error threshold; and storing the at least one of the plurality of data objects to a reviewed pool if the determined difference of at least one of the plurality of data objects is at least one of less than and equal to the predefined error threshold.

2 . The method of hybrid data labeling of claim 1 , further comprising:
merging the reviewed pool and the review labeled at least one of the plurality of data objects into an acceptable label result pool; and storing the acceptable label result pool to the pending label data pool.

3 . The method of hybrid data labeling of claim 2 , further comprising adding metadata comments to the review labeled at least one of the plurality of data objects.

4 . The method of hybrid data labeling of claim 3 , further comprising resetting a portion of the high confidence set to the low confidence set when the determined difference is greater than the predefined error threshold.

5 . The method of hybrid data labeling of claim 4 , further comprising providing feedback to the at least one human labeler when the determined difference is greater than the predefined error threshold.

6 . The method of hybrid data labeling of claim 5 , further comprising dispatching a mis-labeled object from the low confidence set by the at least one human labeler to another of the at least one human labeler.

7 . The method of hybrid data labeling of claim 1 , further comprising training the machine labeler based on the review labeling.

8 . The method of hybrid data labeling of claim 1 , wherein the dispatching of the low confidence set is based on at least one of pre-computed quality, customer provided quantity, bidding price and registered human labor availability.

9 . The method of hybrid data labeling of claim 1 , wherein the determining the difference is based on intersection of union of labels.

10 . The method of hybrid data labeling of claim 1 , wherein the determining the difference is based on matching bonding boxes.

11 . The method of hybrid data labeling of claim 1 , wherein the determining the difference is based on free space differences.

Assignees

• Black Sesame International Holding Ltd

Inventors

• Tsai Mengting
• Wang Guan
• Du Hao

Classifications

• G06V10/774 Primary

  Generating sets of training patterns; Bootstrap methods, e.g. bagging or boosting · CPC title

• G06F18/2155

  characterised by the incorporation of unlabelled data, e.g. multiple instance learning [MIL], semi-supervised techniques using expectation-maximisation [EM] or naïve labelling · CPC title

• G06N3/045

  Combinations of networks · CPC title

• G06F18/241 Primary

  relating to the classification model, e.g. parametric or non-parametric approaches · CPC title

• G06F18/2185

  the supervisor being an automated module, e.g. intelligent oracle · CPC title