Labeling of data for machine learning

Patent metadata

Field	Value
Publication number	US-10902352-B2
Application number	US-202016734570-A
Country	US
Kind code	B2
Filing date	Jan 6, 2020
Priority date	Jun 5, 2014
Publication date	Jan 26, 2021
Grant date	Jan 26, 2021

Abstract

A computer generates labels for machine learning algorithms by retrieving, from a data storage circuit, multiple label sets that contain labels that each classify data points in a corpus of data. A graph is generated that includes a plurality of edges, each edge between two respective labels from different label sets of the multiple label sets. Weights are determined for the plurality of edges based upon a consistency between data points classified by two labels connected by the edges. An algorithm is applied that groups labels from the multiple label sets based upon the weights for the plurality of edges. Data points are identified from the corpus of data that represent conflicts within the grouped labels. An electronic message is transmitted in order to present the identified data points to entities for further classification. A new label set is generated using the further classification received from the entities.

First claim

What is claimed is:

1. A computer implemented method for generating labels for machine learning algorithms, the method comprising:
generating a graph that includes a plurality of edges, each edge between two respective labels from different label sets of multiple label sets, the multiple label sets containing labels that each classify data points in a corpus of data;
determining weights for the plurality of edges based upon a consistency between the data points classified by two labels connected by the edges;
applying an algorithm that creates grouped labels from the multiple label sets and based upon the weights for the plurality of edges; and
identifying data points from the corpus of data that represent conflicts within the grouped labels; and
presenting the identified data points to entities for further classification.

2. The method of claim 1 , further comprising receiving further classification of the multiple label sets from the entities.

3. The method of claim 2 , generating a new label set based upon the grouped labels and the further classification received from the entities.

4. A computer system for generating labels for machine learning algorithms, the computer system comprising:
at least one processor circuit and computer readable storage device that are configured to include:
a label set comparison module configured to:
generate a graph that includes a plurality of edges, each edge between two respective labels from different label sets of multiple label sets, the multiple label sets containing labels that each classify data points in a corpus of data;
determine weights for the plurality of edges based upon a consistency between data points classified by two labels connected by the edges;

a label set coordinator module configured to:
apply an algorithm that creates grouped labels from the multiple label sets and based upon the weights for the plurality of edges; and
identify data points from the corpus of data that represent conflicts within the grouped labels; and

a label set issue handler module configured to:
present the identified data points to entities for further classification.

5. The system of claim 4 , the label set issue handler module further configured to receive further classification of the multiple label sets from the entities.

6. The system of claim 4 , the label set issue handler module further configured to generate a new label set based upon the grouped labels and the further classification received from the entities.

7. A computer program product comprising a computer readable storage medium having program instructions embodied therewith, the program instructions executable by a computer to cause the computer to perform a method comprising:
generating a graph that includes a plurality of edges, each edge between two respective labels from different label sets of multiple label sets, the multiple label sets containing labels that each classify data points in a corpus of data;
determining weights for the plurality of edges based upon a consistency between the data points classified by two labels connected by the edges;
applying an algorithm that creates grouped labels from the multiple label sets and based upon the weights for the plurality of edges;
identifying data points from the corpus of data that represent conflicts within the grouped labels;
presenting the identified data points to entities for further classification.

Assignees

• IBM

Inventors

• Ghosh Prasanta
• Godbole Shantanu R
• Joshi Sachindra
• Merugu Srujana
• Verma Ashish

Classifications

• G06F16/906

  Clustering; Classification · CPC title

• G06F16/9024

  Graphs; Linked lists (G06F16/9027 takes precedence) · CPC title

• G06F16/2228

  Indexing structures · CPC title

• G06N20/00 Primary

  Machine learning · CPC title

• G06F16/2365

  Ensuring data consistency and integrity · CPC title