Organizational data enrichment

Patent metadata

Field	Value
Publication number	US-10242258-B2
Application number	US-201514929104-A
Country	US
Kind code	B2
Filing date	Oct 30, 2015
Priority date	Sep 30, 2015
Publication date	Mar 26, 2019
Grant date	Mar 26, 2019

Abstract

In an example embodiment, a fuzzy join operation is performed by, for each pair of records, evaluating a first plurality of features for both records in the pair of records by calculating term frequency-inverse term frequency (TF-IDF) for each token of each field relevant to each feature and based on the calculated TF-IDF for each token of each field relevant to each feature, computing a similarity score based on the similarity function by adding a weight assigned to the TF-IDF for any token that appears in both records. Then a graph data structure is created, having a node for each record in the plurality of records and edges between each of the nodes, except, for each record pair having a similarity score that does not transgress a first threshold, causing no edge between the nodes for the record pair to appear in the graph data structure.

First claim

What is claimed is:

1. A computer-implemented method for orchestration of a plurality of records, each record corresponding to an organization and containing information about the organization, at least some of the plurality of records corresponding to an identical organization as each other, each record comprising a plurality of fields with one or more tokens in each field, the method comprising:
training, using a supervised machine learning algorithm, a supervised machine learning mapping component to calculate a similarity function between records corresponding to organizations based on a first plurality of features, the training comprising executing the supervised machine learning algorithm on training data to learn previously undefined weights assigned to the plurality of features;
performing a fuzzy join operation by:
for each pair of records in the plurality of records:
evaluating the first plurality of features for both records in the pair of records by calculating term frequency-inverse document frequency (TF-IDF) for each token of each field relevant to each feature in the first plurality of features;
modifying the calculated TF-IDF for each token;
based on the modified calculated TF-IDF for each token of each field relevant to each feature in the first plurality of features, computing a similarity score based on the similarity function using the supervised machine learning mapping component by adding the modified TF-IDF for any token that appears in both records in the pair of records;
creating a graph data structure having a node for each record in the plurality of records and edges between each of the nodes, except, for each record pair having a similarity score that does not transgress a first threshold, causing no edge between the nodes for the record pair to appear in the graph data structure;
clustering nodes in the graph data structure into one or more record clusters using a clustering algorithm; and
for each of the one or more record clusters, fusing each record having a node in the record cluster together to create a single record for each record cluster.

2. The method of claim 1 , wherein the clustering algorithm uses reputation/accuracy of sources of the plurality of records as a factor during clustering.

3. The method of claim 1 , wherein the clustering algorithm uses data impendence among sources of the plurality of records as a factor during clustering.

4. The method of claim 1 , wherein the clustering algorithm uses number of values provided by a single data source as a factor during clustering.

5. The method of claim 1 , wherein the similarity function is a Jaccard similarity function.

6. The method of claim 1 , wherein the similarity function is a Levenstein similarity function.

7. The method of claim 1 , wherein the similarity function is a Jaro-Winkler similarity function.

8. A system comprising:
a computer-readable medium having instructions stored there on, which, when executed by a processor, cause the system to perform operations comprising:
training, using a supervised machine learning algorithm, a supervised machine learning mapping component to calculate a similarity function between records corresponding to organizations based on a first plurality of features, the training comprising executing the supervised machine learning algorithm on training data to learn previously undefined weights assigned to the plurality of features, each record corresponding to an organization and containing information about the organization, at least some of the plurality of records corresponding to an identical organization as each other, each record comprising a plurality of fields with one or more tokens in each field, the method comprising:
performing a fuzzy join operation by:
for each pair of records in the plurality of records:
evaluating the first plurality of features for both records in the pair of records by calculating term frequency-inverse document frequency (TF-IDF) for each token of each field relevant to each feature in the first plurality of features;
modifying the calculated TF-IDF for each token;
based on the modified calculated TF-IDF for each token of each field relevant to each feature in the first plurality of features, computing a similarity score based on the similarity function using the supervised machine learning mapping component by adding the modified TF-IDF for any token that appears in both records in the pair of records;
creating a graph data structure having a node for each record in the plurality of records and edges between each of the nodes, except, for each record pair having a similarity score that does not transgress a first threshold, causing no edge between the nodes for the record pair to appear in the graph data structure;
clustering nodes in the graph data structure into one or more record clusters using a clustering algorithm; and
for each of the one or more record clusters, fusing each record having a node in the record cluster together to create a single record for each record cluster.

9. The system of claim 8 , wherein the clustering algorithm uses reputation/accuracy of sources of the plurality of records as a factor during clustering.

10. The system of claim 8 , wherein the clustering algorithm uses data impendence among sources of the plurality of records as a factor during clustering.

11. The system of claim 8 , wherein the clustering algorithm uses number of values provided by a single data source as a factor during clustering.

12. The system of claim 8 , wherein the similarity function is a Jaccard similarity function.

13. The system of claim 8 , wherein the similarity function is a Levenstein similarity function.

14. The system of claim 8 , wherein the similarity function is a Jaro-Winkler similarity function.

15. A non-transitory machine-readable storage medium comprising instructions, which when implemented by one or more machines, cause the one or more machines to perform operations comprising:
training, using a supervised machine learning algorithm, a supervised machine learning mapping component to calculate a similarity function between records corresponding to organizations based on a first plurality of features, the training comprising executing the supervised machine learning algorithm on training data to learn previously undefined weights assigned to the plurality of features, each record corresponding to an organization and containing information about the organization, at least some of the plurality of records corresponding to an identical organization as each other, each record comprising a plurality of fields with one or more tokens in each field, the method comprising:
performing a fuzzy join operation by:
for each pair of records in the plurality of records:
evaluating the first plurality of features for both records in the pair of records by calculating term frequency-inverse document frequency (TF-IDF) for each token of each field relevant to each feature in the first plurality of features;
modifying the calculated TF-IDF for each token;
based on the modified calculated TF-IDF for each token of each field relevant to each feature in the first plurality of features, computing a similarity score based on the similarity function using the supervised machine learning mapping component by adding the modified TF-IDF for any token that appears in both records in the pair of records;
creating a graph data structure having a node for each record in the plurality of records and edges between each of the nodes, except, for each record pair having a similarity score that does not transgress a first threshold, causi

Assignees

• Microsoft Technology Licensing Llc

Inventors

• Guo Songtao
• Degiere Christopher Matthew
• Kumar Aarti
• Lai Alex Ching
• Li Xian

Classifications

• G06F16/215 Primary

  Improving data quality; Data cleansing, e.g. de-duplication, removing invalid entries or correcting typographical errors · CPC title

• G06V30/19167

  Active pattern learning · CPC title

• G06V30/19173

  Classification techniques · CPC title

• G06V30/19147

  Obtaining sets of training patterns; Bootstrap methods, e.g. bagging or boosting · CPC title

• G06V10/761

  Proximity, similarity or dissimilarity measures · CPC title