Automated entity correlation and classification across heterogeneous datasets

Patent metadata

Field	Value
Publication number	US-10915233-B2
Application number	US-201514864513-A
Country	US
Kind code	B2
Filing date	Sep 24, 2015
Priority date	Sep 26, 2014
Publication date	Feb 9, 2021
Grant date	Feb 9, 2021

Abstract

The present disclosure describes techniques for entity classification and data enrichment of data sets. A data enrichment system is disclosed that can extract, repair, and enrich datasets, resulting in more precise entity resolution and classification for purposes of subsequent indexing and clustering. Disclosed techniques may include performing entity recognition to identify segments of interest that relate to an entity. Related data may be analyzed for classification, which can be used to transform the data for enrichment to its users.

First claim

What is claimed is:

1. A method comprising:
receiving, by a computing system, a data set comprising a column of data from one or more data sources, wherein the computing system is a big data system configured to analyze large data sets;
normalizing, by the computing system, the data set comprising the column of data and having a first format to create a normalized data set in a second column of data by modifying the data in the data set to have a common format for data in the normalized data set;
identifying, by the computing system, a set of patterns for a set of entities in the normalized data set in the column of data using a hierarchy of regular expressions, wherein the set of patterns for the set of entities is identified based on a semantic similarity between the set of patterns for the set of entities in the normalized data set and one or more data sets in a knowledge source,
wherein the knowledge source comprises information published by one of a web site, a web service, or a knowledge store;
extracting, by the computing system, based on the identified set of patterns, entity information corresponding to the set of entities from the normalized data set comprising the data set having the common format;
classifying, by the computing system, the set of entities using the entity information in order to obtain a classification for attributes of the set of entities, wherein the classification for the attributes of the set of entities identifies a type of the set of entities in the column of data;
transforming, automatically by the computing system, the data set based on the classification and the entity information by generating a transformed data set comprising classification attribute metadata for the set of entities in the normalized data set in the second column of data;
determining, by the computing system, a transform script comprising a plurality of transformations applied to the normalized data in the column of data to generate the transformed data set;
rendering, by the computing system, a graphical interface that displays the transformed data set and the transform script comprising the plurality of transformations applied to the normalized data set to generate the transformed data set;
receiving, via the graphical interface, an input by a user selecting the transform script comprising the plurality of transformations;
applying the selected transform script to the data set; and
generating the transformed data set based on the selected transform script.

2. The method of claim 1 , further comprising:
computing a set of pattern metrics, wherein each of the set of pattern metrics is computed for a different pattern in the set of patterns, and wherein the set of patterns includes a plurality of different patterns identified for the set of entities;
determining a difference amongst the set of pattern metrics; and
selecting, based on the difference amongst the set of pattern metrics, a pattern in the set of patterns, wherein the classification is determined based on the selected pattern.

3. The method of claim 1 , further comprising:
identifying text data in the normalized data set, the text data corresponding to each entity of the set of entities;
determining a set of classifications for the set of entities; and
computing a set of classification metrics, each of the set of classification metrics computed for a different classification in the set of classifications; and
wherein the classification is determined based on determining a difference amongst the set of classifications.

4. The method of claim 1 , wherein normalizing the data set to create the normalized data set includes modifying the data set having the first format to an adjusted format of the normalized data set, the adjusted format being different from the first format.

5. The method of claim 1 , wherein identifying the set of patterns includes comparing column data amongst a plurality of columns in the normalized data set, and determining that each of a set of columns in the plurality of columns has an attribute, and wherein the set of patterns for the set of entities is identified based on the normalized data for each column having the attribute.

6. The method of claim 1 , wherein the transformed data set may be generated from the normalized data set by modifying the normalized data set to include data about the classification for the set of entities.

7. The method of claim 1 , further comprising:
rendering, by the computing system, an additional graphical interface that displays the normalized data, that identifies the entity information, and that indicates the classification; and
receiving an input indicating selection of the classification in the additional graphical interface,
wherein the transformed data set is generated upon receiving the input.

8. The method according to claim 1 , wherein the data set comprises a plurality of columns of data.

9. The method according to claim 1 , wherein the graphical interface is an interactive graphical interface.

10. The method according to claim 1 , wherein the graphical interface displays the transformed data set and information indicating a recommended transformation to the normalized data set.

11. The method according to claim 10 , wherein the recommended transformation is performed in accordance with a user input on the graphical interface.

12. The method according to claim 1 , wherein the classification comprises classifying the set of entities into one or more matching domains in accordance with a similarity score.

13. The method according to claim 1 , wherein the plurality of transformations in the transform script comprises a plurality of actions, and wherein the plurality of actions comprise one or more of an update action, a split action, a filter action, an edit column action, an extract action, an insert action, a rename action, a sample action, a join action, an export action, an obfuscate action, a data reformat action, a change case action, or a whitelist filter action.

14. A data enrichment system comprising:
a plurality of data sources; and
a cloud computing infrastructure system comprising:
one or more processors communicatively coupled to the plurality of data sources over at least one communication network; and
a memory coupled to the one or more processors, the memory storing instructions to provide a data enrichment service, wherein the data enrichment system is a big data system configured to analyze large data sets, wherein the instructions, when executed by the one or more processors, cause the one or more processors to:
receive a data set comprising a column of data from one or more data sources of the plurality of data sources;
normalize the data set to create a normalized data set comprising the column of data and having a first format by modifying the data in the data set in a second column of data to have a common format for data in the normalized data set;
identify a set of patterns for a set of entities in the normalized data set in the column of data using a hierarchy of regular expressions, wherein the set of patterns for the set of entities is identified based on a semantic similarity between the set of patterns for the set of entities in the normalized data set and one or more data sets in a knowledge source,
wherein the knowledge source comprises information published by one of a web site, a web service, or a knowledge store;
extract, based on the identified set of patterns, entity information corresponding to the set of entities from the normalized data set comprising the data set having the common format;
classify the set of entities using the entity inf

Assignees

• Oracle Int Corp

Inventors

• Stojanovic Alexander Sasha
• Ogren Philip
• Markey Kevin L
• Kreider Mark

Classifications

• G06T11/26

  Drawing of charts or graphs · CPC title

• G06F16/211

  Schema design and management · CPC title

• G06T2200/24

  involving graphical user interfaces [GUIs] · CPC title

• G06F16/9038

  Presentation of query results · CPC title

• G06F3/04883

  for inputting data by handwriting, e.g. gesture or text · CPC title