Declarative language and visualization system for recommended data transformations and repairs

Patent metadata

Field	Value
Publication number	US-10891272-B2
Application number	US-201514864496-A
Country	US
Kind code	B2
Filing date	Sep 24, 2015
Priority date	Sep 26, 2014
Publication date	Jan 12, 2021
Grant date	Jan 12, 2021

Abstract

The present disclosure relates generally to a data enrichment service that extracts, repairs, and enriches datasets, resulting in more precise entity resolution and correlation for purposes of subsequent indexing and clustering. As the data enrichment service can include a visual recommendation engine and language for performing large-scale data preparation, repair, and enrichment of heterogeneous datasets. This enables the user to select and see how the recommended enrichments (e.g., transformations and repairs) will affect the user's data and make adjustments as needed. The data enrichment service can receive feedback from users through a user interface and can filter recommendations based on the user feedback.

First claim

What is claimed is:

1. A method comprising:
identifying, automatically by a data enrichment system comprising a processor and a memory, patterns in data from one or more data sources by parsing fields in a structure of the data;
matching, automatically by the data enrichment system, the patterns in data from the one or more data sources to entity information from a knowledge service;
generating, automatically by the data enrichment system, one or more transformation scripts for the data based on the entity information, wherein the one or more transformation scripts comprises a script for transforming the data to correspond to the patterns matching the entity information, and wherein the one or more transformation scripts are written in a declarative language;
generating, automatically by the data enrichment system based on a history of user activity, one or more transform recommendations corresponding to the one or more transformation scripts generated for the data based on the entity information;
displaying, by the data enrichment system, the one or more transform recommendations and the data to be transformed on an interactive user interface, wherein the one or more transform recommendations displayed on the interactive user interface are interactive and wherein the one or more transform recommendations that are displayed are based on the identified patterns in the data from the one or more data sources;
filtering, by the data enrichment system, the one or more transform recommendations in accordance with user input on the interactive user interface;
receiving, by the data enrichment system, transformation instructions based on a selection of the one or more transform recommendations on the interactive user interface;
transforming, automatically by the data enrichment system, the data displayed on the interactive user interface by applying the transformation instructions, corresponding to the one or more transformation scripts, to the data displayed on the interactive user interface in order to process large data sets;
displaying the transformed data and the transformation scripts applied to the transformed data on the interactive user interface; and
publishing, by the data enrichment system, the transformed data to one or more data targets based on the transformation instructions.

2. The method of claim 1 , further comprising:
receiving a selection of at least one column of data from at least one of the one or more data sources; and
causing a data profile of the at least one column of data to be displayed, wherein the data profile indicates a plurality of patterns identified in the at least one column of data and at least one data visualization associated with the at least one column of data.

3. The method of claim 1 , wherein the transformation instructions include a transformation instruction to rename at least one column of data based on the entity information.

4. The method of claim 3 , further comprising:
receiving a further transformation instruction to rename the at least one column of data to a default name.

5. The method of claim 1 , wherein the transformation instructions include a transformation instruction to reformat at least one column of data based on the entity information.

6. The method of claim 5 , wherein the transformation instructions include a further transformation instruction to obfuscate at least one column of data based on the entity information, wherein to obfuscate the at least one column of data comprises one of truncating, randomizing, and deleting a portion of the data in a plurality of entries in the at least one column of data.

7. The method of claim 1 , wherein the transformation instructions include an enrichment instruction to add one or more columns of data obtained from the knowledge service based on the entity information.

8. The method according to claim 1 , wherein the one or more data sources comprise one of a table, a column in the table, a dataset, and a document.

9. The method according to claim 1 , wherein the one or more data targets comprises a data warehouse.

10. The method according to claim 1 , wherein the entity information comprises one of a social security number, a telephone number, an address, and a proper name of an entity.

11. The method according to claim 1 , wherein the received transformation instructions is based on a user interaction with the one or more transform recommendations displayed on the interactive user interface.

12. A system comprising:
a plurality of data sources;
a plurality of data targets; and
a cloud computing infrastructure system comprising:
one or more processors communicatively coupled to the plurality of data sources and communicatively coupled to the plurality of data targets, over at least one communication network; and
a memory coupled to the one or more processors, the memory storing instructions to provide a data enrichment service, wherein the instructions, when executed by the one or more processors, cause the one or more processors to:
receive a data enrichment request from a client device;
automatically identify patterns in data from at least one data source specified in the data enrichment request by parsing fields in a structure of the data;
automatically match the patterns in data from the one or more data sources to entity information from a knowledge service;
automatically generate one or more transformation scripts for the data based on the entity information, wherein the one or more transformation scripts comprises a script for transforming the data to correspond to the patterns matching the entity information, and wherein the one or more transformation scripts are written in a declarative language;
automatically generate based on a history of user activity one or more transform recommendations corresponding to the one or more transformation scripts generated for the data based on the entity information;
display the one or more transform recommendations and the data to be transformed on an interactive user interface on the client device, wherein the one or more transform recommendations displayed on the interactive user interface are interactive and wherein the one or more transform recommendations that are displayed are based on the identified patterns in the data from the one or more data sources;
filter the one or more transform recommendations in accordance with user input on the interactive user interface;
receive transformation instructions based on a selection of the one or more transform recommendations on the interactive user interface;
automatically transform the data displayed on the interactive user interface by applying the transformation instructions, corresponding to the one or more transformation scripts, to the data displayed on the interactive user interface in order to process large data sets;
displaying the transformed data and the transformation scripts applied to the transformed data on the interactive user interface; and
publish the transformed data to one or more data targets based on the transformation instructions.

13. The system of claim 12 , wherein the instructions, when executed by the one or more processors, further cause the one or more processors to:
receive a selection of at least one column of data from at least one of the plurality of data sources; and
cause a data profile of the at least one column of data to be displayed in the interactive user interface on the client device, wherein the data profile indicates a plurality of patterns identified in the at least one column of data and at least one data visualization associated with the at least one column of data.

Assignees

• Oracle Int Corp

Inventors

• Stojanovic Alexander Sasha
• Rivas Luis E
• Ogren Philip
• Murray Glenn Allen

Classifications

• G06F16/215 Primary

  Improving data quality; Data cleansing, e.g. de-duplication, removing invalid entries or correcting typographical errors · CPC title

• G06F16/2379

  Updates performed during online database operations; commit processing · CPC title

• G06F16/211 Primary

  Schema design and management · CPC title

• G06F3/04842

  Selection of displayed objects or displayed text elements (G06F3/0482 takes precedence) · CPC title