Systems and methods for comparing data across data sources and platforms

Patent metadata

Field	Value
Publication number	US-11003688-B2
Application number	US-201916560124-A
Country	US
Kind code	B2
Filing date	Sep 4, 2019
Priority date	Sep 4, 2019
Publication date	May 11, 2021
Grant date	May 11, 2021

Abstract

A system for data comparison is disclosed. The system may receive a source configuration metadata. The system may configure a data extraction module to extract data from a data set in response to the source configuration metadata. The system may generate a pre-work data from the data set. The system may compare the pre-work data to generate a post-process data set. The system may generate a report corresponding to the post-process data set.

First claim

What is claimed is:

1. A method, comprising:
receiving, by a computer based system, source configuration metadata specifying a key column for a first data source;
extracting, by the computer based system, data from the first data source using a data extraction module, the data extraction module being configured based at least in art on the source configuration metadata;
receiving, by the computer based system, comparison configuration metadata that comprises a report type;
establishing, by the computer based system, a link between the first data source and a second data source to generate linked data, wherein the link comprises the key column;
inserting, by the computer based system, the linked data into a data frame;
analyzing, by the computer based system, the linked data in the data frame to generate a postprocess data set that comprises a table of differences in data elements in the key column of the first data source and the second data source, the post-process data comprising a match percentage and a mismatch percentage for each attribute in the key column; and
generating, by the computer based system, a report based at least in part on the post-process data set, the report type, the match percentage, and the mismatch percentage.

2. The method of claim 1 , wherein the post-process data set further includes at least one of an overlap data or a sample data.

3. The method of claim 2 , further comprising generating, by the computer based system, at least one of an overlap report based on the overlap data or a sample report based on the sample data.

4. The method of claim 1 , wherein the comparison configuration metadata further includes at least one of a comparison name, a comparison description, a column name, a derived logic, or a filter logic.

5. The method of claim 1 , wherein the source configuration metadata includes a database name.

6. The method of claim 1 , wherein the source configuration metadata includes at least one of a data location, a data directory, a table name, a table filter, or a joining column.

7. The method of claim 1 , wherein the source configuration metadata includes at least one of a host name, a port, a username, a password, an index, or a bucket.

8. A system comprising: a processor; and
a tangible, non-transitory memory configured to communicate with the processor, the tangible, non-transitory memory having instructions stored thereon that, in
response to execution by the processor, cause the processor to perform operations comprising:
receiving, by the processor, source configuration metadata specifying a key column for a first data source;
extracting, by the processor, data from the first data source using a data extraction module, the data extraction module being configured based at least in art on the source configuration metadata;
receiving, by the processor, comparison configuration metadata comprises a report type;
establishing, by the processor, a link between the first data source and a second data to generate linked data, wherein the link comprises the key column;
inserting, by the processor, the linked data into a data frame;
analyzing, by the processor, the linked data in the data frame to generate a postprocess data set that comprises a table of differences in data elements in the key column of the first data source and the second data source, the post-process data comprising a match percentage and a mismatch percentage for each attribute in the key column; and
generating, by the processor a report based at least in part on the post-process data set, the report type, the match percentage, and the mismatch percentage.

9. The system of claim 8 , wherein the post-process data set includes at least one of an overlap data or a sample data.

10. The system of claim 9 , wherein the operations further comprise generating, by the processor, at least one of an overlap report based on the overlap data or a sample report based on the sample data.

11. The system of claim 8 , wherein the comparison configuration metadata further includes at least one of a comparison name, a comparison description, a column name, a derived logic, or a filter logic.

12. The system of claim 8 , wherein the source configuration metadata includes a database name.

13. The system of claim 8 , wherein the source configuration metadata includes at least one of a data location, a data directory, a table name, a table filter, or a joining column.

14. The system of claim 8 , wherein the source configuration metadata includes at least one of a host name, a port, a username, a password, an index, or a bucket.

15. An article of manufacture including a non-transitory, tangible computer readable storage medium having instructions stored thereon that, in response to execution by a computer based system, cause the computer based system to perform operations comprising:
receiving, by the computer based system, source configuration metadata specifying a key column for a first data source;
extracting, by the computer based system, data from the first data source using a data extraction module, the data extraction module being configured based at least in art on the source configuration metadata;
receiving, by the computer based system, comparison configuration metadata that comprises a report type;
establishing, by the computer based system, a link between the first data source and a second data source to generate linked data, wherein the link comprises the key column;
inserting, by the computer based system, the linked data into a data frame;
analyzing, by the computer based system, the linked data in the data frame to generate a postprocess data set that comprises a table of differences in data elements in the key column of the first data source and the second data source, the post-process data comprising a match percentage and a mismatch percentage for each attribute in the key column; and
generating, by the computer based system, a report based at least in part on the post-process data set, the report type, the match percentage, and the mismatch percentage.

16. The article of manufacture of claim 15 , wherein the post-process data set includes at least one of an overlap data or a sample data.

17. The article of manufacture of claim 16 , wherein the operations further comprise generating, by the computer based system, at least one of an overlap report based on the overlap data or a sample report based on the sample data.

18. The article of manufacture of claim 15 , wherein the comparison configuration metadata includes at least one of a comparison name, a comparison description, a column name, a derived logic, or a filter logic.

19. The article of manufacture of claim 15 , wherein the source configuration metadata includes at least one of a data location, a data directory, a table name, a table filter, or a joining column.

20. The article of manufacture of claim 15 , wherein the source configuration metadata includes at least one of a database name, a host name, a port, a username, a password, an index, or a bucket.

Assignees

• American Express Travel Related Services Co Inc

Inventors

• Alvarez Karina
• Arya Sachin
• Bose Sandeep
• Godara Subhash
• Handa Anmol
• Singh Manesh Ajay Paul
• Shah Purvi
• SHARMA Bhupesh

Classifications

• G06F16/28

  Databases characterised by their database models, e.g. relational or object models · CPC title

• G06F16/287

  Visualization; Browsing · CPC title

• G06F16/27 Primary

  Replication, distribution or synchronisation of data between databases or within a distributed database system; Distributed database system architectures therefor · CPC title

• G06F16/288

  Entity relationship models · CPC title