Data extraction and transformation method and system

Patent metadata

Field	Value
Publication number	US-9898515-B1
Application number	US-201414527345-A
Country	US
Kind code	B1
Filing date	Oct 29, 2014
Priority date	Oct 29, 2014
Publication date	Feb 20, 2018
Grant date	Feb 20, 2018

Abstract

A system and method for processing raw transaction records received from multiple data sources. The system and method receive multiple raw transaction records from multiple data sources. Transaction pair records are generated from the raw transaction records. Location and entity fields including raw information are identified from the transaction pair records. The raw location and entity information is resolved to generate resolved location and entity information capable of aggregation and further processing, such as the deriving of analytics.

First claim

What is claimed is:

1. A computer-implemented method comprising:
receiving, by a processing device, a plurality of raw transaction records from a plurality of data sources;
identifying transaction pairs from the raw transaction records, transaction pairs including multiple transactions relating to a common transaction between a transaction source and a transaction destination, at least some transaction pairs including a source transaction, an intermediate transaction, and a destination transaction;
generating a plurality of transaction pair records from the identified transaction pairs, wherein each transaction pair record comprises a plurality of related raw transaction records; identifying one or more selected fields corresponding to one or more selected data categories from each of the plurality of transaction pair records, wherein the one or more selected fields comprise raw information;
wherein the format of the one or more selected fields varies among the transaction pair records such that selected fields are identified based on the use of at least one field identification technique that applies transaction record rules to determine selected fields in at least some transaction pair records;
wherein the one or more selected fields includes at least an entity field;
determining pair match scores corresponding to a plurality of candidate entity names using a similarity measure;
identifying a set of top candidate entity names having similar pair match scores;
performing list matching on the set of top candidate entity names using an adjusted similarity measure to identify a top match;
establishing the top match as the resolved entity information;
resolving the raw information in the one or more identified selected fields to generate resolved information corresponding to the one or more data categories; and
aggregating the resolved information for storing in a data store.

2. The computer-implemented method of claim 1 , wherein the one or more selected fields corresponding to one or more data categories comprise at least one location field and at least one entity field.

3. The computer-implemented method of claim 2 , wherein the at least one location field comprises raw location information and the at least one entity field comprises raw entity information.

4. The computer-implemented method of claim 3 , wherein resolving the raw location information further comprises:
extracting the raw location information from the one or more location fields;
searching one or more geographic databases based on the extracted raw location information;
identifying, based on the search, a plurality of candidate locations comprising city information and state information;
determining a score for each of the plurality of candidate locations; and
identifying a resolved location based on the scores for each of the plurality of candidate locations.

5. The computer-implemented method of claim 3 , wherein resolving the raw entity information further comprises:
searching an entity database based on an entity query associated with the raw entity information to identify a plurality of candidate entity names; and
performing pairwise matching based on the identified plurality of candidate entity names to generate a pair match score for each of the identified plurality of candidate entity names.

6. A system comprising:
a memory; and
a processing device coupled to the memory, the processing device configured to:
receive a plurality of raw transaction records from a plurality of data sources; identify transaction pairs from the raw transaction records, the transaction pairs including multiple transactions relating to a common transaction between a transaction source and a transaction destination, at least some transaction airs including a source transaction, an intermediate transaction, and a destination transaction;
generate a plurality of transaction pair records from the identified transaction pairs, wherein each transaction pair record comprises a plurality of related raw transaction records;
identify one or more selected fields corresponding to one or more selected data categories from each of the plurality of transaction pair records, wherein the one or more selected fields comprise raw information;
wherein the format of the one or more selected fields varies among the transaction pair records such that selected fields are identified based on the use of at least one field identification technique that applies transaction record rules to determine selected fields in at least some transaction pair records;
wherein the one or more selected fields includes at least an entity field;
determine pair match scores corresponding to a plurality of candidate entity names using a similarity measure;
identify a set of top candidate entity names having similar pair match scores;
perform list matching on the set of top candidate entity names using an adjusted similarity measure to identify a top match;
establish the top match as the resolved entity information;
resolve the raw information in the one or more identified selected fields to generate resolved information corresponding to the one or more data categories; and
aggregate the resolved information for storing in a data store.

7. The system of claim 6 , wherein the one or more selected fields corresponding to one or more selected data categories comprise at least one location field and at least one entity field.

8. The system of claim 7 , wherein the at least one location field comprises raw location information and the at least one entity field comprises raw entity information.

9. The system of claim 8 , wherein the processing device is configured to resolve the raw location information by:
extracting the raw location information from the one or more location fields;
searching one or more geographic databases based on the extracted raw location information;
identifying, based on the search, a plurality of candidate locations comprising city information and state information;
determining a score for each of the plurality of candidate locations; and
identifying a resolved location based on the scores for each of the plurality of candidate locations.

10. The system of claim 8 , wherein the processing device is configured to resolve the raw entity information by:
searching an entity database based on an entity query associated with the raw entity information to identify a plurality of candidate entity names; and
performing pairwise matching based on the identified plurality of candidate entity names to generate a pair match score for each of the identified plurality of candidate entity names.

11. A non-transitory computer-readable storage medium comprising instructions that, when executed by a processing device, cause the processing device to perform operations comprising:
receiving a plurality of raw transaction records from a plurality of data sources; identifying transaction pairs from the raw transaction records, the transaction pairs including multiple transactions relating to a common transaction between a transaction source and a transaction destination, at least some transaction airs including a source transaction, an intermediate transaction, and a destination transaction;
generating a plurality of transaction pair records from the identified transaction pairs, wherein each transaction pair record comprises a plurality of related raw transaction records;
identifying one or more selected fields corresponding to one or more selected data categories from each of the plurality of transaction pair records, wherein the one or more selected fields comprise raw information;
wherein the format of the one or more selected fields varie

Assignees

• Jpmorgan Chase Bank Na

Inventors

• Avagyan Vahram
• Fang Andrea
• Prior Robert J
• Shah Vipul V
• Hajek Jonathan Scott

Classifications

• G06F16/29

  Geographical information databases · CPC title

• G06F16/214

  Database migration support · CPC title

• G06F16/254 Primary

  Extract, transform and load [ETL] procedures, e.g. ETL data flows in data warehouses · CPC title

• G06F16/285

  Clustering or classification · CPC title

• G06F17/30241

  Physics · mapped topic