Data transformations with metadata

Patent metadata

Field	Value
Publication number	US-10521404-B2
Application number	US-201816105292-A
Country	US
Kind code	B2
Filing date	Aug 20, 2018
Priority date	Nov 18, 2015
Publication date	Dec 31, 2019
Grant date	Dec 31, 2019

Abstract

A system may receive a request to derive an output variable from a source variable. The request may include proposed logic to derive the output variable from the source variable. The system may then compare the proposed logic to existing logic to determine the proposed logic is new. In response to the proposed logic being new, the system may generate transformation code configured to execute the proposed logic. The system may further schedule the transformation code for execution at a predetermined time, and then execute the transformation code to generate data for the output variable.

First claim

The invention claimed is:

1. A method comprising:
generating, by a computer based system, metadata that describes a proposed logic;
looking up, by the computer based system, metadata that describes an existing logic in a metadata store;
comparing, by the computer based system, the metadata of the proposed logic to the metadata of the existing logic to determine that the proposed logic is new;
applying, by the computer based system, the proposed logic to a source variable to generate a requested output variable;
returning, by the computer based system, at least one of an existing output variable, a location of the existing output variable, or a copy of the existing output variable rather than a newly derived of the requested output variable; and
applying, by the computer based system, the proposed logic for the requested output variable to the source variable.

2. The method of claim 1 , further comprising generating, by the computer based system, a transformation code in response to the proposed logic being new, wherein the transformation code is configured to execute the proposed logic.

3. The method of claim 1 , further comprising executing, by the computer based system, a transformation code to generate output data for the requested output variable at a predetermined time.

4. The method of claim 1 , further comprising receiving, by the computer based system, a request to derive the requested output variable.

5. The method of claim 1 , wherein the applying the proposed logic uses a transformation code that is dynamically generated and includes machine executable code.

6. The method of claim 1 , further comprising scheduling, by the computer based system, a transformation code for execution at a predetermined time.

7. The method of claim 6 , wherein the scheduling is based on existing scheduled tasks.

8. The method of claim 1 , wherein a request to derive the requested output variable includes a requested execution time.

9. The method of claim 1 , further comprising comparing, by the computer based system, the proposed logic to the existing logic to determine the proposed logic is new.

10. The method of claim 1 , wherein the proposed logic derives output data for the requested output variable from the source variable.

11. The method of claim 1 , further comprising managing, by the computer based system, output data by evaluating logic to limit duplicative output data.

12. The method of claim 1 , further comprising storing, by the computer based system and in response to the proposed logic being new, the metadata describing the proposed logic in the metadata store for comparison to later requests for the requested output variable.

13. The method of claim 1 , wherein the computer based system runs on a distributed file system.

14. The method of claim 1 , further comprising storing, by the computer based system, data generated for the requested output variable in a supported data storage format in a big data management system.

15. The method of claim 1 , wherein the computer based system is a transformation platform.

16. A computer-based system, comprising:
a processor; and
a tangible, non-transitory memory configured to communicate with the processor, the tangible, non-transitory memory having instructions stored thereon that, in response to execution by the processor, cause a computer based system to perform operations comprising:
generating, by the computer based system, metadata that describes a proposed logic;
looking up, by the computer based system, metadata that describes an existing logic in a metadata store;
comparing, by the computer based system, the metadata of the proposed logic to the metadata of the existing logic to determine that the proposed logic is new;
applying, by the computer based system, the proposed logic to a source variable to generate a requested output variable;
returning, by the computer based system, at least one of an existing output variable, a location of the existing output variable, or a copy of the existing output variable rather than a newly derived of the requested output variable; and
applying, by the computer based system, the proposed logic for the requested output variable to the source variable.

17. The system of claim 16 , further comprising generating, by the computer based system, a transformation code in response to the proposed logic being new, wherein the transformation code is configured to execute the proposed logic.

18. The system of claim 16 , further comprising executing, by the computer based system, a transformation code to generate output data for the requested output variable at a predetermined time.

19. The system of claim 16 , further comprising receiving, by the computer based system, a request to derive the requested output variable.

20. An article of manufacture including a non-transitory, tangible computer readable storage medium having instructions stored thereon that, in response to execution by a processor of a computer based system, cause the computer based system to perform operations comprising:
generating, by the computer based system, metadata that describes a proposed logic;
looking up, by the computer based system, metadata that describes an existing logic in a metadata store;
comparing, by the computer based system, the metadata of the proposed logic to the metadata of the existing logic to determine that the proposed logic is new;
applying, by the computer based system, the proposed logic to a source variable to generate a requested output variable;
returning, by the computer based system, at least one of an existing output variable, a location of the existing output variable, or a copy of the existing output variable rather than a newly derived of the requested output variable; and
applying, by the computer based system, the proposed logic for the requested output variable to the source variable.

Assignees

• American Express Travel Related Services Co Inc

Inventors

• Arasan Ravi
• Argüello Carmen Patricia
• Bose Sandeep
• Joshi Kunal Chandrashekhar
• Meyer Matthew Kent
• Ponnambalam Marimuthu
• Prabhakar Himanshu
• Ramasamy Gurusamy
• Seideman Jeremy D

Classifications

• G06F8/10

  Requirements analysis; Specification techniques · CPC title

• G06F16/182 Primary

  Distributed file systems · CPC title

• G06F16/164

  File meta data generation · CPC title

• G06F16/254

  Extract, transform and load [ETL] procedures, e.g. ETL data flows in data warehouses · CPC title

• G06F8/30 Primary

  Creation or generation of source code · CPC title