Lineage data for data records

Patent metadata

Field	Value
Publication number	US-12061571-B2
Application number	US-202318313142-A
Country	US
Kind code	B2
Filing date	May 5, 2023
Priority date	Nov 18, 2015
Publication date	Aug 13, 2024
Grant date	Aug 13, 2024

Abstract

Systems and methods of the present disclosure may read source data corresponding to a source variable and apply a transformation to the source variable to generate an output variable. Accordingly, one such method comprises receiving a request to access an output variable by applying one or more data transformations to input data; before applying an actual data transformation to the input data, identifying that a match exists with a database of prior data transformations that have been applied to source variables to generate derived variables; identifying that a source variable in the matching one or more data transformations is flagged as sensitive data; and processing the request to access the output variable subject to an access restriction in response to the identification of the source variable that is flagged as sensitive data.

First claim

What is claimed is:

1. A method comprising:
receiving, by a processor, a request to access an output variable by applying one or more data transformations to input data;
before applying an actual data transformation to the input data, identifying, by the processor, that a match exists with a database of prior data transformations that have been applied to source variables to generate derived variables or have been applied to the derived variables to generate existing output variables and the one or more data transformations to the input data;
identifying, by the processor, that a source variable in the matching one or more data transformations is flagged as sensitive data or that a derived variable in the matching one or more data transformations is derived from the source variable that is flagged as sensitive data; and
processing, by the processor, the request to access the output variable subject to an access restriction in response to the identification of the source variable that is flagged as sensitive data.

2. The method of claim 1 , wherein the source variable identifies a field in a record that includes the input data that is requested to be transformed to output data corresponding to the output variable.

3. The method of claim 1 , wherein the database stores a logic map for the existing output variables, wherein the logic map depicts a sequence of data transformations applied to one or more of the source variables that were performed to generate the existing output variables.

4. The method of claim 1 , wherein the database stores a history of one or more transformation processes performed on the source variables to produce the existing output variables such that forward and backward transformation linkages can be re-created for use in analytics.

5. The method of claim 1 , wherein the one or more data transformations includes stripping white space or truncating numbers to a predetermined length.

6. The method of claim 1 , wherein the database comprises a distributed file system.

7. The method of claim 1 , wherein the output variable is configured for ingestion into a data storage format.

8. A computer-based system, comprising:
a computing device comprising a processor and a memory;
machine-readable instructions stored in the memory that, when executed by the processor, cause the computing device to at least:
receive a request to access an output variable by applying one or more data transformations to input data;
before applying an actual data transformation to the input data, identify that a match exists with a database of prior data transformations that have been applied to source variables to generate derived variables or have been applied to the derived variables to generate existing output variables and the one or more data transformations to the input data;
identify that a source variable in the matching one or more data transformations is flagged as sensitive data or that a derived variable in the matching one or more data transformations is derived from the source variable that is flagged as sensitive data; and
process the request to access the output variable subject to an access restriction in response to the identification of the source variable that is flagged as sensitive data.

9. The computer-based system of claim 8 , wherein the source variable identifies a field in a record that includes the input data that is requested to be transformed to output data corresponding to the output variable.

10. The computer-based system of claim 8 , wherein the database stores a logic map for the existing output variables, wherein the logic map depicts a sequence of data transformations applied to one or more of the source variables that were performed to generate the existing output variables.

11. The computer-based system of claim 8 , wherein the database comprises a history of one or more transformation processes performed on the source variables to produce the existing output variables such that forward and backward transformation linkages can be re-created for use in analytics.

12. The computer-based system of claim 8 , wherein the one or more data transformations includes stripping white space or truncating numbers to a predetermined length.

13. The computer-based system of claim 8 , wherein the database comprises a distributed file system.

14. The computer-based system of claim 8 , wherein the output variable is configured for ingestion into a data storage format.

15. A computer readable storage medium having instructions stored thereon that in response to execution by a processor of a computing device, cause the computing device to at least:
receive a request to access an output variable by applying one or more data transformations to input data;
before applying an actual data transformation to the input data, identify that a match exists with a database of prior data transformations that have been applied to source variables to generate derived variables or have been applied to the derived variables to generate existing output variables and the one or more data transformations to the input data;
identify that a source variable in the matching one or more data transformations is flagged as sensitive data or that a derived variable in the matching one or more data transformations is derived from the source variable that is flagged as sensitive data; and
process the request to access the output variable subject to an access restriction in response to the identification of the source variable that is flagged as sensitive data.

16. The computer readable storage medium of claim 15 , wherein the source variable identifies a field in a record that includes the input data that is requested to be transformed to output data corresponding to the output variable.

17. The computer readable storage medium of claim 15 , wherein the database stores a logic map for the existing output variables, wherein the logic map depicts a sequence of data transformations applied to one or more of the source variables that were performed to generate the existing output variables.

18. The computer readable storage medium of claim 15 , wherein the database comprises a history of one or more transformation processes performed on the source variables to produce the existing output variables such that forward and backward transformation linkages can be re-created for use in analytics.

19. The computer readable storage medium of claim 15 , wherein the database comprises a distributed file system.

20. The computer readable storage medium of claim 15 , wherein the output variable is configured for ingestion into a data storage format.

Assignees

• American Express Travel Related Services Co Inc

Inventors

• Aggarwal Ambika
• Arasan Ravi
• Bose Sandeep
• Das Debasish
• Kaushik Ravi K
• Meyer Matthew Kent
• Ramasamy Gurusamy
• Seideman Jeremy D

Classifications

• G06F16/2471

  Distributed queries · CPC title

• G06F16/116 Primary

  Details of conversion of file system types or formats · CPC title