Generating a data lineage record to facilitate source system and destination system mapping

Patent metadata

Field	Value
Publication number	US-2019197149-A1
Application number	US-201715852398-A
Country	US
Kind code	A1
Filing date	Dec 22, 2017
Priority date	Dec 22, 2017
Publication date	Jun 27, 2019
Grant date	—

Abstract

A device may generate, in a first data structure of a first system, a data lineage record. The data lineage record may include attributes that are common among a plurality of records of the first data structure. The data lineage record may be used to map the first data structure and a second data structure of a second system. The device may populate the data lineage record with unique values for the attributes. The unique values may be used to identify the attributes of the data lineage record after the data lineage record has been included in the second data structure. The device may provide, from the first system, data included in the first data structure to the second system. The data lineage record may be included in the data provided to the second system. The device may perform an action related to the data provided to the second system.

First claim

1 . A first system, comprising:
one or more memories; and one or more processors, communicatively coupled to the one or more memories, to:
generate, in a first data structure of the first system, a data lineage record,
the data lineage record including attributes that are common among a plurality of records of the first data structure,
the data lineage record to be used to map the first data structure and a second data structure of a second system;

populate the data lineage record with unique values for the attributes of the data lineage record,
each of the unique values corresponding to one of the attributes, and
the unique values to be used to identify the attributes of the data lineage record after the data lineage record has been included in the second data structure;

provide, from the first system, data included in the first data structure to the second system to cause the data to be included in the second data structure of the second system,
the data lineage record being included in the data provided to the second system;
receive, from the second system, information regarding a manner in which the data lineage record was included in the second data structure by the second system,
the information regarding the manner identifying one or more of:
 whether all or a portion of the data lineage record was included in the second data structure,
 whether any of the unique values in the data lineage record were modified from what the first system provided, or
 whether the second data structure includes any attributes that are different from the attributes in the data lineage record;

map the first data structure and the second data structure based on the information regarding the manner; and

perform an action related to the data after mapping the first data structure and the second data structure.

2 . The first system of claim 1 , where, when mapping the first data structure and the second data structure, the one or more processors are to:
map the first data structure and the second data structure by mapping the unique values of the data lineage record in the first data structure and the unique values of the data lineage record in the second data structure.

3 . The first system of claim 1 , where a unique value, of the unique values, includes:
a unique alphanumeric string, a unique numerical value, a unique date stamp, or a unique timestamp.

4 . The first system of claim 1 , where the data lineage record includes:
an identifier that identifies that the data lineage record is to be used to determine the data lineage of the data.

5 . The first system of claim 1 , where the information regarding the manner includes information indicating a manner in which the second system modified the unique values of the data lineage record prior to including the data lineage record in the second data structure.

6 . The first system of claim 1 , where the one or more processors, when performing the action, are to:
generate a report that identifies a mapping of the first data structure and the second data structure; and output the report for display after generating the report.

7 . The first system of claim 1 , where the one or more processors are further to:
receive a request to perform analytics on a subset of the data in the first data structure; identify the data lineage record in the subset of the data based on an identifier included in the data lineage record; remove the data lineage record from the subset of the data on which analytics are to be performed after identifying the data lineage record; and perform the analytics on the subset of the data after removing the data lineage record from the subset of the data.

8 . A non-transitory computer-readable medium storing instructions, the instructions comprising:
one or more instructions that, when executed by one or more processors, cause the one or more processors to:
generate, in a first data structure of a first system, a data lineage record,
the data lineage record including attributes that are common among a plurality of records of the first data structure;

populate the data lineage record with unique values for the attributes of the data lineage record,
each of the unique values corresponding to one of the attributes;

provide, from the first system, the data lineage record and data included in the first data structure to a second system to facilitate inclusion of the data lineage record and the data in a second data structure of the second system;
receive, from the second system, information regarding a manner in which the data lineage record was included in the second data structure by the second system,
the information regarding the manner identifying one or more of:
whether all or a portion of the data lineage record was included in the second data structure,
whether any of the unique values in the data lineage record were modified from what the first system provided, or
whether the second data structure includes any attributes that are different from the attributes in the data lineage record;

map the first data structure and the second data structure based on the information regarding the manner; and
perform an action related to the data after mapping the first data structure and the second data structure.

9 . The non-transitory computer-readable medium of claim 8 , where the one or more instructions, when executed by the one or more processors, further cause the one or more processors to:
generate the unique values for the attributes by generating at least one of:
a unique date stamp,
a set of unique Boolean values,
a unique timestamp,
a unique alphanumeric string, or
a unique numerical value; and
where the one or more instructions, that cause the one or more processors to populate the data lineage record with the values, cause the one or more processors to:
populate the data lineage record with at least one of:
the unique date stamp,
the unique timestamp,
the unique alphanumeric string, or
the unique numerical value.

10 . The non-transitory computer-readable medium of claim 8 , where a data type of a value, of the unique values, matches a data type of a corresponding attribute, of the attributes, for the value.

11 . The non-transitory computer-readable medium of claim 8 , where the unique values are different from other values of one or more other records of the first data structure.

12 . The non-transitory computer-readable medium of claim 8 ,
where the attributes include a first attribute and a second attribute, where the unique values include a first value and a second value, where the one or more instructions, that cause the one or more processors to populate the data lineage record with the unique values, cause the one or more processors to:
populate the first attribute of the data lineage record with the first value; and
populate the second attribute of the data lineage record with the second value.

13 . The non-transitory computer-readable medium of claim 8 , where the one or more instructions, that cause the one or more processors to generate the data lineage record, cause the one or more processors to:
insert the data lineage record into the first data structure; and insert another attribute into the first data structure,
the other attribute to identify that the data lineage record is to be used to facilitate mapping of the first data structure and the second data structure.

14 . The non-transitory computer-readable medium of claim 8 , where the one or more instructions, when executed by the one or more processors, further cause the one or more processors t

Assignees

• Capital One Services Llc

Inventors

• Chapin Ann

Classifications

• G06F17/30303

  Physics · mapped topic

• G06F17/30539

  Physics · mapped topic

• G06F17/30241

  Physics · mapped topic

• G06F17/30371 Primary

  Physics · mapped topic

• G06F17/30566

  Physics · mapped topic