Shipping of data through ETL stages

Patent metadata

Field	Value
Publication number	US-10262049-B2
Application number	US-201615190223-A
Country	US
Kind code	B2
Filing date	Jun 23, 2016
Priority date	Jun 23, 2016
Publication date	Apr 16, 2019
Grant date	Apr 16, 2019

Abstract

Performing an extract, transform, and load (ETL) process. Column data is received by a stage of the ETL process. The size of the received column data is ascertained by the stage. In response to determining that the size of the column data exceeds a predefined threshold, the stage saves the column data and creates a data locator associated with the column data. The created data locator advances through successive downstream stages of the ETL process as a replacement for the column data.

First claim

What is claimed is:

1. A computer-implemented method for shipping data between stages of an extract, transform, and load (ETL) process, the method comprising:
receiving, by a stage of the ETL process, column data;
ascertaining, by the stage, a size of the column data received; and
in response to a determining, by the stage, that the size of the received column data exceeds a predefined threshold:
saving, by the stage, the column data; and

creating, by the stage, a data locator associated with the column data, which advances through successive downstream stages of the ETL process as a replacement for the column data, wherein the created data locator is advanced through each of the successive downstream stages until the associated column data is required at a downstream stage of the successive downstream stages, wherein, in response to a determination by the downstream stage that the column data associated with the received data locator is required, the downstream stage requests the launched data provider to provide the saved column data associated with the received data locator, wherein, upon receiving the request by the data provider launched by the stage that saved the column data, utilizing the metadata of the data locator in retrieving the associated column data, and wherein the downstream stage, upon receiving the column data from the launched data provider, in response to a determination that the received column data is modified, creates another data locator associated with the modified column data, which advances through successive downstream stages as a replacement for the modified column data.

2. The method of claim 1 , wherein the created data locator is a data structure that includes metadata, the metadata comprising a hash key, wherein the hash key corresponds to the column data associated with the data locator.

3. The method of claim 2 , wherein the stage creating the data locator further launches a data provider, the data provider utilizing the hash key of the data locator to identify the associated column data saved by the stage.

4. A computer program product for shipping data between stages of an extract, transform, and load (ETL) operation, the computer program product comprising a non-transitory computer readable storage medium having program instructions embodied therewith, wherein the computer readable storage medium is not a transitory signal per se, the program instructions readable by a processor to cause the processor to perform a method comprising:
receiving, by a stage of the ETL process, column data;
ascertaining, by the stage, a size of the column data received; and
in response to a determining, by the stage, that the size of the received column data exceeds a predefined threshold:
saving, by the stage, the column data; and
creating, by the stage, a data locator associated with the column data, which advances through successive downstream stages of the ETL process as a replacement for the column data, wherein the created data locator is advanced through each of the successive downstream stages until the associated column data is required at a downstream stage of the successive downstream stages, wherein, in response to a determination by the downstream stage that the column data associated with the received data locator is required, the downstream stage requests the launched data provider to provide the saved column data associated with the received data locator, wherein, upon receiving the request by the data provider launched by the stage that saved the column data, utilizing the metadata of the data locator in retrieving the associated column data, and wherein the downstream stage, upon receiving the column data from the launched data provider, in response to a determination that the received column data is modified, creates another data locator associated with the modified column data, which advances through successive downstream stages as a replacement for the modified column data.

5. The computer program product of claim 4 , wherein the stage creating the data locator further launches a data provider, the data provider utilizing the hash key of the data locator to identify the associated column data saved by the stage.

6. The computer program product of claim 4 , wherein the created data locator is advanced through each of the successive downstream stages until the associated column data is required at a downstream stage of the successive downstream stages.

7. A computer system for shipping data between stages of an extract, transform, and load (ETL) operation, the computer system comprising, a processor to perform a method comprising:
receiving, by a stage of the ETL process, column data;
ascertaining, by the stage, a size of the column data received; and
in response to a determining, by the stage, that the size of the received column data exceeds a predefined threshold:
saving, by the stage, the column data; and
creating, by the stage, a data locator associated with the column data, which advances through successive downstream stages of the ETL process as a replacement for the column data, wherein the created data locator is advanced through each of the successive downstream stages until the associated column data is required at a downstream stage of the successive downstream stages, wherein, in response to a determination by the downstream stage that the column data associated with the received data locator is required, the downstream stage requests the launched data provider to provide the saved column data associated with the received data locator, wherein, upon receiving the request by the data provider launched by the stage that saved the column data, utilizing the metadata of the data locator in retrieving the associated column data, and wherein the downstream stage, upon receiving the column data from the launched data provider, in response to a determination that the received column data is modified, creates another data locator associated with the modified column data, which advances through successive downstream stages as a replacement for the modified column data.

8. The computer system of claim 7 , wherein the created data locator is a data structure that includes metadata, the metadata comprising a hash key, wherein the hash key corresponds to the column data associated with the data locator.

9. The computer system of claim 8 , wherein the stage creating the data locator further launches a data provider, the data provider utilizing the hash key of the data locator to identify the associated column data saved by the stage.

Assignees

• IBM

Inventors

• Chintakayala Kalyanji
• Bonagiri Krishna K
• Jacobson Eric A

Classifications

• H04L63/06

  for supporting key management in a packet data network (cryptographic mechanisms or cryptographic arrangements for key management H04L9/08) · CPC title

• G06F16/254 Primary

  Extract, transform and load [ETL] procedures, e.g. ETL data flows in data warehouses · CPC title

• G06F16/221

  Column-oriented storage; Management thereof · CPC title

• G06F17/30315

  Physics · mapped topic

• G06F17/30563 Primary

  Physics · mapped topic