Shipping of data through etl stages

Patent metadata

Field	Value
Publication number	US-2017371939-A1
Application number	US-201615190223-A
Country	US
Kind code	A1
Filing date	Jun 23, 2016
Priority date	Jun 23, 2016
Publication date	Dec 28, 2017
Grant date	—

Abstract

Performing an extract, transform, and load (ETL) process. Column data is received by a stage of the ETL process. The size of the received column data is ascertained by the stage. In response to determining that the size of the column data exceeds a predefined threshold, the stage saves the column data and creates a data locator associated with the column data. The created data locator advances through successive downstream stages of the ETL process as a replacement for the column data.

First claim

What is claimed is:

1 . A computer-implemented method for shipping data between stages of an extract, transform, and load (ETL) process, the method comprising:
receiving, by a stage of the ETL process, column data; ascertaining, by the stage, a size of the column data received; and in response to a determining, by the stage, that the size of the received column data exceeds a predefined threshold:
saving, by the stage, the column data; and
creating, by the stage, a data locator associated with the column data, which advances through successive downstream stages of the ETL process as a replacement for the column data.

2 . The method of claim 1 , wherein the created data locator is a data structure that includes metadata, the metadata comprising a hash key, wherein the hash key corresponds to the column data associated with the data locator.

3 . The method of claim 2 , wherein the stage creating the data locator further launches a data provider, the data provider utilizing the hash key of the data locator to identify the associated column data saved by the stage.

4 . The method of claim 1 , wherein the created data locator is advanced through each of the successive downstream stages until the associated column data is required at a downstream stage of the successive downstream stages.

5 . The method of claim 4 , wherein, in response to a determination by the downstream stage that the column data associated with the received data locator is required, the downstream stage requests the launched data provider to provide the saved column data associated with the received data locator.

6 . The method of claim 5 , wherein, upon receiving the request by the data provider launched by the stage that saved the column data, utilizing the metadata of the data locator in retrieving the associated column data.

7 . The method of claim 6 , wherein the downstream stage, upon receiving the column data from the launched data provider, in response to a determination that the received column data is modified, creates another data locator associated with the modified column data, which advances through successive downstream stages as a replacement for the modified column data.

8 . A computer program product for shipping data between stages of an extract, transform, and load (ETL) operation, the computer program product comprising a computer readable storage medium having program instructions embodied therewith, wherein the computer readable storage medium is not a transitory signal per se, the program instructions readable by a processor to cause the processor to perform a method comprising:
receiving, by a stage of the ETL process, column data; ascertaining, by the stage, a size of the column data received; and in response to a determining, by the stage, that the size of the received column data exceeds a predefined threshold:
saving, by the stage, the column data; and
creating, by the stage, a data locator associated with the column data, which advances through successive downstream stages of the ETL process as a replacement for the column data.

9 . The computer program product of claim 8 , wherein the stage creating the data locator further launches a data provider, the data provider utilizing the hash key of the data locator to identify the associated column data saved by the stage.

10 . The computer program product of claim 8 , wherein the created data locator is advanced through each of the successive downstream stages until the associated column data is required at a downstream stage of the successive downstream stages.

11 . The computer program product of claim 10 , wherein, in response to a determination by the downstream stage that the column data associated with the received data locator is required, the downstream stage requests the launched data provider to provide the saved column data associated with the received data locator.

12 . The computer program product of claim 11 , wherein the downstream stage, upon receiving the column data from the launched data provider, in response to a determination that the received column data is modified, creates another data locator associated with the modified column data, which advances through successive downstream stages as a replacement for the modified column data.

13 . A computer system for shipping data between stages of an extract, transform, and load (ETL) operation, the computer program product comprising a computer readable storage medium having program instructions embodied therewith, wherein the computer readable storage medium is not a transitory signal per se, the program instructions readable by a processor to cause the processor to perform a method comprising:
receiving, by a stage of the ETL process, column data; ascertaining, by the stage, a size of the column data received; and in response to a determining, by the stage, that the size of the received column data exceeds a predefined threshold:
saving, by the stage, the column data; and
creating, by the stage, a data locator associated with the column data, which advances through successive downstream stages of the ETL process as a replacement for the column data.

14 . The computer system of claim 13 , wherein the created data locator is a data structure that includes metadata, the metadata comprising a hash key, wherein the hash key corresponds to the column data associated with the data locator.

15 . The computer system of claim 14 , wherein the stage creating the data locator further launches a data provider, the data provider utilizing the hash key of the data locator to identify the associated column data saved by the stage.

16 . The computer system of claim 13 , wherein the created data locator is advanced through each of the successive downstream stages until the associated column data is required at a downstream stage of the successive downstream stages.

17 . The computer system of claim 16 , in response to a determination by the downstream stage that the column data associated with the received data locator is required, the downstream stage requests the launched data provider to provide the saved column data associated with the received data locator.

18 . The computer system of claim 17 , wherein upon receiving the request by the data provider launched by the stage that saved the column data, utilizing the metadata of the data locator in retrieving the associated column data.

19 . The computer system of claim 18 , wherein the downstream stage, upon receiving the column data from the launched data provider, in response to a determination that the received column data is modified, creates another data locator associated with the modified column data, which advances through successive downstream stages as a replacement for the modified column data.

Assignees

• IBM

Inventors

• Chintakayala Kalyanji
• Bonagiri Krishna K
• Jacobson Eric A

Classifications

• H04L63/06

  for supporting key management in a packet data network (cryptographic mechanisms or cryptographic arrangements for key management H04L9/08) · CPC title

• G06F16/254 Primary

  Extract, transform and load [ETL] procedures, e.g. ETL data flows in data warehouses · CPC title

• G06F16/221

  Column-oriented storage; Management thereof · CPC title

• G06F17/30563 Primary

  Physics · mapped topic

• G06F17/30315

  Physics · mapped topic