Shipping of data though ETL stages

Patent metadata

Field	Value
Publication number	US-10216816-B2
Application number	US-201815987450-A
Country	US
Kind code	B2
Filing date	May 23, 2018
Priority date	Jun 23, 2016
Publication date	Feb 26, 2019
Grant date	Feb 26, 2019

Abstract

Performing an extract, transform, and load (ETL) process. Column data is received by a stage of the ETL process. The size of the received column data is ascertained by the stage. In response to determining that the size of the column data exceeds a predefined threshold, the stage saves the column data and creates a data locator associated with the column data. The created data locator advances through successive downstream stages of the ETL process as a replacement for the column data.

First claim

What is claimed is:

1. A computer program product comprising one or more computer-readable tangible storage media and program instructions stored on at least one of the one or more computer-readable tangible storage media, the program instructions executable by a processor to cause the processor to perform a method of shipping data between stages of an extract, transform, and load (ETL) process, wherein the method comprising:
receiving, by a stage of a plurality of stages associated with the ETL process, column data;
ascertaining, by the stage, a size of the received column data; and
in response to determining, by the stage, that the size of the received column data exceeds a predefined threshold:
saving, by the stage, the column data;
creating, by the stage, a data locator associated with the column data, wherein the created data locator is a data structure that includes metadata comprising (i) a hash key, (ii) a source indicator, and (iii) a timestamp, wherein the hash key corresponds to the saved column data associated with the data locator, the source indicator corresponds to the stage, and the timestamp indicates when the column data was received in the stage,

advancing the created data locator through each of a plurality of successive downstream stages;
receiving, at the stage, an indication that a respective downstream stage requires the column data based on the data locator received at the respective downstream stage;
determining, by the respective successive stage and based on the received data locator, whether the saved column data is a latest version;
in response to determining that the saved column data is the latest version, forwarding the saved column data directly from the stage to the respective downstream stage; and
in response to determining that the saved column data is not the latest version, forwarding the latest version of the column data directly to the respective downstream stage from an upstream stage, different than the stage, containing the latest version of the column data.

Assignees

• IBM

Inventors

• Chintakayala Kalyanji
• Bonagiri Krishna K
• Jacobson Eric A

Classifications

• H04L63/06

  for supporting key management in a packet data network (cryptographic mechanisms or cryptographic arrangements for key management H04L9/08) · CPC title

• G06F16/254 Primary

  Extract, transform and load [ETL] procedures, e.g. ETL data flows in data warehouses · CPC title

• G06F16/221

  Column-oriented storage; Management thereof · CPC title

• G06F17/30315

  Physics · mapped topic

• G06F17/30563 Primary

  Physics · mapped topic