Shipping of data through etl stages

Patent metadata

Field	Value
Publication number	US-2018260456-A1
Application number	US-201815987375-A
Country	US
Kind code	A1
Filing date	May 23, 2018
Priority date	Jun 23, 2016
Publication date	Sep 13, 2018
Grant date	—

Abstract

Performing an extract, transform, and load (ETL) process. Column data is received by a stage of the ETL process. The size of the received column data is ascertained by the stage. In response to determining that the size of the column data exceeds a predefined threshold, the stage saves the column data and creates a data locator associated with the column data. The created data locator advances through successive downstream stages of the ETL process as a replacement for the column data.

First claim

What is claimed is:

1 . A computer system for shipping data between stages of an extract, transform, and load (ETL) process, comprising:
one or more processors, one or more computer-readable memories, one or more computer-readable tangible storage media, and program instructions stored on at least one of the one or more computer-readable tangible storage media for execution by at least one of the one or more processors via at least one of the one or more computer-readable memories, wherein the computer system is capable of performing a method comprising: receiving, by a stage of the ETL process, column data; ascertaining, by the stage, a size of the column data received; and in response to a determining, by the stage, that the size of the received column data exceeds a predefined threshold: saving, by the stage, the column data; and creating, by the stage, a data locator associated with the column data, which advances through successive downstream stages of the ETL process as a replacement for the column data, wherein the created data locator is a data structure that includes metadata, the metadata comprising (i) a hash key, (ii) a source indicator, and (iii) a timestamp, wherein the hash key corresponds to the column data associated with the data locator, the source indicator corresponds to the source of origin, and the timestamp indicates when the data was placed in the stage, and wherein the created data locator is advanced through each of the successive downstream stages until the associated column data is required at a downstream stage of the successive downstream stages.

Assignees

• IBM

Inventors

• Chintakayala Kalyanji
• Bonagiri Krishna K
• Jacobson Eric A

Classifications

• H04L63/06

  for supporting key management in a packet data network (cryptographic mechanisms or cryptographic arrangements for key management H04L9/08) · CPC title

• G06F16/254 Primary

  Extract, transform and load [ETL] procedures, e.g. ETL data flows in data warehouses · CPC title

• G06F16/221

  Column-oriented storage; Management thereof · CPC title

• G06F17/30563 Primary

  Physics · mapped topic

• G06F17/30315

  Physics · mapped topic