Run time insertion and removal of buffer operators

Patent metadata

Field	Value
Publication number	US-9679033-B2
Application number	US-201414221834-A
Country	US
Kind code	B2
Filing date	Mar 21, 2014
Priority date	Mar 21, 2014
Publication date	Jun 13, 2017
Grant date	Jun 13, 2017

Abstract

Insertion, in a data warehouse process, of a buffer operator between an upstream data operator and a downstream data operator. The processor processes a data record for the upstream data operator to pass to the downstream data operator. In response to determining that the downstream data operator is not ready to accept the data record from the upstream data operator, a buffer operator is inserted between the upstream data operator and the downstream data operator. The data record processed by the upstream data operator is passed to the buffer operator for storing into a buffer allocated for use by the buffer operator. The data record stored into the buffer is passed to the downstream data operator when the downstream data operator is ready to accept data. In another aspect, when the allocated buffer utilization decreases to a threshold value, the buffer operator is removed and the buffers are released.

First claim

What is claimed is:

1. A method for insertion, in a data warehouse process, of a buffer operator between an upstream data operator and a downstream data operator, the method comprising:
processing, by a processor, a data record for the upstream data operator to pass to the downstream data operator via a direct data communication link;
determining whether the downstream data operator is ready to accept the data record based on a response signal by the downstream data operator, wherein the response signal includes an acknowledgment that the downstream data operator is ready to accept the data record;
in response to determining, by the processor, that the downstream data operator is not ready to accept the data record from the upstream data operator based on the downstream data operator failing to provide the response signal within a specified time interval, inserting, by the processor, a buffer operator between the upstream data operator and the downstream data operator, wherein the buffer operator is inserted using at least one additional data communication link apart from the direct data communication link between the upstream data operator and the downstream data operator, and wherein the direct data communication link is unused until the buffer operator is removed;
passing, by the processor, the data record processed by the upstream data operator to the buffer operator for storing into a buffer allocated for use by the buffer operator;
passing, by the processor, the data record stored into the buffer to the downstream data operator when the downstream data operator is ready to accept data; and
in response to determining, by the processor, that a buffer utilization of the buffer allocated to the buffer operator decreases to a threshold value:
draining the buffer;
removing, by the processor, the buffer operator and releasing the buffer, wherein removal of the buffer operator include re-establishment of direct communication between upstream data operator and downstream data operator over the direct data communication link; and
passing, by the processor, data records processed by the upstream data operator directly to the downstream data operator.

2. The method according to claim 1 , wherein the buffer operator is inserted using the direct data communication link between the upstream data operator and the downstream data operator, and wherein re-establishing direct communication between the upstream data operator and the downstream data operator includes bypassing the buffer operator without removing the buffer operator.

3. The method according to claim 1 , wherein the buffer utilization threshold value is a configurable value.

4. The method according to claim 3 , wherein the buffer utilization threshold value is a percentage of the allocated buffer.

5. The method according to claim 3 , wherein the buffer utilization threshold value is a number of data records in the buffer.

6. A computer program product for insertion, in a data warehouse process, of a buffer operator between an upstream data operator and a downstream data operator, the computer program product comprising one or more computer readable storage medium and program instructions stored on at least one of the one or more computer readable storage medium, the program instructions comprising:
program instructions to process, by a processor, a data record for the upstream data operator to pass to the downstream data operator via a direct data communication link;
program instructions to determine whether the downstream data operator is ready to accept the data record based on a response signal by the downstream data operator, wherein the response signal includes an acknowledgment that the downstream data operator is ready to accept the data record;
in response to program instructions to determine, by the processor, that the downstream data operator is not ready to accept the data record from the upstream data operator based on the downstream data operator failing to provide the response signal within a specified time interval, program instructions to insert, by the processor, a buffer operator between the upstream data operator and the downstream data operator, wherein the buffer operator is inserted using at least one additional data communication link apart from the direct data communication link between the upstream data operator and the downstream data operator, and wherein the direct data communication link is unused until the buffer operator is removed;
program instructions to pass, by the processor, the data record processed by the upstream data operator to the buffer operator for storing into a buffer allocated for use by the buffer operator;
program instructions to pass, by the processor, the data record stored into the buffer to the downstream data operator when the downstream data operator is ready to accept data; and
in response to determining that a buffer utilization of the buffer allocated to the buffer operator decreases to a threshold value:
program instructions to drain the buffer;
program instructions to remove the buffer operator and releasing the buffer, wherein removal of the buffer operator include re-establishment of direct communication between upstream data operator and downstream data operator over the direct data communication link; and
program instructions to pass data records processed by the upstream data operator directly to the downstream data operator.

7. The computer program product according to claim 6 , wherein the buffer operator is inserted using the direct data communication link between the upstream data operator and the downstream data operator, and wherein re-establishing direct communication between the upstream data operator and the downstream data operator includes bypassing the buffer operator without removing the buffer operator.

8. The computer program product according to claim 6 , wherein the buffer utilization threshold value is a configurable value.

9. The computer program product according to claim 8 , wherein the buffer utilization threshold value is a percentage of the allocated buffer.

10. The computer program product according to claim 8 , wherein the buffer utilization threshold value is a number of data records in the buffer.

11. A computer system for insertion, in a data warehouse process, of a buffer operator between an upstream data operator and a downstream data operator, the computer system comprising one or more processors, one or more computer readable memories, one or more computer readable storage medium, and program instructions stored on at least one of the one or more storage medium for execution by at least one of the one or more processors via at least one of the one or more memories, the program instructions comprising:
program instructions to process, by a processor, a data record for the upstream data operator to pass to the downstream data operator via a direct data communication link;
program instructions to determine whether the downstream data operator is ready to accept the data record based on a response signal by the downstream data operator, wherein the response signal includes an acknowledgment that the downstream data operator is ready to accept the data record;
in response to program instructions to determine, by the processor, that the downstream data operator is not ready to accept the data record from the upstream data operator based on the downstream data operator failing to provide the response signal within a specified time interval, program instructions to insert, by the processor, a buffer operator between the upstream data operator and the downstream data operator, wherein the buffer operator is inserted using at least one additional da

Assignees

• IBM

Inventors

• Bonagiri Krishna K
• Chintakayala Kalyanji
• Gupta Ritesh K

Classifications

• G06F16/254 Primary

  Extract, transform and load [ETL] procedures, e.g. ETL data flows in data warehouses · CPC title

• G06F17/30563 Primary

  Physics · mapped topic