Efficient data caching management in scalable multi-stage data processing systems

Patent metadata

Field	Value
Publication number	US-10452612-B2
Application number	US-201715423384-A
Country	US
Kind code	B2
Filing date	Feb 2, 2017
Priority date	Sep 6, 2016
Publication date	Oct 22, 2019
Grant date	Oct 22, 2019

Abstract

According to some example embodiments, a method includes: receiving, by a processor, from a data source, a processing profile comprising input data blocks and a plurality of operations for executing using the input data blocks; executing, by the processor, one or more of the operations of the processing profile to generate a new output data after each of the executed one or more operations; storing, by the processor, the new output data from at least one of the one or more operations as intermediate cache data; and transmitting, by the processor, the new output data from a final operation from among the one or more operations to the data source for display thereby.

First claim

What is claimed is:

1. A system comprising:
a processor; and
a memory coupled to the processor, wherein the memory stores instructions that, when executed by the processor, cause the processor to:
receive, from a data source, a processing profile comprising input data blocks and a plurality of operations for executing using the input data blocks;
determine whether or not a block of stored intermediate cache data corresponds to an operation from among of the plurality of operations;
in response to determining the block of stored intermediate cache data corresponds to the operation from among the plurality of operations, generate a simplified processing profile based on the block of stored intermediate cache data, the simplified processing profile comprising a subset of the plurality of operations of the processing profile without the operation corresponding to the block of stored intermediate cache data;
execute the simplified processing profile by generating a new output data after each operation of the simplified processing profile;
store the new output data from at least one operation as intermediate cache data; and
transmit the new output data from a final operation in the simplified processing profile to the data source for display thereby.

2. The system of claim 1 , wherein the instructions further cause the processor to, in response to determining the block of stored intermediate cache data corresponds to the operation from among the plurality of operations, identify a location of the stored intermediate cache data among a plurality of worker nodes.

3. The system of claim 1 , wherein generating the simplified processing profile comprises removing the operation corresponding to the block of stored intermediate cache data.

4. The system of claim 1 , wherein the instructions further cause the processor to:
identify a candidate worker node from among a plurality of worker nodes for storing the new output data according to a load balance calculation of at least one of storage space of each of the worker nodes and input/output bandwidth of each of the worker nodes; and
store the new output data at the identified candidate worker node.

5. The system of claim 1 , wherein the instructions further cause the processor to:
identify whether or not there is sufficient space among a plurality of worker nodes to store the new output data; and
in response to determining there is not sufficient space among the plurality of worker nodes, clear a block of pre-stored intermediate cache data having a lower priority level than the new output data.

6. A method comprising:
receiving, by a processor, from a data source, a processing profile comprising input data blocks and a plurality of operations for executing using the input data blocks;
determining, by the processor, whether or not a block of stored intermediate cache data corresponds to an operation from among of the plurality of operations;
in response to determining the block of stored intermediate cache data corresponds to the operation from among the plurality of operations, removing, by the processor, the operation from the processing profile to generate a simplified processing profile, the simplified processing profile comprising a subset of the plurality of operations of the processing profile without the operation corresponding to the block of stored intermediate cache data;
executing, by the processor, the simplified processing profile by generating a new output data after each operation of the simplified processing profile;
storing, by the processor, the new output data from at least one operation as intermediate cache data; and
transmitting, by the processor, the new output data from a final operation in the simplified processing profile to the data source for display thereby.

7. The method of claim 6 , further comprising, in response to determining the block of stored intermediate cache data corresponds to the operation from among the plurality of operations, identifying, by the processor, a location of the stored intermediate cache data among a plurality of worker nodes.

8. The method of claim 6 , wherein generating the simplified processing profile comprises removing, by the processor, the operation corresponding to the block of stored intermediate cache data.

9. The method of claim 6 , further comprising identifying, by the processor, a candidate worker node from among a plurality of worker nodes for storing the new output data according to a load balance calculation of at least one of storage space of each of the worker nodes and input/output bandwidth of each of the worker nodes.

10. The method of claim 9 , further comprising storing, by the processor, the new output data at the identified candidate worker node.

11. The method of claim 6 , further comprising:
identifying, by the processor, whether or not there is sufficient space among a plurality of worker nodes to store the new output data; and
in response to determining there is not sufficient space among the plurality of worker nodes, clearing, by the processor, a block of pre-stored intermediate cache data having a lower priority level than the new output data.

12. A method comprising:
receiving, by a processor, from a data source, a processing profile comprising input data blocks and a plurality of operations for executing using the input data blocks;
determining, by the processor, whether or not a block of stored intermediate cache data corresponds to an operation from among of the plurality of operations; and
in response to determining the block of stored intermediate cache data corresponds to the operation from among the plurality of operations, removing, by the processor, the operation from the processing profile to generate a simplified processing profile, the simplified processing profile comprising a subset of the plurality of operations of the processing profile without the operation corresponding to the block of stored intermediate cache data;
executing, by the processor, one or more of the operations of the processing profile to generate a new output data after each of the executed one or more operations;
storing, by the processor, the new output data from at least one of the one or more operations as intermediate cache data; and
transmitting, by the processor, the new output data from a final operation from among the one or more operations to the data source for display thereby.

13. The method of claim 12 , further comprising, in response to determining the block of stored intermediate cache data corresponds to the operation from among the plurality of operations, identifying, by the processor, a location of the stored intermediate cache data among a plurality of worker nodes.

14. The method of claim 12 , wherein the method further comprises executing, by the processor, each of the plurality of operations from the subset.

15. The method of claim 12 , further comprising identifying, by the processor, a candidate worker node from among a plurality of worker nodes for storing the new output data according to a load balance calculation of at least one of storage space of each of the worker nodes and input/output bandwidth of each of the worker nodes.

16. The method of claim 15 , further comprising storing, by the processor, the new output data at the identified candidate worker node.

17. The method of claim 12 , further comprising:
identifying, by the processor, whether or not there is sufficient space among a plurality of worker nodes to store the new output data; and
in response to determining there is not sufficient space among

Assignees

• Samsung Electronics Co Ltd

Inventors

• Wang Jiayin
• Yang Zhengyu
• Evans Thomas David

Classifications

• G06F3/0634

  by changing the state or mode of one or more devices · CPC title

• G06F3/0685

  Hybrid storage combining heterogeneous device types, e.g. hierarchical storage, hybrid arrays · CPC title

• G06F3/061

  Improving I/O performance · CPC title

• G06F12/0811

  with multilevel cache hierarchies · CPC title

• G06F16/172 Primary

  Caching, prefetching or hoarding of files · CPC title