Distributed execution of data processing pipelines

Patent metadata

Field	Value
Publication number	US-10866831-B2
Application number	US-201816009996-A
Country	US
Kind code	B2
Filing date	Jun 15, 2018
Priority date	Jun 15, 2018
Publication date	Dec 15, 2020
Grant date	Dec 15, 2020

Abstract

A method for executing a data processing pipeline may be provided. The method may include identifying a file providing a runtime environment required for executing a series of data processing operations comprising the data processing pipeline. The file may be identified based on one or more tags associated with the data processing pipeline. The one or more tags may specify at least one runtime requirement for the series of data processing operations. The file may be executed to generate an executable package that includes a plurality of components required for executing the series of data processing operations. The series of data processing operations included in the data processing pipeline may be executed by at least executing the executable package to provide the runtime environment required for executing the series of data processing operations. Related systems and articles of manufacture, including computer program products, are also provided.

First claim

What is claimed is:

1. A system, comprising:
at least one data processor; and
at least one memory storing instructions which, when executed by the at least one data processor, result in operations comprising:
dividing a graph representative of a data processing pipeline, the data processing pipeline including a series of data processing operations, the graph being divided into a first subgraph and a second subgraph, each of the first subgraph and the second subgraph including some but not all of the series of data processing operations included in the data processing pipeline;
identifying a first file providing a first runtime environment required for executing a first data processing operation included in the first subgraphs;
identifying a second file providing a second runtime environment required for executing a second data processing operation included in the second subgraph, each of the first file and the second file comprising a script including a sequence of instructions, and executing the sequence of instructions generates an executable package that includes a plurality of components required for executing each of the first data processing operation and the second data processing operation; and
triggering an execution of the data processing pipeline by at least sending the first file to a first computing node in a cluster of computing nodes and the second file to a second computing node in the cluster of computing nodes the first computing node being scheduled to execute the first data processing operation included in the first subgraph, and the second computing node being scheduled to execute the second data processing operation included in the second subgraph.

2. The system of claim 1 , wherein the first computing node is further scheduled to execute a third data processing operation include in the first subgraph, wherein the second runtime environment is required for executing the third data processing operation, and wherein the execution of the data processing pipeline is further triggered by at least sending, to the first computing node, the second file.

3. The system of claim 1 , further comprising:
scheduling the first computing node to execute the first data processing operation and the second computing node to execute the second data processing operation, the scheduling being based on at least one of a load balance and available resources across the cluster of computing nodes.

4. The system of claim 1 , wherein the plurality of components required for executing the first data processing operation includes at least one of a programming code, a runtime, a library, an environment variable, and a configuration file.

5. The system of claim 1 , wherein the graph includes a plurality of nodes interconnected by one or more edges, wherein each of the plurality of nodes correspond to a data processing operation from the series of data processing operations, and wherein the one or more edges indicate a flow of data between different data processing operations.

6. The system of claim 1 , wherein the first file is identified based at least on a first tag associated with the first file matching a second tag associated with the first data processing operation, wherein the first tag specifies at least one runtime requirement supported by the first file, and wherein the second tag specifies at least one runtime requirement of the first data processing operation.

7. The system of claim 1 , wherein executing the first data processing operation includes at least one of receiving data from a database and sending data to the database.

8. A computer-implemented method, comprising:
dividing a graph representative of a data processing pipeline, the data processing pipeline including a series of data processing operations, the graph being divided into a first subgraph and a second subgraph, each of the first subgraph and the second sub graph including some but not all of the series of data processing operations included in the data processing pipeline;
identifying a first file providing a first runtime environment required for executing a first data processing operation included in the first subgraphs;
identifying a second file providing a second runtime environment required for executing a second data processing operation included in the second subgraph, each of the first file and the second file comprising a script including a sequence of instructions, and executing the sequence of instructions generates an executable package that includes a plurality of components required for executing each of the first data processing operation and the second data processing operation; and
triggering an execution of the data processing pipeline by at least sending the first file to a first computing node in a cluster of computing nodes and the second file to a second computing node in the cluster of computing nodes, the first computing node being scheduled to execute the first data processing operation included in the first subgraph, and the second computing node being scheduled to execute the second data processing operation included in the second subgraph.

9. The method of claim 8 , wherein the first computing node is further scheduled to execute a third data processing operation include in the first subgraph, wherein the second runtime environment is required for executing the third data processing operation, and wherein the execution of the data processing pipeline is further triggered by at least sending, to the first computing node, the second file.

10. The method of claim 8 , further comprising:
scheduling the first computing node to execute the first data processing operation and the second computing node to execute the second data processing operation, the scheduling being based on at least one of a load balance and available resources across the cluster of computing nodes.

11. The method of claim 8 , wherein the plurality of components required for executing the first data processing operation includes at least one of a programming code, a runtime, a library, an environment variable, and a configuration file.

12. The method of claim 8 , wherein the graph includes a plurality of nodes interconnected by one or more edges, wherein each of the plurality of nodes correspond to a data processing operation from the series of data processing operations, and wherein the one or more edges indicate a flow of data between different data processing operations.

13. The method of claim 8 , wherein the first file is identified based at least on a first tag associated with the first file matching a second tag associated with the first data processing operation, wherein the first tag specifies at least one runtime requirement supported by the first file, and wherein the second tag specifies at least one runtime requirement of the first data processing operation.

14. A non-transitory computer-readable medium storing instructions, which when executed by at least one data processor, result in operations comprising:
dividing a graph representative of a data processing pipeline, the data processing pipeline including a series of data processing operations, the graph being divided into a first subgraph and a second subgraph, each of the first subgraph and the second sub graph including some but not all of the series of data processing operations included in the data processing pipeline;
identifying a first file providing a first runtime environment required for executing a first data processing operation included in the first subgraphs;
identifying a second file providing a second runtime environment required for executing a second data processing operation included in

Assignees

• Sap Se

Inventors

• Seif Simon
• Gruschko Boris
• Fitzer Joachim

Classifications

• G06F9/5027 Primary

  the resource being a machine, e.g. CPUs, Servers, Terminals · CPC title

• G06F9/5072

  Grid computing · CPC title

• G06F9/4881 Primary

  Scheduling strategies for dispatcher, e.g. round robin, multi-level priority queues · CPC title

• G06F9/5083

  Techniques for rebalancing the load in a distributed system · CPC title

• G06F2209/5017

  Task decomposition · CPC title