Parallel execution of stateful black box operators

Patent metadata

Field	Value
Publication number	US-12293216-B2
Application number	US-202117554802-A
Country	US
Kind code	B2
Filing date	Dec 17, 2021
Priority date	Dec 17, 2021
Publication date	May 6, 2025
Grant date	May 6, 2025

Abstract

Some embodiments provide a system and method to receive, as an input, configuration properties of a group of operators of a data pipeline, the data pipeline including a specified multiplicity greater than one (1); generate, as an output, a configuration for two new operators, including a first new operator and a second new operator; and automatically insert the first new operator and the second new operator into a deployment of the data pipeline, the first new operator being inserted before a number of replicas of the group of operators of the data pipeline corresponding to the specified multiplicity and the second new operator being inserted after the number of replicas of the group of operators of the data pipeline corresponding to the specified multiplicity.

First claim

What is claimed is:

1. A computer-implemented method, the method comprising:
receiving, as an input, configuration properties of a group of operators of a data pipeline, the data pipeline including a specified multiplicity greater than 1;
generating, as an output, a configuration for two new operators, including a first new operator that transforms a dataset output of a preceding operator according to the configuration properties of the group of operators of the data pipeline and dispatches the transformed dataset to the replicas of the group of operators of the data pipeline and a second new operator that collects results of the replicas of the group of operators of the data pipeline and computes, based on the configuration properties of the group of operators of the data pipeline and an order preserving requirement, a result dataset that it provides to a succeeding operator; and
automatically inserting the first new operator and the second new operator into a deployment of the data pipeline, the first new operator being inserted before a number of replicas of the group of operators of the data pipeline corresponding to the specified multiplicity and the second new operator being inserted after the number of replicas of the group of operators of the data pipeline corresponding to the specified multiplicity.

2. The method of claim 1 , wherein the configuration properties specify properties defining a parallel computation of an execution of the group of operators of the data pipeline.

3. The method of claim 2 , wherein the configuration properties comprise:
characteristics of a restructure configuration of an input dataset into another dataset;
characteristics of a transduction configuration of an input dataset with respect to an incremental state; and
characteristics of an aggregation configuration of results obtained for each element of the input dataset.

4. The method of claim 1 , further comprising receiving a predefined indication of a value for at least one of the configuration properties.

5. The method of claim 1 , further comprising executing of the group of operators of the deployed data pipeline having the first new operator and the second new operator inserted therein, wherein the group of operators are executed in parallel.

6. The method of claim 1 , wherein an execution of the group of operators of the deployed data pipeline having the first new operator and the second new operator inserted therein generates a result equivalent to a result generated using a deployment of the data pipeline without any replicas of the group of operators of the data pipeline.

7. A system comprising:
a memory storing processor-executable program code; and
a processing unit to execute the processor-executable program code to cause the system to:
receive, as an input, configuration properties of a group of operators of a data pipeline, the data pipeline including a specified multiplicity greater than 1;
generate, as an output, a configuration for two new operators, including a first new operator that transforms a dataset output of a preceding operator according to the configuration properties of the group of operators of the data pipeline and dispatches the transformed dataset to the replicas of the group of operators of the data pipeline and a second new operator that collects results of the replicas of the group of operators of the data pipeline and computes, based on the configuration properties of the group of operators of the data pipeline and an order preserving requirement, a result dataset that it provides to a succeeding operator, and
automatically insert the first new operator and the second new operator into a deployment of the data pipeline, the first new operator being inserted before a number of replicas of the group of operators of the data pipeline corresponding to the specified multiplicity and the second new operator being inserted after the number of replicas of the group of operators of the data pipeline corresponding to the specified multiplicity.

8. The system of claim 7 , wherein the configuration properties specify properties defining a parallel computation of an execution of the group of operators of the data pipeline.

9. The system of claim 8 , wherein the configuration properties comprise:
characteristics of a restructure configuration mapping of an input dataset into another dataset;
characteristics of a transduction configuration mapping of an input dataset with respect to an incremental state; and
characteristics of an aggregation configuration mapping of results obtained for each element of the input dataset.

10. The system of claim 7 , further comprising receiving a predefined indication of a value for at least one of the configuration properties.

11. The system of claim 7 , further comprising executing of the group of operators of the deployed data pipeline having the first new operator and the second new operator inserted therein, wherein the group of operators are executed in parallel.

12. The system of claim 7 , wherein an execution of the group of operators of the deployed data pipeline having the first new operator and the second new operator inserted therein generates a result equivalent to a result generated using a deployment of the data pipeline without any replicas of the group of operators of the data pipeline.

13. A non-transitory, computer-readable medium storing program code, the program code executable by a computer processor to cause the processor to perform a method comprising:
receiving, as an input, configuration properties of a group of operators of a data pipeline, the data pipeline including a specified multiplicity greater than 1;
generating, as an output, a configuration for two new operators, including a first new operator that transforms a dataset output of a preceding operator according to the configuration properties of the group of operators of the data pipeline and dispatches the transformed dataset to the replicas of the group of operators of the data pipeline and a second new operator that collects results of the replicas of the group of operators of the data pipeline and computes, based on the configuration properties of the group of operators of the data pipeline and an order preserving requirement, a result dataset that it provides to a succeeding operator; and
automatically inserting the first new operator and the second new operator into a deployment of the data pipeline, the first new operator being inserted before a number of replicas of the group of operators of the data pipeline corresponding to the specified multiplicity and the second new operator being inserted after the number of replicas of the group of operators of the data pipeline corresponding to the specified multiplicity.

14. The medium of claim 13 , wherein the configuration properties specify properties defining a parallel computation of an execution of the group of operators of the data pipeline.

15. The medium of claim 14 , wherein the configuration properties comprise:
characteristics of a restructure configuration of an input dataset into another dataset;
characteristics of a transduction configuration of an input dataset with respect to an incremental state; and
characteristics of an aggregation configuration of results obtained for each element of the input dataset.

16. The medium of claim 13 , further comprising receiving a predefined indication of a value for at least one of the configuration properties.

17. The medium of claim 13 , further comprising executing of the group of operators of the deployed data pipeline having the f

Assignees

• Sap Se

Inventors

• Simon Eric

Classifications

• G06F9/52

  Program synchronisation; Mutual exclusion, e.g. by means of semaphores · CPC title

• G06F9/4843 Primary

  by program, e.g. task dispatcher, supervisor, operating system · CPC title

• G06F9/46 Primary

  Multiprogramming arrangements · CPC title