Computerized control of execution pipelines

Patent metadata

Field	Value
Publication number	US-10831550-B2
Application number	US-201815936606-A
Country	US
Kind code	B2
Filing date	Mar 27, 2018
Priority date	Mar 27, 2018
Publication date	Nov 10, 2020
Grant date	Nov 10, 2020

Abstract

Systems, methods, and other embodiments associated with controlling an execution pipeline are described. In one embodiment, a method includes generating an execution pipeline for executing a plurality of tasks. The example method may also include evaluating execution definitions of the tasks to identify execution properties of the plurality of tasks. The example method may also include assigning each task to an execution environment selected from a set of execution environments based upon execution properties of the task matching execution properties of the execution environments. The example method may also include controlling the execution pipeline to execute each task within the assigned execution environments.

First claim

What is claimed is:

1. A non-transitory computer-readable medium storing computer-executable instructions that when executed by a processor of a computer causes the processor to:
generate an execution pipeline for executing a plurality of tasks derived from a run definition comprising execution definitions of each task of the plurality of tasks, wherein the run definition includes declarative statements describing how each of the tasks are to be executed;
evaluate the execution definitions to identify execution properties of each of the plurality of tasks;
assign each task of the plurality of tasks to an execution environment selected from a set of execution environments based upon the execution properties of each of the tasks matching execution properties of each of the execution environments, wherein each execution environment executes each of the tasks with different resources and functionality;
evaluate a first execution definition of a first task to identify first execution properties specifying functions to execute and resource requirements of the first task;
compare resource requirements and functions specified by the first execution properties against execution properties of available resources and functions provided by each execution environment to assign ranks to each execution environment based upon a degree to which available resources and functions match or exceed the resource requirements and functions specified by the first execution properties;
assign the first task to an execution environment having a highest rank or a rank exceeding a threshold; and
control the execution pipeline to execute each task within the assigned execution environments, wherein the execution definitions of each task are transformed into formats compatible for execution by the assigned execution environments.

2. The non-transitory computer-readable medium of claim 1 , wherein the instructions to assign each task comprise instructions to:
evaluate the first execution definition of the first task to identify the first execution properties specifying functions to execute and resource requirements of the first task;
evaluate a second execution definition of a second task to identify second execution properties specifying functions to execute and resource requirements of the second task;
assign the first task to the first execution environment selected from the set of execution environments based upon the resource requirements and functions specified by the first execution properties matching execution properties of available resources and functions provided by the first execution environment; and
assign the second task to a second execution environment selected from the set of execution environments based upon the resource requirements and functions specified by the second execution properties matching execution properties of available resources and functions provided by the second execution environment.

3. The non-transitory computer-readable medium of claim 1 , wherein the instructions comprise instructions to:
implement asynchronous execution tracking of task execution by the assigned execution environments; and
provide access to results of the asynchronous execution tracking.

4. The non-transitory computer-readable medium of claim 1 , wherein the instructions comprise instructions to:
define a schedule for executing a subset of the tasks across a plurality of assigned execution environments; and
control the execution pipeline to execute the subset of the tasks across the plurality of assigned execution environments according to the schedule.

5. The non-transitory computer-readable medium of claim 1 , wherein the instructions to assign each task comprise instructions to:
determine that a transform task is to be executed within an in-memory execution environment as an atomic operation with results of the transform task being persisted into a database based upon execution properties of the transform task; and
assign the transform task to the in-memory execution environment.

6. The non-transitory computer-readable medium of claim 1 , wherein the instructions comprise instructions to:
create a set of execution pipelines comprising at least the execution pipeline;
assign subsets of the plurality of tasks to each of the execution pipelines of the set of execution pipelines; and
control the set of execution pipelines for parallel execution of tasks of the plurality of tasks.

7. The non-transitory computer-readable medium of claim 1 , wherein the instructions comprise instructions to:
transform an execution definition of a task from a declarative statement format into a format used by an assigned execution environment by creating commands understood by the assigned execution environment from the declarative statements within the execution definition.

8. A computing system, comprising:
a processor connected to memory; and
a pipeline module stored on a non-transitory computer readable medium and configured with instructions that when executed by the processor cause the processor to:
generate an execution pipeline for executing a plurality of tasks derived from a run definition comprising execution definitions of each task of the plurality of tasks, wherein the run definition includes declarative statements describing how each of the tasks are to be executed;
evaluate the execution definitions to identify execution properties of the plurality of tasks;
assign each task of the plurality of tasks to an execution environment selected from a set of execution environments based upon the execution properties of the tasks matching execution properties of the execution environments, wherein each execution environment executes tasks with different resources and functionality;
evaluate a first execution definition of a first task to identify first execution properties specifying functions to execute and resource requirements of the first task;
compare resource requirements and functions specified by the first execution properties against execution properties of available resources and functions provided by each execution environment to assign ranks to each execution environment based upon a degree to which available resources and functions match or exceed the resource requirements and functions specified by the first execution properties;
assign the first task to an execution environment having a highest rank or a rank exceeding a threshold; and
control the execution pipeline to execute each task within the assigned execution environments, wherein execution definitions of each task are transformed into formats compatible for execution by the assigned execution environments.

9. The computing system of claim 8 , wherein the instructions comprise instructions that cause the processor to:
evaluate an execution definition of a volume data task to identify an amount of data to be processed;
compare the amount of data to be processed to execution properties of available data storage and processing resources provided by execution environments to identify a database execution environment providing data storage and processing resources matching or exceeding the amount of data to be processed; and
assign the volume data task to the database execution environment.

10. The computing system of claim 8 , wherein the comprise instructions that cause the processor to:
evaluate an execution definition of a task to determine that the task encapsulates a representational state transfer (REST) service;
compare the execution definition to execution properties of services provided by execution environments to identify a remote cluster execution environment providing services corresponding to the REST service; and
assign the

Assignees

• Oracle Financial Services Software Ltd

Inventors

• Vadapandeshwara Rajaram N
• Rajkumar Charles

Classifications

• G06F9/5038

  considering the execution order of a plurality of tasks, e.g. taking priority or time dependency constraints into consideration (scheduling strategies G06F9/4881 and subgroups) · CPC title

• G06F9/4843 Primary

  by program, e.g. task dispatcher, supervisor, operating system · CPC title

• G06F9/5044 Primary

  considering hardware capabilities · CPC title

• G06F2209/503

  Resource availability · CPC title