Deploying parallel data integration applications to distributed computing environments

Patent metadata

Field	Value
Publication number	US-9323619-B2
Application number	US-201313842960-A
Country	US
Kind code	B2
Filing date	Mar 15, 2013
Priority date	Mar 15, 2013
Publication date	Apr 26, 2016
Grant date	Apr 26, 2016

Abstract

System, method, and computer program product to process parallel computing tasks on a distributed computing system, by computing an execution plan for a parallel computing job to be executed on the distributed computing system, the distributed computing system comprising a plurality of compute nodes, generating, based on the execution plan, an ordered set of tasks, the ordered set of tasks comprising: (i) configuration tasks, and (ii) execution tasks for executing the parallel computing job on the distributed computing system, and launching a distributed computing application to assign the tasks of the ordered set of tasks to the plurality of compute nodes to execute the parallel computing job on the distributed computing system.

First claim

What is claimed is:

1. A system, comprising:
one or more computer processors; and
a memory containing a program, which, when executed by the one or more computer processors, performs an operation comprising:
computing, by an integration application, an execution plan for a parallel computing job comprising a set of parallel processing operators to be executed on a distributed computing system, the distributed computing system comprising a plurality of compute nodes, each compute node having a distinct memory;
generating, by the integration application, based on the execution plan, an ordered set of tasks specifying a sequence for executing the set of parallel processing operators of the parallel computing job on the distributed computing system, wherein each of the ordered set of tasks is a driver to at least one parallel processing operator of the set of parallel processing operators, wherein the at least one parallel processing operator is executed on the distributed computing system without modification, and wherein the ordered set of tasks comprises:
(i) one or more configuration tasks, wherein each configuration task includes loading a dynamic library of the at least one parallel processing operator and creating at least one socket channel for communication between each configuration tasks and the at least one parallel processing operator, and
(ii) one or more execution tasks for executing the at least one parallel processing operator; and

launching a distributed computing application to assign the tasks of the ordered set of tasks to the plurality of compute nodes to execute the parallel computing job on the distributed computing system.

2. The system of claim 1 , the operation further comprising:
determining, based on the distributed computing system being a run time computing environment for the parallel computing job, that the parallel computing job is to be executed on the distributed computing system, and not a parallel computing system.

3. The system of claim 1 , wherein the one or more execution tasks includes: (i) starting the at least one parallel processing operator, (ii) monitoring an execution of the at least one parallel processing operator, (iii) passing input data to the at least one parallel processing operator, and (iv) receiving output data from the at least one parallel processing operator.

4. The system of claim 1 , the operation further comprising: responsive to detecting, at a point of failure, a failure in execution of the parallel computing job on the distributed computing system, resuming execution of the parallel computing job on the distributed computing system at the point of failure, without having to restart execution of the parallel computing job on the distributed computing system.

5. A computer program product, comprising:
a non-transitory computer-readable storage medium having computer-readable program code embodied therewith, the computer-readable program code comprising:
computer-readable program code configured to compute, by an integration application, an execution plan for a comprising a set of parallel processing operators to be executed on a distributed computing system, the distributed computing system comprising a plurality of compute nodes, each compute node having a distinct memory;
computer-readable program code configured to generate, by the integration application, based on the execution plan, an ordered set of tasks specifying a sequence for executing the set of parallel processing operators of the parallel computing job on the distributed computing system, wherein each of the ordered set of tasks is a driver to at least one parallel processing operator of the set of parallel processing operators, wherein the at least one parallel processing operator is executed on the distributed computing system without modification, and wherein the ordered set of tasks comprises:
(i) one or more configuration tasks, wherein each configuration task includes loading a dynamic library of the at least one parallel processing operator and creating at least one socket channel for communication between each configuration tasks and the at least one parallel processing operator, and
(ii) one or more execution tasks for executing the at least one parallel processing operator; and

computer-readable program code configured to launch a distributed computing application to assign the tasks of the ordered set of tasks to the plurality of compute nodes to execute the parallel computing job on the distributed computing system.

6. The computer program product of claim 5 , the computer-readable program code further comprising:
determining, based on the distributed computing system being a run time computing environment for the parallel computing job, that the parallel computing job is to be executed on the distributed computing system, and not a parallel computing system.

7. The computer program product of claim 5 , wherein the one or more execution tasks includes: (i) starting the at least one parallel processing operator, (ii) monitoring an execution of the at least one parallel processing operator, (iii) passing input data to the at least one parallel processing operator, and (iv) receiving output data from the at least one parallel processing operator.

8. The computer program product of claim 5 , the computer-readable program code further comprising:
responsive to detecting, at a point of failure, a failure in execution of the parallel computing job on the distributed computing system, resuming execution of the parallel computing job on the distributed computing system at the point of failure, without having to restart execution of the parallel computing job on the distributed computing system.

Assignees

• IBM

Inventors

• Jacobson Eric A
• Li Yong
• Mudambi Shyam R
• Pu Xiaoyan

Classifications

• G06F9/4881

  Scheduling strategies for dispatcher, e.g. round robin, multi-level priority queues · CPC title

• G06F9/5038

  considering the execution order of a plurality of tasks, e.g. taking priority or time dependency constraints into consideration (scheduling strategies G06F9/4881 and subgroups) · CPC title

• G06F9/5066 Primary

  Algorithms for mapping a plurality of inter-dependent sub-tasks onto a plurality of physical CPUs (mappping at compile time, see G06F8/451) · CPC title

• G06F11/1415 Primary

  at system level · CPC title

• G06F9/5027

  the resource being a machine, e.g. CPUs, Servers, Terminals · CPC title