Deploying parallel data integration applications to distributed computing environments

Patent metadata

Field	Value
Publication number	US-9594637-B2
Application number	US-201414224640-A
Country	US
Kind code	B2
Filing date	Mar 25, 2014
Priority date	Mar 15, 2013
Publication date	Mar 14, 2017
Grant date	Mar 14, 2017

Abstract

System, method, and computer program product to process parallel computing tasks on a distributed computing system, by computing an execution plan for a parallel computing job to be executed on the distributed computing system, the distributed computing system comprising a plurality of compute nodes, generating, based on the execution plan, an ordered set of tasks, the ordered set of tasks comprising: (i) configuration tasks, and (ii) execution tasks for executing the parallel computing job on the distributed computing system, and launching a distributed computing application to assign the tasks of the ordered set of tasks to the plurality of compute nodes to execute the parallel computing job on the distributed computing system.

First claim

What is claimed is:

1. A method to process parallel computing tasks on a distributed computing system, comprising:
computing, by an integration application, an execution plan for a parallel computing job comprising a set of parallel processing operators to be executed on the distributed computing system, the distributed computing system comprising a plurality of compute nodes, each computing node having a distinct memory;
generating, by the integration application, based on the execution plan, an ordered set of tasks specifying a sequence for executing the set of parallel processing operators of the parallel computing job on the distributed computing system, wherein each of the ordered set of tasks is a driver to at least one parallel processing operator of the set of parallel processing operators, wherein the at least one parallel processing operator is executed on the distributed computing system without modification, and wherein the ordered set of tasks comprises:
(i) one or more configuration tasks, wherein each configuration task includes loading a dynamic library of the at least one parallel processing operator and creating at least one socket channel for communication between each configuration tasks and the at least one parallel processing operator, and
(ii) one or more execution tasks for executing the at least one parallel processing operator; and
launching a distributed computing application to assign the tasks of the ordered set of tasks to the plurality of compute nodes to execute the parallel computing job on the distributed computing system.

2. The method of claim 1 , further comprising:
determining, based on the distributed computing system being a run time computing environment for the parallel computing job, that the parallel computing job is to be executed on the distributed computing system, and not a parallel computing system.

3. The method of claim 1 , wherein the one or more execution tasks include: (i) starting the at least one parallel operator, (ii) monitoring an execution of the at least one parallel processing operator, (iii) passing input data to the at least one parallel processing operator, and (iv) receiving output data from the at least one parallel processing operator.

4. The method of claim 1 , further comprising: responsive to detecting, at a point of failure, a failure in execution of the parallel computing job on the distributed computing system, resuming execution of the parallel computing job on the distributed computing system at the point of failure, without having to restart execution of the parallel computing job on the distributed computing system.

Assignees

• IBM

Inventors

• Jacobson Eric A
• Li Yong
• Mudambi Shyam R
• Pu Xiaoyan

Classifications

• G06F11/1415 Primary

  at system level · CPC title

• G06F9/5038

  considering the execution order of a plurality of tasks, e.g. taking priority or time dependency constraints into consideration (scheduling strategies G06F9/4881 and subgroups) · CPC title

• G06F9/5027

  the resource being a machine, e.g. CPUs, Servers, Terminals · CPC title

• G06F9/5066 Primary

  Algorithms for mapping a plurality of inter-dependent sub-tasks onto a plurality of physical CPUs (mappping at compile time, see G06F8/451) · CPC title

• G06F9/4881

  Scheduling strategies for dispatcher, e.g. round robin, multi-level priority queues · CPC title