Framework to improve parallel job workflow

Patent metadata

Field	Value
Publication number	US-2016203031-A1
Application number	US-201615076239-A
Country	US
Kind code	A1
Filing date	Mar 21, 2016
Priority date	Jun 28, 2013
Publication date	Jul 14, 2016
Grant date	—

Abstract

Embodiments of the inventive subject matter include receiving, in a distributed computing environment, a plurality of files for execution. Embodiments further include identifying, by parsing the plurality of files, code segments contained in each of the plurality of files. Embodiments further include determining, based on a comparison of the code segments and definitions contained in a distributed computing basic function library, a first group of the code segments that include configuration tasks and a second group of the code segments that include computational tasks. Embodiments further include combining the first group of the code segments to form a super configuration task. Embodiments further include creating an executable code, wherein the executable code comprises the super configuration task and the second group of code segments. Embodiments further include allocating the executable code to one or more nodes. Embodiments further include executing the executable code on the one or more nodes.

First claim

What is claimed is:

1 . A method comprising:
receiving, in a distributed computing environment, a plurality of files for execution; identifying, by parsing the plurality of files, code segments contained in each of the plurality of files; determining, based on a comparison of the code segments and definitions contained in a distributed computing basic function library, a first group of the code segments that include configuration tasks and a second group of the code segments that include computational tasks; combining the first group of the code segments to form a super configuration task; creating an executable code, wherein the executable code comprises the super configuration task and the second group of the code segments; allocating the executable code to one or more nodes of the distributed computing environment; and executing the executable code on the one or more nodes of the distributed computing environment.

2 . The method of claim 1 , wherein the super configuration task is only executed once.

3 . The method of claim 1 , wherein the plurality of files are written based on the distributed computing basic function library.

4 . The method of claim 1 , wherein the determining the first group of the code segments and the second group of the code segments comprises:
identifying, from each of the plurality of files based on the definitions contained in the distributed computing basic function library, scheduling tasks and network resource management tasks as configuration tasks; and identifying tasks that are not configuration tasks as being computational tasks.

5 . The method of claim 4 , wherein the scheduling tasks are tasks used for launching one or more of the computational tasks and the network resource management tasks are tasks used for managing network resources needed for executing one or more of the computational tasks.

6 . The method of claim 5 , wherein the network resource management tasks include network initialization tasks and network clear-up tasks.

7 . The method of claim 6 , wherein the network initialization tasks include collecting network addresses for the computational tasks to form a network address table.

8 . The method of claim 7 , wherein the network clear-up tasks include clearing the network address table.

9 . The method of claim 1 , wherein the executing the executable code on one or more nodes of the distributed computing environment comprises:
executing the super configuration task and one or more of the computational tasks in serial; and executing one or more of the computational tasks at least partially in parallel.

Assignees

• IBM

Inventors

• Chai Hua
• Guo Chuan Wei
• He Jun
• Li Guang Lei
• Li Rong

Classifications

• H04L41/0893

  Assignment of logical groups to network elements · CPC title

• G06F9/5083 Primary

  Techniques for rebalancing the load in a distributed system · CPC title

• H04L41/0886 Primary

  Fully automatic configuration · CPC title

• G06F9/52

  Program synchronisation; Mutual exclusion, e.g. by means of semaphores · CPC title

• H04L41/5096

  wherein the managed service relates to distributed or central networked applications · CPC title