Job distribution within a grid environment

Patent metadata

Field	Value
Publication number	US-11275609-B2
Application number	US-201916363562-A
Country	US
Kind code	B2
Filing date	Mar 25, 2019
Priority date	Jun 20, 2012
Publication date	Mar 15, 2022
Grant date	Mar 15, 2022

Abstract

According to one aspect of the present disclosure, a technique for job distribution within a grid environment includes receiving jobs at a submission cluster for distribution of the jobs to one or more execution clusters where each of the execution clusters has one or more execution hosts and maintains a pending job queue. A resource capacity corresponding to each respective execution cluster is determined, and resource requirements for the jobs is determined. A length of the pending job queue indicating a quantity of pending jobs to maintain in the pending job queue for the respective execution cluster is dynamically calculated and periodically adjusted based on the resource capacity and the resource requirements of the respective execution clusters. The jobs are allocated to the respective execution clusters according to the length of the pending job queue of the respective execution clusters.

First claim

What is claimed is:

1. A method for job distribution within a grid environment, comprising:
receiving jobs at a submission cluster for distribution of the jobs to one or more execution clusters, each of the execution clusters having one or more execution hosts, and each of the execution clusters maintaining a pending job queue;
determining a resource capacity corresponding to each respective execution cluster;
determining resource requirements for the jobs;
dynamically calculating and periodically adjusting a length of the pending job queue indicating a quantity of pending jobs to maintain in the pending job queue for the respective execution cluster based on the resource capacity and the resource requirements of the respective execution clusters; and
allocating the jobs to the respective execution clusters according to the length of the pending job queue of the respective execution clusters.

2. The method of claim 1 , further comprising calculating the length of the pending job queue for a respective execution cluster based on an amount of resources allocated to running jobs and an amount of resources requested by forwarded pending jobs for the respective execution cluster.

3. The method of claim 1 , further comprising adjusting the length of the pending job queue for a respective execution cluster based on a particular resource being requested by a particular job.

4. The method of claim 1 , further comprising using a forwarding resource ratio to derive the length of a respective pending job queue length, the forwarding resource ratio applied to the resource capacity of a respective execution cluster.

5. The method of claim 1 , further comprising using a forwarding resource ratio to derive the length of a respective pending job queue length, the forwarding resource ratio applied to a particular type of resource of a respective execution cluster.

6. The method of claim 1 , further comprising using a forwarding resource ratio to derive the length of a respective pending job queue length, the forwarding resource ratio applied to all resources of a respective execution cluster.

7. The method of claim 1 , further comprising using a first forwarding resource ratio and a second forwarding resource ratio to derive the length of a respective pending job queue length, the first and second forwarding resource ratios applied to different levels within the environment.

8. The method of claim 7 , wherein the first forwarding resource ratio is applied at a higher level within the environment than the second forwarding resource ratio, and wherein the second forwarding resource ratio will overwrite the first forwarding resource ratio.

9. A system for job distribution within a grid environment, comprising:
a submission cluster, having a processor, for distributing jobs to at least one of a plurality of execution clusters, wherein each execution cluster has one or more execution hosts, and wherein each of the execution clusters maintains a pending job queue, and wherein the submission cluster comprises logic executable by a processor unit to:
determine a resource capacity corresponding to each respective execution cluster;
determine resource requirements for the jobs;
dynamically calculate and periodically adjust a length of the pending job queue indicating a quantity of pending jobs to maintain in the pending job queue for the respective execution cluster based on the resource capacity and the resource requirements of the respective execution clusters; and
allocate the jobs to the respective execution clusters according to the length of the pending job queue of the respective execution clusters.

10. The system of claim 9 , wherein the logic is executable to calculate the length of the pending job queue for a respective execution cluster based on an amount of resources allocated to running jobs and an amount of resources requested by forwarded pending jobs for the respective execution cluster.

11. The system of claim 9 , wherein the logic is executable to adjust the length of the pending job queue for a respective execution cluster based on a particular resource being requested by a particular job.

12. The system of claim 9 , wherein the logic is executable to use a forwarding resource ratio to derive the length of a respective pending job queue length, the forwarding resource ratio applied to the resource capacity of a respective execution cluster.

13. The system of claim 9 , wherein the logic is executable to use a forwarding resource ratio to derive the length of a respective pending job queue length, the forwarding resource ratio applied to a particular type of resource of a respective execution cluster.

14. The system of claim 9 , wherein the logic is executable to use a forwarding resource ratio to derive the length of a respective pending job queue length, the forwarding resource ratio applied to all resources of a respective execution cluster.

15. The system of claim 9 , wherein the logic is executable to use a first forwarding resource ratio and a second forwarding resource ratio to derive the length of a respective pending job queue length, the first and second forwarding resource ratios applied to different levels within the environment.

16. The system of claim 15 , wherein the first forwarding resource ratio is applied at a higher level within the environment than the second forwarding resource ratio, and wherein the logic is executable to overwrite the first forwarding resource ratio with the second forwarding resource ratio.

17. A computer program product for job distribution within a grid environment, the computer program product comprising:
a non-transitory computer readable medium having computer readable program code embodied therewith, the computer readable program code comprising computer readable program code configured to:
receive and distribute jobs to at least one of a plurality of execution clusters, wherein each execution cluster has one or more execution hosts, and wherein each of the execution clusters maintains a pending job queue;
determine a resource capacity corresponding to each respective execution cluster;
determine resource requirements for the jobs;
dynamically calculate and periodically adjust a length of the pending job queue indicating a quantity of pending jobs to maintain in the pending job queue for the respective execution cluster based on the resource capacity and the resource requirements of the respective execution clusters; and
allocate the jobs to the respective execution clusters according to the length of the pending job queue of the respective execution clusters.

18. The computer program product of claim 17 , wherein the computer readable program code is configured to calculate the length of the pending job queue for a respective execution cluster based on an amount of resources allocated to running jobs and an amount of resources requested by forwarded pending jobs for the respective execution cluster.

19. The computer program product of claim 17 , wherein the computer readable program code is configured to use a forwarding resource ratio to derive the length of a respective pending job queue length, the forwarding resource ratio applied to the resource capacity of a respective execution cluster.

20. The computer program product of claim 17 , wherein the computer readable program code is configured to use a first forwarding resource ratio and a second forwarding resource ratio to derive the length of a respective pending job queue length, the first and second forwarding reso

Assignees

• IBM

Inventors

• Chen Chong
• Liu Fang
• Wang Qi
• Yuan Shutao

Classifications

• G06F9/5072 Primary

  Grid computing · CPC title

• H04L67/1001

  for accessing one among a plurality of replicated servers · CPC title

• G06F9/5038

  considering the execution order of a plurality of tasks, e.g. taking priority or time dependency constraints into consideration (scheduling strategies G06F9/4881 and subgroups) · CPC title

• G06F9/5061

  Partitioning or combining of resources · CPC title

• G06F9/505

  considering the load · CPC title