Distributed computer task management of interrelated network computing tasks

Patent metadata

Field	Value
Publication number	US-10841236-B1
Application number	US-201815941654-A
Country	US
Kind code	B1
Filing date	Mar 30, 2018
Priority date	Mar 30, 2018
Publication date	Nov 17, 2020
Grant date	Nov 17, 2020

Abstract

A system can manage distribution of computing jobs among a plurality of third-party network or cloud computing providers to maximize utilization of available computing resources purchased or otherwise obtained by an entity. The system can determine a dependency relationship between jobs and distribute the jobs among the network computing providers based at least in part on the dependency relationship between the jobs. Moreover, the system can use machine learning algorithms to generate one or more prediction algorithms to predict future computing resource usage demands for performing a set of scheduled and unscheduled jobs. Based at least in part on the resource prediction, the dependency relationship between jobs, service level agreements with network computing service providers, and job resource requirements, the system can determine an improved or optimal distribution of jobs among the network computing service providers that satisfies or best satisfies one or more objective functions to maximize resource utilization.

First claim

What is claimed:

1. A computer-implemented method comprising:
as implemented by an interactive computing system configured with specific computer-executable instructions,
receiving a plurality of job requests, the job requests corresponding to a plurality of jobs to be performed at a plurality of computing clusters comprising one or more network computing systems, each computing cluster located at a different data center, wherein each of the one or more network computing systems provide computing resources that are capable of performing at least one of the plurality of jobs;
generating a job dependency topology indicating a dependency relationship between two or more jobs of the plurality of jobs;
determining a predicted computer resource usage to process the plurality of jobs by applying at least the job dependency topology and job metadata associated with the plurality of jobs as inputs to a prediction model, wherein the prediction model predicts computing resource usage to complete sets of jobs, wherein the prediction model is generated based at least in part on historical data relating to previously completed jobs, and wherein the prediction model comprises a set of weighted parameters that are combined to form the prediction model, the set of weighted parameters determined based on the historical data;
determining a network resource allocation at the plurality of computing clusters based at least in part on the predicted computer resource usage;
allocating network computing resources at the plurality of computing clusters based at least in part on the network resource allocation determined based at least in part on the predicted computer resource usage; and
distributing the plurality of jobs to the plurality of computing clusters based at least in part on the job dependency topology and the allocated network computing resources, wherein at least one job is provided to a different computing cluster of the plurality of computing clusters than at least one other job of the plurality of jobs.

2. The computer-implemented method of claim 1 , wherein each of a plurality of data centers that host the plurality of computing clusters is associated with a different entity than the interactive computing system, and wherein at least one of the plurality of data centers is associated with a different entity than at least one other of the plurality of data centers.

3. The computer-implemented method of claim 1 , wherein the job dependency topology further indicates a dependency relationship between a first portion of a job of the plurality of jobs and a second portion of a job of the plurality of jobs.

4. The computer-implemented method of claim 1 , wherein the job dependency topology indicates that at least one job from the plurality of jobs is not dependent on another job from the plurality of jobs.

5. The computer-implemented method of claim 1 , wherein the job metadata comprises data relating to at least one of a job type, a job priority level, a job owner that requested the job, or a job due date.

6. The computer-implemented method of claim 1 , further comprising generating the prediction model based at least in part on a machine learning algorithm.

7. The computer-implemented method of claim 1 , wherein determining the network resource allocation comprises providing the predicted computer resource usage to an objective function that determines the network resource allocation based on an objective.

8. The computer-implemented method of claim 7 , wherein the objective comprises one or more of reducing processing time, reducing cost, increasing utilization of available computing resources, or satisfying a service level agreement.

9. The computer-implemented method of claim 1 , further comprising updating the job dependency topology based at least in part on the network resource allocation to identify network computing resources assigned to each job within the job dependency topology.

10. The computer-implemented method of claim 1 , wherein allocating the network computing resources at the plurality of computing clusters further comprises maintaining an unallocated portion of network computing resources available at the plurality of computing clusters to enable the processing of unscheduled jobs.

11. A system comprising:
an electronic data store system configured to store job dependency topologies; and
a computing task management system comprising one or more hardware processors, the computing task management system configured to execute specific computer-executable instructions to at least:
receive a plurality of job requests, the job requests corresponding to a plurality of jobs to be performed at a plurality of computing clusters comprising one or more network computing systems, each computing cluster located at a different data center, wherein each of the one or more network computing systems provide computing resources that are capable of performing at least one of the plurality of jobs;
generate a job dependency topology indicating a dependency relationship between two or more jobs of the plurality of jobs;
store the job dependency topology at the electronic data store;
determine a predicted computing resource usage to process the plurality of jobs by applying at least the job dependency topology and job metadata associated with the plurality of jobs as input to a prediction model, wherein the prediction model predicts computing resource usage to complete sets of jobs, wherein the prediction model is generated based at least in part on historical data relating to previously completed jobs, and wherein the prediction model comprises a set of weighted parameters that are combined to form the prediction model, the set of weighted parameters determined based on the historical data;
determine a computing resource allocation at the plurality of computing clusters based at least in part on the predicted computing resource usage;
allocate computing resources at the plurality of computing clusters based at least in part on the computing resource allocation determined based at least in part on the predicted computing resource usage; and
distribute the plurality of jobs to the plurality of computing clusters based at least in part on the job dependency topology and the allocated computing resources, wherein at least one job is provided to a different computing cluster of the plurality of computing clusters than at least one other job of the plurality of jobs.

12. The system of claim 11 , wherein the dependency relationship between the two or more jobs comprises a dependency between a portion of a first job of the plurality of jobs and a portion of a second job of the plurality of jobs.

13. The system of claim 11 , wherein the computing task management system is further configured to generate the prediction model based at least in part on a machine learning algorithm.

14. The system of claim 11 , wherein the computing task management system is further configured to execute specific computer-executable instructions to at least determine the computing resource allocation using an objective function configured to maximize computing resource utilization of available computing resources at the plurality of computing clusters.

15. The system of claim 11 , wherein the computing task management system is further configured to execute specific computer-executable instructions to at least allocate a subset of the computing resources at the plurality of computing clusters to process one or more unscheduled jobs received separately from the plurality of job requests, and wherein the plurality of job requests comprise scheduled jobs.

Assignees

• Electronic Arts Inc

Inventors

• Jin Yu
• Narravula Sundeep
• Aghdaie Navid
• Zaman Kazi Atif-Uz
• Ganeshan Preethi
• Agarwal Tushar
• Feng Cong
• Zagieboylo Drew John

Classifications

• H04L67/1023

  based on a hash applied to IP addresses or costs · CPC title

• H04L67/1014

  based on the content of a request · CPC title

• H04L67/1008

  based on parameters of servers, e.g. available memory or workload (monitoring of computer activity G06F11/30) · CPC title

• H04L47/83

  based on usage prediction · CPC title

• H04L41/147

  for predicting network behaviour · CPC title