Scheduling Applications in CPU and GPU Hybrid Environments

Patent metadata

Field	Value
Publication number	US-2020210228-A1
Application number	US-201816236546-A
Country	US
Kind code	A1
Filing date	Dec 30, 2018
Priority date	Dec 30, 2018
Publication date	Jul 2, 2020
Grant date	—

Abstract

A method may include receiving instructions to process a first application in response to a user request. The method also includes determining whether to store the first application in a first processing queue or a second processing queue based on a comparison between a CPU processing cost associated with the first application and a GPU processing cost associated with the first application. Further, the method includes grouping a first set of applications stored in the first processing queue according to CPU grouping criteria and grouping a second set of applications stored in the second processing queue according to GPU batching criteria. The method also includes causing a CPU to process the grouped first set of applications and a plurality of GPUs to process the grouped second set of applications.

First claim

What is claimed is:

1 . A system, comprising:
a plurality of central processing units (CPUs), each having a plurality of CPU cores; a plurality of graphical processing units (GPUs); one or more hardware processors; and a memory storing computer-executable instructions, that in response to execution by the one or more hardware processors, causes the system to perform operations comprising:
receiving a request to process one or more applications;
for each application in the one or more applications:
determining a respective CPU cost corresponding to processing of the application by the CPU and a respective GPU cost corresponding to processing of the application by the GPU; and
based on comparing the respective CPU cost and the respective GPU cost, determining whether to store the application in a first processing queue associated with the CPU or in a second processing queue associated with the plurality of GPUs;

grouping a first set of applications assigned to the first processing queue into one or more CPU tasks according to CPU grouping criteria;
causing the CPU to process the one or more CPU tasks;
grouping a second set of applications assigned to the second processing queue into one or more GPU batches according to GPU batching criteria; and
causing the plurality of GPUs to process the one or more GPU batches.

2 . The system of claim 1 , wherein the CPU cost corresponds to a first amount of time to process the application by the plurality of CPUs, and the GPU cost corresponds to a second amount of time to process the application by the plurality of GPUs.

3 . The system of claim 1 , wherein the determining whether to store the application in the first processing queue or in the second processing queue further comprises:
determining a first CPU cost and a first GPU cost associated with a first application of the one or more applications; and in response to determining that the first CPU cost is less than the second GPU cost, storing the first application in the first processing queue.

4 . The system of claim 1 , wherein the determining whether to store the application in the first processing queue or in the second processing queue further comprises:
determining a first CPU cost and a first GPU cost associated with a first application of the one or more applications; and in response to determining that the first GPU cost is less than the second CPU cost, storing the first application in the second processing queue.

5 . The system of claim 1 , wherein the first processing queue includes applications that were previously assigned to the first processing queue based one or more previous requests.

6 . The system of claim 1 , wherein the second processing queue includes applications that were previously assigned to the second processing queue based one or more previous requests.

7 . The system of claim 1 , wherein the grouping the first set of applications into the one or more CPU tasks according to the CPU grouping criteria further comprises:
determining a number of threads included in the first processing queue, wherein a number of CPU tasks corresponding to the one or more CPU tasks is equal to the number of threads.

8 . The system of claim 7 , wherein each CPU task is associated with respective total CPU cost comprised of a sum of the respective CPU costs of each application included in the CPU task, and wherein a difference between any two respective total CPU costs is within a threshold range.

9 . The system of claim 1 , wherein the grouping the second set of applications according to the GPU batching criteria further comprises:
determining that first applications from the second set of applications are of a first application type; grouping the first multiple applications into a first GPU batch of the one or more GPU batches; determining that second applications from the second set of applications are of a second application type that is different from the first application type; and grouping the second multiple applications into a second GPU batch of the one or more GPU batches.

10 . The system of claim 9 , wherein the causing the plurality of GPUs to process the one or more GPU batches further comprises:
causing the first GPU batch to be processed by a first GPU of the plurality of GPUs and the second GPU batch to be processed by a second GPU of the plurality of GPUs, the first GPU being different from the second GPU.

11 . A method, comprising:
storing, by a system comprising one or more hardware processors, first data of a first application in a CPU processing queue based on a first CPU processing cost corresponding to a first application identifier of the first application being less than a first GPU processing cost corresponding to the first application identifier; storing second data of a second application in a GPU processing queue based on a second GPU processing cost corresponding to a second application identifier of the second application being less than a second CPU processing cost corresponding to the second application identifier; generating a batch that includes the second data and third data of a third application stored in the GPU processing queue based on determining that the second application identifier matches a third application identifier associated with the third application; causing a first GPU of a plurality of GPUs to process the batch; and in response to determining an amount of time for the first GPU to process the batch, determining whether to adjust the second GPU processing cost corresponding to the second application identifier.

12 . The method of claim 11 , wherein the second GPU processing cost includes a previously determined amount of time taken for at least one of the plurality of GPUs to process data of an application associated with the second application identifier.

13 . The method of claim 12 , wherein the determining whether to adjust the second GPU processing cost further comprises:
determining whether the amount of time for the first GPU to process the batch is different from the previously determined amount of time.

14 . The method of claim 11 , further comprising:
causing a CPU to process the first data; and in response to determining a second amount of time for the CPU to process the first data, adjusting the first CPU processing cost corresponding to the first application identifier.

15 . The method of claim 14 , wherein the causing the CPU to process the first data further comprises:
grouping the first data with other data associated with other applications included in the CPU processing queue based at least in part on the first CPU processing cost.

16 . The method of claim 11 , wherein the first application corresponds to a first neural network model, and the second application corresponds to a second neural network model.

17 . A non-transitory computer readable medium storing computer-executable instructions that in response to execution by one or more hardware processors, causes a payment provider system to perform operations comprising:
receiving instructions to process a first application in response to a user request; determining whether to store the first application in a first processing queue or a second processing queue based on a comparison between a CPU processing cost associated with the first application and a GPU processing cost associated with the first application; grouping a first set of applications stored in the first processing queue according to CPU grouping criteria; grouping a second set of applications stored in the seco

Assignees

• Paypal Inc

Inventors

• Wu Yuehao
• Xu Xin
• Zhang Junhua
• Zhang Pengshan

Classifications

• G06N3/045

  Combinations of networks · CPC title

• G06F9/4843

  by program, e.g. task dispatcher, supervisor, operating system · CPC title

• G06F9/5027

  the resource being a machine, e.g. CPUs, Servers, Terminals · CPC title

• G06T1/20 Primary

  Processor architectures; Processor configuration, e.g. pipelining · CPC title

• G06F9/4881 Primary

  Scheduling strategies for dispatcher, e.g. round robin, multi-level priority queues · CPC title