Pipelining and parallelizing graph execution method for neural network model computation and apparatus thereof

Patent metadata

Field	Value
Publication number	US-12468921-B2
Application number	US-202217838342-A
Country	US
Kind code	B2
Filing date	Jun 13, 2022
Priority date	Apr 27, 2022
Publication date	Nov 11, 2025
Grant date	Nov 11, 2025

Abstract

The present disclosure provides a pipelining and parallelizing graph execution method for neural network model computation and apparatus, and provides a pipelining and parallelizing graph execution method for neural network model computation and apparatus in a deep learning training system. The method includes the graph execution flow in a neural network model computation process and a process of cooperative work of all functional modules. The pipelining and parallelizing graph execution method for neural network model computation includes creating a graph executive on a native machine according to a physical computation graph compiled and generated by a deep learning framework.

First claim

What is claimed is:

1 . A pipelining and parallelizing graph execution method for neural network model computation, wherein several executives are provided in a neural network model; a total of 2*N executives are provided, and N is a positive integer; several memory blocks are provided in the executive; the method specifically comprises the following steps:
S1, dividing training data into several batches of subdata; S2, inputting the several batches of subdata into the neural network model in sequence; executing, by an nth executive, self-kernel function computation on an ith batch of subdata after the ith batch of subdata is input, and writing an execution result into an idle memory block of the nth executive; then inputting an (i+1)th batch of subdata, wherein i and n are both positive integers; S3, executing, by the nth executive, the operation in S2 on the (i+1)th batch of subdata, and sending an address of the memory block where the ith batch is located to an (n+1)th executive after the (i+1)th batch of subdata is input; parsing, by the (n+1)th executive, the memory block where the ith batch is located to obtain an execution result of the nth executive on the ith batch of subdata, executing the self-kernel function computation by taking the execution result of the nth executive as input data of the (n+1)th executive, and writing the execution result into an idle memory block of the (n+1)th executive; then inputting an (i+2)th batch of subdata; S4, executing, by the nth executive, the operation in S2 on the (i+2)th batch of subdata, and executing, by the nth executive and the (n+1)th executive, the operation in S3 on the (i+1)th batch of subdata after the (i+2)th batch of subdata is input; at the same time, sending, by the (n+1)th executive, the address of the memory block where the ith batch is located to an (n+2)th executive; parsing, by the (n+2)th executive, the memory block where the ith batch is located to obtain an execution result of the (n+1)th executive on the ith batch of subdata, executing the self-kernel function computation by taking the execution result of the (n+1)th executive as input data of the (n+2)th executive, and writing the execution result into an idle memory block of the (n+2)th executive; S5, reclaiming, by the nth executive, the memory block sent to the (n+1)th executive; and S6, executing, by the last executive, the self-kernel function computation; writing the execution result to a memory block of the last executive; and reclaiming the memory block on its own immediately at the end of the execution.

2 . The pipelining and parallelizing graph execution method for neural network model computation according to claim 1 , wherein before executing the self-kernel function computation, an executive may check whether there is an idle memory block in the executive, execute the self-kernel function computation on the ith batch of subdata under the condition that there is an idle memory block, and otherwise, instruct the ith batch to wait for an idle memory block.

3 . The pipelining and parallelizing graph execution method for neural network model computation according to claim 2 , wherein for an (N*n+1)th batch of subdata, before executing the self-kernel function computation, the executive may check whether the executive where an (N*n−1)th batch of subdata is located completes execution, wherein n is a positive integer.

4 . The pipelining and parallelizing graph execution method for neural network model computation according to claim 1 , wherein the step S5 specifically comprises the following operations:
S51, informing, by the (n+1)th executive, the nth executive that the memory block sent to the (n+1)th executive has been consumed; S52, reclaiming, by the nth executive, the memory block sent to the (n+1)th executive, and marking the memory block as being free.

5 . The pipelining and parallelizing graph execution method for neural network model computation according to claim 1 , further comprising constructing an executive, wherein the constructing an executive specifically comprises the following substeps:
S01, creating an operator kernel function task queue: adding a current operator kernel function computation task into a current kernel function task queue in sequence; S02, creating a thread of an executive: acquiring, by the thread of the executive, a current task to be processed in sequence from the kernel function task queue, and submitting the current task to be processed to a thread pool; S03, creating an executive of a kernel function: creating an executive used for operator kernel function computation according to a current kernel function task and context information of a current thread, and using the executive to run the kernel function task in the task queue; S04, creating an event recall queue: adding tasks that have been processed by a task executive into an event recall queue; S05, creating a thread of the event recall queue: taking out and returning, by the thread of the event recall queue, the tasks that have been processed in the event recall queue.

6 . A neural network model computation-oriented graph execution apparatus, comprising an executive construction module and an executive pipelining and parallelizing working module, wherein the executive construction module is configured to construct an executive; and the executive pipelining and parallelizing working module is configured to implement the pipelining and parallelizing graph execution method for neural network model computation according to claim 1 .

7 . A neural network model computation-oriented graph execution apparatus, comprising a memory and one or more processors, wherein the memory stores an executable code; and the one or more processors, when executing the executable code, implement the pipelining and parallelizing graph execution method for neural network model computation according to claim 1 .

8 . A computer-readable storage medium on which a program is stored, wherein the program, when executed by a processor, implements the pipelining and parallelizing graph execution method for neural network model computation according to claim 1 .

Assignees

• Zhejiang Lab

Inventors

• Wang Hongsheng
• Tan Bowen
• Bao Hujun
• Chen Guang

Classifications

• G06F9/3885

  using a plurality of independent parallel functional units · CPC title

• G06F16/9024

  Graphs; Linked lists (G06F16/9027 takes precedence) · CPC title

• G06F2209/5017

  Task decomposition · CPC title

• G06F9/4843

  by program, e.g. task dispatcher, supervisor, operating system · CPC title

• G06F9/5022

  Mechanisms to release resources · CPC title