Training neural networks based on dual pipeline architectures

Patent metadata

Field	Value
Publication number	US-12511516-B2
Application number	US-202117151007-A
Country	US
Kind code	B2
Filing date	Jan 15, 2021
Priority date	Oct 29, 2020
Publication date	Dec 30, 2025
Grant date	Dec 30, 2025

Abstract

Embodiments of the present disclosure include systems and methods for training neural networks based on dual pipeline architectures. In some embodiments, a first set of compute elements are configured to implement a first set of layers of a first instance of a neural network. A second set of compute elements are configured to implement a second set of layers of the first instance of the neural network. The second set of compute elements are further configured to implement a first set of layers of a second instance of the neural network. The first set of compute elements are further configured to implement a second set of layers of the second instance of the neural network. The first set of layers of the first instance of the neural network and the first set of layers of the second instance of the neural network are each configured to receive training data.

First claim

What is claimed is:

1 . A system comprising:
a first set of computing hardware configured to implement a first set of layers including an input layer of a first instance of a neural network; and a second set of computing hardware configured to implement a second set of layers including an output layer of the first instance of the neural network; wherein the second set of computing hardware are further configured to implement a first set of layers including an input layer of a second instance of the neural network, the second instance being a separate instance of the neural network than the first instance; wherein the first set of computing hardware are further configured to implement a second set of layers including an output layer of the second instance of the neural network; wherein the first set of layers of the first instance of the neural network is configured to receive a first set of training data and process the first set of training data from the first set of layers to the second set of layers of the first instance of the neural network in a first feedforward pass, wherein the first set of layers of the second instance of the neural network is configured to receive a second set of training data and process the second set of training data from the first set of layers to the second set of layers of the second instance of the neural network in a second feedforward pass.

2 . The system of claim 1 , wherein the first instance of the neural network is configured to process the first set of training data in the first feedforward pass in parallel with the second instance of the neural network processing the second set of training data in the second feedforward pass.

3 . The system of claim 1 , wherein the first set of layers of the first instance of the neural network is further configured to receive a first set of training data during a feedforward stage of training the first instance of the neural network and store the first set of training data for processing during a backpropagation stage of training the first instance of the neural network, wherein the first set of layers of the second instance of the neural network is further configured to receive a second set of training data during a feedforward stage of training the second instance of the neural network and store the second set of training data for processing during a backpropagation stage of training the second instance of the neural network.

4 . The system of claim 1 , wherein configuring the first set of computing hardware to implement the first set of layers of the first instance of the neural network and the second set of layers of the second instance of the neural network and configuring the second set of computing hardware to implement the second set of layers of the first instance of the neural network and the first set of layers of the second instance of the neural network improves the efficiency of training the first and second instances of the neural network.

5 . The system of claim 1 , wherein configuring the first set of computing hardware to implement the first set of layers of the first instance of the neural network and the second set of layers of the second instance of the neural network and configuring the second set of computing hardware to implement the second set of layers of the first instance of the neural network and the first set of layers of the second instance of the neural network improves memory utilization across the first and second sets of computing hardware.

6 . The system of claim 1 , wherein the first set of layers of the first instance of the neural network is configured to receive a set of training data every n cycles, wherein n is the number of cycles the first set of layers of the first instance of the neural network uses to process a feedforward pass and a backpropagation pass.

7 . A method comprising:
configuring a first set of computing hardware to implement a first set of layers including an input layer of a first instance of a neural network; configuring a second set of computing hardware to implement a second set of layers including an output layer of the first instance of the neural network; configuring the second set of computing hardware to implement a first set of layers including an input layer of a second instance of the neural network, the second instance being a an instance of the neural network that is independent of the first instance; and configuring the first set of computing hardware to implement a second set of layers including an output layer of the second instance of the neural network, wherein the first set of layers of the first instance of the neural network is configured to receive a first set of training data and process the first set of training data from the first set of layers to the second set of layers of the first instance of the neural network in a first feedforward pass, wherein the first set of layers of the second instance of the neural network is configured to receive a second set of training data and process the second set of training data from the first set of layers to the second set of layers of the second instance of the neural network in a second feedforward pass.

8 . The method of claim 7 , wherein the first instance of the neural network and the second instance of the neural network are configured to process training data in parallel.

9 . The method of claim 7 further comprising:
configuring the first set of layers of the first instance of the neural network to receive a first set of training data during a feedforward stage of training the first instance of the neural network;
configuring the first set of layers of the first instance of the neural network to store the first set of training data for processing during a backpropagation stage of training the first instance of the neural network
configuring the first set of layers of the second instance of the neural network to receive a second set of training data during a feedforward stage of training the second instance of the neural network; and
configuring the first set of layers of the second instance of the neural network to store the second set of training data for processing during a backpropagation stage of training the second instance of the neural network.

10 . The method of claim 7 , wherein configuring the first set of computing hardware to implement the first set of layers of the first instance of the neural network and the second set of layers of the second instance of the neural network and configuring the second set of computing hardware to implement the second set of layers of the first instance of the neural network and the first set of layers of the second instance of the neural network improves the efficiency of training the first and second instances of the neural network.

11 . The method of claim 7 , wherein configuring the first set of compute elements to implement the first set of layers of the first instance of the neural network and the second set of layers of the second instance of the neural network and configuring the second set of compute elements to implement the second set of layers of the first instance of the neural network and the first set of layers of the second instance of the neural network improves memory utilization across the first and second sets of compute elements.

12 . A system comprising:
a first pipeline for training a first instance of a neural network, the first pipeline comprising a first artificial intelligence (AI) accelerator configured to implement an input layer of the first instance of the neural network and a second AI accelerator configured to implement an output layer of the first instance of the neu

Assignees

• Microsoft Technology Licensing Llc

Inventors

• Heddes Mattheus
• Hoefler Torsten
• Colwell Kenneth Andrew
• Phanishayee Amar

Classifications

• G06N3/084

  Backpropagation, e.g. using gradient descent · CPC title

• G06N3/098

  Distributed learning, e.g. federated learning · CPC title

• G06N3/09

  Supervised learning · CPC title

• G06N3/0499

  Feedforward networks · CPC title

• G06N3/045

  Combinations of networks · CPC title