Distributed processing architecture

Patent metadata

Field	Value
Publication number	US-11076210-B1
Application number	US-202016883814-A
Country	US
Kind code	B1
Filing date	May 26, 2020
Priority date	Jan 3, 2020
Publication date	Jul 27, 2021
Grant date	Jul 27, 2021

Abstract

Embodiments of the present disclosure include techniques for processing neural networks. Various forms of parallelism may be implemented using topology that combines sequences of processors. In one embodiment, the present disclosure includes a computer system comprising one or more processor groups, the processor groups each comprising a plurality of processors. A plurality of network switches are coupled to subsets of the plurality of processor groups. In one embodiment, the switches may be optical network switches. Processors in the processor groups may be configurable to form sequences, and the network switches are configurable to form at least one sequence across one or more of the plurality of processor groups to perform neural network computations. Various alternative configurations for creating Hamiltonian cycles are disclosed to support data parallelism, pipeline parallelism, layer parallelism, or combinations thereof.

First claim

What is claimed is:

1. A computer system comprising:
a plurality of processor groups, the processor groups comprising a plurality of series configured processors to process a partitioned neural network, wherein different processors in the plurality of processor groups process different layers, stages, or model instances of the partitioned neural network; and
one or more optical network switches coupled to the plurality of processor groups through edge processors of the series configured processors in each processor group,
wherein at least a subset of the processors in the processor groups are configured to form sequences such that at least a portion of the processors communicate data for said layers, stages, or model instances with at least two adjacent processor in the sequence, and
wherein the optical network switches are configured to form at least one sequence of processors across one or more of the plurality of processor groups to perform neural network computations.

2. The computer system of claim 1 wherein the one or more optical network switches maintain a channel between the one or more processor groups to form said at least one sequence, the channel having an associated minimum dedicated bandwidth to maintain the channel between the processors in said at least one sequence across the one or more processor groups.

3. The computer system of claim 1 wherein processors in edge positions of the processor groups are coupled to the plurality of optical network switches over an optical link.

4. The computer system of claim 3 wherein edge processors in the processor groups are coupled to at least one optical module.

5. The computer system of claim 3 wherein edge processor ports on opposite sides of a row of processors are coupled to a first optical network switch.

6. The computer system of claim 5 wherein edge processor ports on opposite sides of a column of processors are coupled to second optical network switch.

7. The computer system of claim 1 wherein the processor groups are coupled to a plurality of row optical network switches and a plurality of column optical network switches.

8. The computer system of claim 1 wherein the processor groups are configured in a multi-dimensional cluster array.

9. The computer system of claim 8 wherein processor groups along a particular dimension of the cluster array have edge processor ports coupled to corresponding same dimension optical network switches.

10. The computer system of claim 9 wherein each optical network switch alone one dimension is coupled to at least two optical network switches along another dimension.

11. The computer system of claim 1 wherein the one or more optical network switches are a plurality of optical network switches coupled together over two or more optical links.

12. The computer system of claim 11 wherein two or more of the optical network switches are directly coupled together over the two or more optical links.

13. The computer system of claim 11 wherein at least two of the optical network switches are coupled together through at least two processor groups, wherein a first processor group is coupled to a first optical network switch over a first optical link, the first processor group is coupled to a second optical network switch over a second optical link, a second processor group is coupled to the first optical network switch over a third optical link, the second processor group is coupled to the second optical network switch over a fourth optical link.

14. The computer system of claim 1 wherein the plurality of optical network switches are coupled to at least one intermediate optical network switch to couple processors in at least a subset of the plurality of processor groups in series.

15. The computer system of claim 1 wherein processors configured to form the sequence are configured to process a plurality of partial layers of the partitioned neural network.

16. The computer system of claim 1 wherein processors configured to form the sequence are configured to process one or more pipeline stages of the partitioned neural network.

17. The computer system of claim 1 wherein processors configured to form the sequence are configured to adjust weights across a plurality of model instances of the neural network.

18. A method for processing a neural network, the method comprising:
configuring a plurality of processors arranged in a plurality of processor groups to perform neural network computations, wherein the plurality processors are configured in series to process a partitioned neural network, wherein different processors in the plurality of processor groups process different layers, stages, or model instances of the partitioned neural network, and wherein at least a subset of the processors in the processor groups are configured to form sequences of processors;
configuring a plurality of optical network switches to coupled subsets of the plurality of processor groups together to form at least one sequence of processors across one or more processor groups through edge processors of the series configured processors in each processor group such that at least a portion of the processors communicate data for said layers, stages, or model instances with at least two adjacent processor in the sequence; and
performing the neural network computations by the at least one sequence of processors.

Assignees

• Microsoft Technology Licensing Llc

Inventors

• Hoefler Torsten
• Heddes Mattheus C
• Belk Jonathan R

Classifications

• H04Q11/0067 Primary

  Provisions for optical access or distribution networks, e.g. Gigabit Ethernet Passive Optical Network (GE-PON), ATM-based Passive Optical Network (A-PON), PON-Ring · CPC title

• G06N3/063

  using electronic means · CPC title

• G06N3/02

  Neural networks · CPC title

• H04Q2011/0079

  Operation or maintenance aspects · CPC title

• G06F15/17387 Primary

  Three dimensional, e.g. hypercubes · CPC title