Low latency matrix multiply unit

Patent metadata

Field	Value
Publication number	US-10698976-B2
Application number	US-201916529662-A
Country	US
Kind code	B2
Filing date	Aug 1, 2019
Priority date	May 17, 2017
Publication date	Jun 30, 2020
Grant date	Jun 30, 2020

Abstract

Methods, systems, and apparatus for a matrix multiply unit implemented as a systolic array of cells are disclosed. Each cell of the matrix multiply includes: a weight matrix register configured to receive a weight input from either a transposed or a non-transposed weight shift register; a transposed weight shift register configured to receive a weight input from a horizontal direction to be stored in the weight matrix register; a non-transposed weight shift register configured to receive a weight input from a vertical direction to be stored in the weight matrix register; and a multiply unit that is coupled to the weight matrix register and configured to multiply the weight input of the weight matrix register with a vector data input in order to obtain a multiplication result.

First claim

What is claimed is:

1. A matrix multiply unit configured to perform neural network computations of a neural network, the matrix multiply unit implemented as a systolic array of cells, the systolic array of cells arranged in a two-dimensional format, each cell of the array of cells comprising:
a weight matrix register configured to receive a weight input of the neural network from one or more weight storing registers;
the one or more weight storing registers, wherein the one or more weight storing registers are configured to receive weight inputs of the neural network to be stored in the weight matrix register from both a first direction of the two-dimensional format and a second direction of the two-dimensional format, the second direction being different from the first direction; and
a multiply unit that is coupled to the weight matrix register and configured to multiply the weight input of the weight matrix register with a vector data input of the neural network in order to obtain a multiplication result.

2. The matrix multiply unit of claim 1 , wherein each cell further comprises:
a multiplexer configured to:
select the weight input from the weight inputs received from the first direction and the second direction; and
send the selected weight input to the weight matrix register.

3. The matrix multiply unit of claim 1 , further comprising a first weight holding register configured to hold the weight input from the first direction.

4. The matrix multiply unit of claim 3 , further comprising a second weight holding register configured to hold the weight input from the second direction.

5. The matrix multiply unit of claim 4 , wherein the weight input from the first direction is loaded from the one or more weight storing registers into the first weight holding register, wherein the weight input from the second direction is loaded from the one or more weight storing registers into the second weight holding register.

6. The matrix multiply unit of claim 5 , wherein the weight matrix register is loaded with the one of the weight input from the first direction and the weight input from the second direction.

7. The matrix multiply unit of claim 6 , wherein the weight input in the weight matrix register is used in any number of cycles of multiplications.

8. The matrix multiply unit of claim 7 , wherein during the number of cycles of multiplications, additional weight inputs are shifted into the one or more weight storing registers in preparation for a subsequent set of one or more multiplications.

9. The matrix multiply unit of claim 7 , wherein during the number of cycles of multiplications, another weight input of the neural network is multiplied with another vector data input of the neural network to obtain another multiplication result.

10. The matrix multiply unit of claim 1 , wherein the vector data input moves by one multi-cell per clock cycle.

11. The matrix multiply unit of claim 1 , wherein the one or more weight storing registers comprise a transposed weight shift register and a non-transposed weight shift register that is physically separate from the transposed weight shift register.

12. The matrix multiply unit of claim 1 , wherein the one or more weight storing registers that are configured to receive the weights comprise:
a first weight storing register configured to receive a first weight input over a first wired path from another cell in the array that is along the first direction; and
a second weight storing register configured to receive a second weight input over a second wired path from another cell in the array that is along the second direction, wherein the weight input is at least one of the first weight input or the second weight input.

Assignees

• Google Llc

Inventors

• Phelps Andrew Everett
• Jouppi Norman Paul

Classifications

• G06N3/08

  Learning methods · CPC title

• G06N3/04

  Architecture, e.g. interconnection topology · CPC title

• G06F9/30036

  Instructions to perform operations on packed data, e.g. vector, tile or matrix operations · CPC title

• G06N3/063

  using electronic means · CPC title

• G06F7/5443

  Sum of products (for applications thereof, see the relevant places, e.g. G06F17/10, H03H17/00) · CPC title