Low latency matrix multiply unit

Patent metadata

Field	Value
Publication number	US-10635740-B2
Application number	US-201815983043-A
Country	US
Kind code	B2
Filing date	May 17, 2018
Priority date	May 17, 2017
Publication date	Apr 28, 2020
Grant date	Apr 28, 2020

Abstract

Methods, systems, and apparatus for a matrix multiply unit implemented as a systolic array of cells are disclosed. The matrix multiply unit may include cells arranged in columns of the systolic array. Two chains of weight shift registers per column of the systolic array are in the matrix multiply unit. Each weight shift register is connected to only one chain and each cell is connected to only one weight shift register. A weight matrix register per cell is configured to store a weight input received from a weight shift register. A multiply unit is coupled to the weight matrix register and configured to multiply the weight input of the weight matrix register with a vector data input in order to obtain a multiplication result.

First claim

What is claimed is:

1. A matrix multiply unit implemented as a two-dimensional systolic array comprising:
a plurality of cells arranged in columns of the systolic array, wherein each cell of the plurality of cells includes a weight shift register of a plurality of weight shift registers in each column of the columns, a weight matrix register of a plurality of weight matrix registers in each column of the columns, and a multiply unit of a plurality of multiply units in each column of the columns;
two shift chains for each column of the columns, the two shift chains for each column being wired paths over which weight inputs of a neural network are transmitted to the plurality of weight shift registers in the column,
wherein, for each cell in each column of the columns:
the weight shift register is connected to and configured to receive a corresponding weight input of the weight inputs from only one shift chain of the two shift chains;
the weight matrix register is configured to receive the corresponding weight input from the weight shift register and store the corresponding weight input; and
the multiply unit is coupled to and configured to receive the corresponding weight input from the weight matrix register, the multiply unit configured to multiply the corresponding weight input with a vector data input to determine a multiplication result.

2. The matrix multiply unit of claim 1 , wherein the weight inputs are transmitted in pairs via the two shift chains from a vector register containing the weight inputs.

3. The matrix multiply unit of claim 2 , wherein when the two weight inputs are available at the vector register, the two weight inputs are shifted on a clock cycle to corresponding weight shift registers of the plurality of weight shift registers.

4. The matrix multiply unit of claim 2 , further comprising a holding register for each column of the columns, the holding register configured to hold and provide the corresponding weight input when the weight inputs are unavailable from the vector register.

5. The matrix multiply unit of claim 4 , wherein when the two weight inputs are unavailable at the vector register:
on a first clock cycle when a first weight input becomes available, the holding register is loaded with the first weight input as a held value that is not transmitted via the two shift chains at the first clock cycle; and
on a subsequent clock cycle, when a second weight input becomes available, the second weight input and the held value are transmitted over the two shift chains to corresponding weight shift registers, each shift chain transmitting a corresponding held value.

6. The matrix multiply unit of claim 1 , wherein:
each shift chain has two injection points for injecting the weight inputs, the two injection points comprising a first point at the top of the column and a second point at another location in the column.

7. The matrix multiply unit of claim 6 , further comprising:
a vector register configured to provide the vector input data, the vector register configured to contain packed sets of four 8-bit integers each representing a separate weight input.

8. The matrix multiply unit of claim 7 , wherein:
two of the four integers are injected at the first point; and
the other two of the four integers are injected at the second point.

9. The matrix multiply unit of claim 1 , wherein when the weight inputs are in the weight matrix register, the weight inputs are used in a plurality of multiplication cycles.

10. The matrix multiply unit of claim 9 , wherein during the plurality of multiplication cycles, additional weight inputs are transmitted into the weight shift registers in preparation for a next set of one or more multiplications.

11. The matrix multiply unit of claim 9 , wherein during the plurality of multiplication cycles, another corresponding weight input received from the weight matrix register is multiplied with another vector data input to determine another multiplication result.

12. The matrix multiply unit of claim 1 , wherein the vector data input moves by one multi-cell per clock cycle.

13. The matrix multiply unit of claim 1 , wherein the weight values are transmitted based on instructions when the instructions are received.

Assignees

• Google Llc

Inventors

• Phelps Andrew Everett
• Jouppi Norman Paul

Classifications

• G06N3/08

  Learning methods · CPC title

• G06N3/04

  Architecture, e.g. interconnection topology · CPC title

• G06F5/015

  having at least two separately controlled shifting levels, e.g. using shifting matrices (G06F5/012 takes precedence) · CPC title

• G06F17/16 Primary

  Matrix or vector computation {, e.g. matrix-matrix or matrix-vector multiplication, matrix factorization (matrix transposition G06F7/78)} · CPC title

• G06F15/8046

  Systolic arrays · CPC title