Depth concatenation using a matrix computation unit

Patent metadata

Field	Value
Publication number	US-10699182-B2
Application number	US-201916531774-A
Country	US
Kind code	B2
Filing date	Aug 5, 2019
Priority date	Mar 7, 2017
Publication date	Jun 30, 2020
Grant date	Jun 30, 2020

Abstract

Methods, systems, and apparatus, including computer programs encoded on computer storage media, for depth concatenation using a matrix computation unit. One of the methods includes: receiving a request to process network inputs to a neural network using an integrated circuit, the neural network comprising a depth concatenation neural network layer; and generating instructions that, when executed by the integrated circuit, cause the integrated circuit to perform operations comprising: for each spatial location in a first input tensor to the depth concatenation layer and a second input tensor to the depth concatenation layer: multiplying, using the matrix computation unit, a second depth vector for the spatial location by a shift weight matrix for the depth concatenation layer to generate a shifted second depth vector; and adding the shifted second depth vector and a first input depth vector for the spatial location to generate a concatenated depth vector.

First claim

What is claimed is:

1. A method comprising:
receiving a request to process network inputs to a neural network using an integrated circuit that performs neural network computations in hardware using a matrix computation unit, the neural network comprising a depth concatenation neural network layer that specifies a concatenation of two input tensors along a depth dimension, wherein the two input tensors each (i) have a plurality of dimensions that includes the depth dimension and (ii) have the same number of values along all of the plurality of dimensions that are not the depth dimension; and
generating instructions that, when executed by the integrated circuit, cause the integrated circuit to, during processing of a network input by the neural network, generate a layer output tensor that satisfies the specification of the depth concatenation neural network layer by performing operations comprising:
for each spatial location in a first input tensor to the depth concatenation layer and a second input tensor to the depth concatenation layer, wherein the first input tensor has z 1 values along the depth dimension and the second input tensor has z 2 values along the depth dimension, and wherein each spatial location is a combination of coordinates that includes a respective coordinate for each of the plurality of dimensions that are not the depth dimension:
multiplying, using the matrix computation unit, a second depth vector for the spatial location in the second input tensor by a shift weight matrix for the depth concatenation layer to generate a shifted second depth vector, the shift weight matrix being a (z 1 +z 2 ) by (z 1 +z 2 ) matrix, wherein the second depth vector for the spatial location includes as its first z 2 entries the first z 2 entries in the second input tensor that have the combination of coordinates for the spatial location and as its remaining z 1 entries zeroes or garbage data, and wherein the shifted second depth vector has zeroes as its first z 1 entries and entries of the second depth vector as its remaining z 2 entries; and
adding the shifted second depth vector and a first input depth vector for the spatial location in the first input tensor to generate a concatenated depth vector, the first input depth vector having as its first z 1 entries the first z 1 entries in the first input tensor that have the combination of coordinates for the spatial location and as its remaining z 2 entries zeroes or garbage data.

2. The method of claim 1 , the operations further comprising:
moving the first input depth vector to a set of output sum-in registers of the matrix computation unit; and
wherein adding the shifted second depth vector and the first input depth vector comprises:
moving the shifted second depth vector into the set of output sum-in registers of the matrix computation unit while the first input depth vector is stored in the set of output sum-in registers of the matrix computation unit.

3. The method of claim 2 , wherein moving the first input depth vector comprises:
multiplying the first input depth vector by a modified identity weight matrix for the depth concatenation layer using the matrix computation unit.

4. The method of claim 3 , further comprising:
generating the modified identity weight matrix for the depth concatenation layer; and
storing the modified identity weight matrix for the depth concatenation layer in a memory accessible to the integrated circuit.

5. The method of claim 1 , further comprising:
generating the shift weight matrix for the depth concatenation layer; and
storing the shift weight matrix for the depth concatenation layer in a memory accessible to the integrated circuit.

6. The method of claim 5 , further comprising:
determining that the number of entries along the depth dimension in the output tensor does not exceed a maximum vector length for the matrix computation unit; and
generating the shift weight matrix for the depth concatenation layer in response to determining that the number of depth dimensions in the output tensor does not exceed the maximum vector length for the matrix computation unit.

7. The method of claim 1 , wherein the shift weight matrix for the depth concatenation layer is a matrix having all entries be zero except for a diagonal row of ones starting at the first entry of the z 2 -th column of the matrix.

8. A system comprising one or more computers and one or more storage devices storing first instructions that when executed by the one or more computers cause the one or more computers to perform first operations comprising: receiving a request to process network inputs to a neural network using an integrated circuit that performs neural network computations in hardware using a matrix computation unit, the neural network comprising a depth concatenation neural network layer that specifies a concatenation of two input tensors along a depth dimension, wherein the two input tensors each (i) have a plurality of dimensions that includes the depth dimension and (ii) have the same number of values along all of the plurality of dimensions that are not the depth dimension; and
generating second instructions that, when executed by the integrated circuit, cause the integrated circuit to, during processing of a network input by the neural network, generate a layer output tensor that satisfies the specification of the depth concatenation neural network layer by performing second operations comprising:
for each spatial location in a first input tensor to the depth concatenation layer and a second input tensor to the depth concatenation layer, wherein the first input tensor has z 1 values along the depth dimension and the second input tensor has z 2 values along the depth dimension, and wherein each spatial location is a combination of coordinates that includes a respective coordinate for each of the plurality of dimensions that are not the depth dimension:
multiplying, using the matrix computation unit, a second depth vector for the spatial location in the second input tensor by a shift weight matrix for the depth concatenation layer to generate a shifted second depth vector, the shift weight matrix being a (z 1 +z 2 ) by (z 1 +z 2 ) matrix, wherein the second depth vector for the spatial location includes as its first z 2 entries the first z 2 entries in the second input tensor that have the combination of coordinates for the spatial location and as its remaining z 1 entries zeroes or garbage data, and wherein the shifted second depth vector has zeroes as its the first z 1 entries and entries of the second depth vector as the its remaining last z 2 entries; and
adding the shifted second depth vector and a first input depth vector for the spatial location in the first input tensor to generate a concatenated depth vector, the first input depth vector having as its first z 1 entries the first z 1 entries in the first input tensor that have the combination of coordinates for the spatial location and as its remaining z 2 entries zeroes or garbage data.

9. The system of claim 8 , the second operations further comprising:
moving the first input depth vector to a set of output sum-in registers of the matrix computation unit; and
wherein adding the shifted second depth vector and the first input depth vector comprises:
moving the shifted second depth vector into the set of output sum-in registers of the matrix computation unit while the first input depth vector is stored in the set of output sum-in registers of the matrix computation unit.

10. The system of claim 9 , wherein moving the first input depth vector comprises:
multiplying the first input depth vector by a modified identity weight matrix for the depth concatenation

Assignees

• Google Llc

Inventors

• Gulland William John
• Young Reginald Clifford

Classifications

• G06N3/065

  Analogue means · CPC title

• G06N3/045

  Combinations of networks · CPC title

• G06N3/063 Primary

  using electronic means · CPC title

• G06N3/0464

  Convolutional networks [CNN, ConvNet] · CPC title

• G06N3/0442

  characterised by memory or gating, e.g. long short-term memory [LSTM] or gated recurrent units [GRU] · CPC title