Vector computation unit in a neural network processor

Patent metadata

Field	Value
Publication number	US-2016342889-A1
Application number	US-201514845117-A
Country	US
Kind code	A1
Filing date	Sep 3, 2015
Priority date	May 21, 2015
Publication date	Nov 24, 2016
Grant date	—

Abstract

A circuit for performing neural network computations for a neural network comprising a plurality of layers, the circuit comprising: activation circuitry configured to receive a vector of accumulated values and configured to apply a function to each accumulated value to generate a vector of activation values; and normalization circuitry coupled to the activation circuitry and configured to generate a respective normalized value from each activation value.

First claim

What is claimed is:

1 . A circuit for performing neural network computations for a neural network comprising a plurality of layers, the circuit comprising:
activation circuitry configured to receive a vector of accumulated values and configured to apply a function to each accumulated value to generate a vector of activation values; and normalization circuitry coupled to the activation circuitry and configured to generate a respective normalized value for each activation value.

2 . The circuit of claim 1 , where the activation circuitry receives the vector of accumulated values from a systolic array in the circuit.

3 . The circuit of claim 1 , where the normalization circuitry comprises a plurality of normalization register columns, each normalization register column comprising a plurality of normalization registers connected in series, each normalization register column configured to receive a respective activation value, where the normalization circuitry is configured to form groups around one or more normalization registers, each group corresponding to a normalization unit, and each normalization unit configured to calculate a respective normalized value for the respective activation value.

4 . The circuit of claim 3 , where each normalization register is configured to pass the distinct activation value to an adjacent normalization column.

5 . The circuit of claim 3 , where each group is formed using a normalization radius parameter.

6 . The circuit of claim 3 , where each normalization unit is configured to:
receive the respective activation value; generate a respective intermediate normalized value from the respective activation value; and send the respective intermediate normalized value to one or more neighboring normalization units.

7 . The circuit of claim 6 , where generating the respective intermediate normalized value comprises generating a square of the respective activation value.

8 . The circuit of claim 6 , where each normalization unit is further configured to:
receive, from one or more neighboring normalization units, one or more intermediate normalized values generated from activation values; sum each intermediate normalized value to generate an index; use the index to access one or more values from a lookup table; generate a scaling factor from the one or more values and the index; and generate the respective normalized value from the scaling factor and the respective activation value.

9 . The circuit of claim 1 , further comprising pooling circuitry configured to receive the normalized values and configured to pool the normalized values to generate a pooled value.

10 . The circuit of claim 6 , where the pooling circuitry is configured to store the plurality of normalized values in a plurality of registers and a plurality of memory units,
where the plurality of registers and the plurality of memory units are connected in series, where each register stores one normalized value and each memory unit stores a plurality of normalized values, where the pooling circuitry is configured to, after every clock cycle, shift a given normalized value to a subsequent register or memory unit, and where the pooling circuitry is configured to generate the pooled value from the normalized values.

11 . The circuit of claim 1 , further comprising pooling circuitry configured to receive the activation values and configured to pool the activation values to generate a pooled value.

12 . The circuit of claim 11 , where the pooling circuitry is configured to store the plurality of activation values in a plurality of registers and a plurality of memory units,
where the plurality of registers and the plurality of memory units are connected in series, where each register stores one normalized value and each memory unit stores a plurality of activation values, where the pooling circuitry is configured to, after every clock cycle, shift a given activation value to a subsequent register or memory unit, and where the pooling circuitry is configured to generate the pooled value from the activation values.

Assignees

• Google Inc

Inventors

• Thorson Gregory Michael
• Clark Christopher Aaron
• Luu Dan

Classifications

• G06N3/048

  Activation functions · CPC title

• G06N3/063 Primary

  using electronic means · CPC title

• G06N5/04

  Inference or reasoning models · CPC title

• G06N3/08 Primary

  Learning methods · CPC title

• G06N3/0499

  Feedforward networks · CPC title