Vector computation unit in a neural network processor

Patent metadata

Field	Value
Publication number	US-10192162-B2
Application number	US-201514845117-A
Country	US
Kind code	B2
Filing date	Sep 3, 2015
Priority date	May 21, 2015
Publication date	Jan 29, 2019
Grant date	Jan 29, 2019

Abstract

A circuit for performing neural network computations for a neural network comprising a plurality of layers, the circuit comprising: activation circuitry configured to receive a vector of accumulated values and configured to apply a function to each accumulated value to generate a vector of activation values; and normalization circuitry coupled to the activation circuitry and configured to generate a respective normalized value from each activation value.

First claim

What is claimed is:

1. A system for performing neural network computations for a neural network having a plurality of neural network layers, the system comprising:
normalization circuitry comprising:
M sets of normalization registers, wherein M is a positive integer greater than one, and wherein each set of normalization registers comprises multiple normalization registers, each normalization register configured to:
receive, in a staggered manner over multiple clock cycles, a set of activated values for a neural network layer; and
store subsets of the activated values in the normalization register; and

multiple normalization units, each normalization unit communicatively coupled to multiple sets of normalization registers and each normalization unit configured to:
obtain, from two or more sets of the normalization registers, the subsets of activated values stored in the normalization registers; and
normalize the subsets of activated values obtained from the two or more sets of normalization registers to generate a normalized value for determining an activation input for a subsequent neural network layer of the neural network.

2. The system of claim 1 , further comprising:
activation circuitry configured to:
receive multiple input vectors, wherein each input vector includes multiple accumulated values; and
determine, based on the multiple input vectors, multiple output vectors, wherein each output vector includes multiple activated values.

3. The system of claim 2 , wherein the activation circuitry is configured to determine the set of activated values based on at least one output vector of the multiple output vectors.

4. The system of claim 1 , wherein at least two of the normalization registers are connected in series, such that an activated value provided by a first normalization register of the multiple normalization registers is received by a second normalization register of the multiple normalization registers.

5. The system of claim 1 , wherein the set of activated values is represented by data that includes multiple activated values of the set of activated values.

6. The system of claim 1 , wherein the set of activated values are represented by data that includes a square of multiple activated values of the set of activated values.

7. The system of claim 1 , wherein normalizing the subsets of activated values to generate the normalized value comprises determining a sum of the activated values in the subsets.

8. The system of claim 7 , wherein normalizing the subsets of activated values to generate the normalized value comprises:
determining a multiplication factor based on the sum of the activated values in the subsets; and
multiplying an activation value by the multiplication factor to determine the normalized value.

9. The system of claim 1 , wherein normalizing the subsets of activated values to generate the normalized value comprises normalizing the subsets of activated values using a lookup table.

10. The system of claim 1 , wherein a normalization unit of the multiple normalization units is configured to provide the normalized value to another normalization unit of the multiple normalization units.

11. The system of claim 1 , wherein the system further comprises pooling circuitry configured to aggregate multiple normalized values to generate a pooled value.

12. The system of claim 11 , wherein aggregating the multiple normalized values comprises applying an aggregation function to the multiple normalized values.

13. The system of claim 11 , wherein the pooling circuitry comprises multiple registers and multiple memory units connected in series.

14. A method for performing neural network computations for a neural network having a plurality of neural network layers using normalization circuitry that includes M sets of normalization registers and multiple normalization units, each normalization unit communicatively coupled to multiple sets of normalization registers, wherein M is a positive integer greater than one, and wherein each set of normalization registers include multiple normalization registers, the method comprising:
receiving, by each set of the normalization registers in a staggered manner over multiple clock cycles, a set of activated values for a neural network layer; and
storing, by each set of the normalization registers, subsets of the activated values in the normalization registers of the set; and
obtaining, by each normalization unit of the multiple normalization units and from two or more sets of the normalization registers, the subsets of activated values stored in the normalization registers; and
normalizing, by each normalization unit of the multiple normalization units, the subsets of activated values obtained from the two or more sets of normalization registers to generate a normalized value for determining an activation input for a subsequent neural network layer of the neural network.

15. The method of claim 14 , wherein the set of activated values is represented by data that includes multiple activated values of the set of activated values.

16. The method of claim 14 , wherein the set of activated values is represented by data that includes a square of multiple activated values of the set of activated values.

17. The method of claim 14 , wherein normalizing the subsets of activated values to generate the normalized value comprises determining a sum of the activated values in the subsets.

18. A normalization circuitry comprising:
M sets of normalization registers, wherein M is a positive integer greater than one, and wherein each set of normalization registers comprises multiple normalization registers, each normalization register configured to:
receive, in a staggered manner over multiple clock cycles, a set of activated values for a neural network layer; and
store subsets of the activated values in the normalization register; and

multiple normalization units, each normalization unit communicatively coupled to multiple sets of normalization registers and each normalization unit configured to:
obtain, from two or more sets of the normalization registers, the subsets of activated values stored in the normalization registers; and
normalize the subsets of activated values obtained from the two or more sets of normalization registers to generate a normalized value for determining an activation input for a subsequent neural network layer of the neural network.

19. The normalization circuitry of claim 18 , wherein at least two of the normalization registers are connected in series, such that an activated value provided by a first normalization register of the multiple normalization registers is received by a second normalization register of the multiple normalization registers.

20. The normalization circuitry of claim 18 , wherein normalizing the subsets of activated values to generate the normalized value comprises determining a sum of the activated values in the subsets.

Assignees

• Google Llc

Inventors

• Thorson Gregory Michael
• Clark Christopher Aaron
• Luu Dan

Classifications

• G06N3/048

  Activation functions · CPC title

• G06N5/04

  Inference or reasoning models · CPC title

• G06N3/063 Primary

  using electronic means · CPC title

• G06N3/08 Primary

  Learning methods · CPC title

• G06N3/0499

  Feedforward networks · CPC title