Numerical representation for neural networks

Patent metadata

Field	Value
Publication number	US-11062202-B2
Application number	US-201917262717-A
Country	US
Kind code	B2
Filing date	Jul 17, 2019
Priority date	Jul 25, 2018
Publication date	Jul 13, 2021
Grant date	Jul 13, 2021

Abstract

Techniques in advanced deep learning provide improvements in one or more of accuracy, performance, and energy efficiency. An array of processing elements comprising a portion of a neural network accelerator performs flow-based computations on wavelets of data. Each processing element has a respective compute element and a respective routing element. Each compute element has a respective floating-point unit enabled to optionally and/or selectively perform floating-point operations in accordance with a programmable exponent bias and/or various floating-point computation variations. In some circumstances, the programmable exponent bias and/or the floating-point computation variations enable neural network processing with improved accuracy, decreased training time, decreased inference latency, and/or increased energy efficiency.

First claim

What is claimed is:

1. An apparatus comprising:
a programmable processor enabled to execute instructions, the instructions comprising floating-point instructions, and first and second load control resource instructions;
a first control resource of the programmable processor enabled to receive an exponent bias responsive to an operation corresponding to the first load control resource instruction;
a floating-point unit of the programmable processor enabled to perform floating-point operations corresponding to the floating-point instructions, the performing floating-point operations being in accordance with floating-point operands and floating-point results each comprising a respective biased exponent, the performing floating-point operations being responsive to the exponent bias, and compatible, according to the exponent bias, with interpreting the biased exponents of the floating-point operands and producing the biased exponents of the floating-point results; and
wherein
the apparatus further comprises a second control resource of the programmable processor enabled to receive an exponent size specifier responsive to an operation corresponding to the second load control resource instruction,
the floating-point unit is further enabled to perform the floating-point operations responsive to the exponent size specifier specifying one of a plurality of floating-point formats, and a first of the plurality of floating-point formats comprises a biased exponent of N bits and a mantissa of M bits, and a second of the plurality of floating-point formats comprises a biased exponent of (N+delta) bits and a mantissa of (M-delta) bits, and
the performing floating-point operations is further compatible, according to the exponent size specifier, with interpreting the biased exponents of the floating-point operands and producing the biased exponents of the floating-point results.

2. The apparatus of claim 1 , wherein the programmable processor is one of a plurality of like programmable processors fabricated on a wafer in accordance with wafer-scale integration, and a datacenter element enabled to perform neural network processing comprises the wafer.

3. The apparatus of claim 1 , wherein the programmable processor comprises one or more hardware registers comprising at least one of the first and the second control resources, and at least one of the first and the second load control resource instructions is a load hardware register instruction.

4. The apparatus of claim 1 , wherein at least one of the first and the second control resources is memory-mapped, and at least one of the first and the second load control resource instructions is a memory store instruction.

5. The apparatus of claim 1 , wherein:
the instructions further comprise a third load control resource instruction,
the apparatus further comprises a third control resource of the programmable processor enabled to receive a rounding mode specifier responsive to an operation corresponding to the third load control resource instruction, and
the floating-point unit is further enabled to perform the floating-point operations in accordance with one of a plurality of rounding modes specified by the rounding mode specifier, and at least one of the rounding modes comprises saturating the floating-point results to a value comprising a maximum biased exponent.

6. The apparatus of claim 1 , wherein:
the instructions further comprise a third load control resource instruction,
the apparatus further comprises a third control resource of the programmable processor enabled to receive a flush-to-zero specifier responsive to an operation corresponding to the third load control resource instruction, and
the floating-point unit is further enabled to perform the floating-point operations responsive to the flush-to-zero specifier specifying a flush-to-zero mode, and the flush-to-zero mode comprises flushing subnormal results to zero.

7. An apparatus comprising:
a programmable processor enabled to execute instructions, the instructions comprising floating-point instructions, and first and second load control resource instructions;
a first control resource of the programmable processor enabled to receive an exponent bias responsive to an operation corresponding to the first load control resource instruction;
a floating-point unit of the programmable processor enabled to perform floating-point operations corresponding to the floating-point instructions, the performing floating-point operations being in accordance with floating-point operands and floating-point results each comprising a respective biased exponent, the performing floating-point operations being responsive to the exponent bias, and compatible, according to the exponent bias, with interpreting the biased exponents of the floating-point operands and producing the biased exponents of the floating-point results; and
wherein
the apparatus further comprises a second control resource of the programmable processor enabled to receive a maximum biased exponent normal specifier responsive to an operation corresponding to the second load control resource instruction,
the floating-point unit is further enabled to perform the floating-point operations responsive to the maximum biased exponent normal specifier specifying one of a plurality of floating-point modes, and a first of the plurality of floating-point modes comprises using a maximum biased exponent to represent a biased exponent, and
the performing floating-point operations is further compatible, according to the maximum biased exponent normal specifier, with interpreting the biased exponents of the floating-point operands and producing the biased exponents of the floating-point results.

8. The apparatus of claim 7 , wherein:
the instructions further comprise a third load control resource instruction,
the apparatus further comprises a third control resource of the programmable processor enabled to receive a rounding mode specifier responsive to an operation corresponding to the third load control resource instruction, and
the floating-point unit is further enabled to perform the floating-point operations in accordance with one of a plurality of rounding modes specified by the rounding mode specifier, and at least one of the rounding modes comprises saturating the floating-point results to a value comprising a maximum biased exponent.

9. The apparatus of claim 7 , wherein a second of the plurality of floating-point modes comprises using the maximum biased exponent to represent at least one of: infinite numbers and IEEE compatible NaN.

10. The apparatus of claim 7 , wherein the programmable processor is one of a plurality of like programmable processors fabricated on a wafer in accordance with wafer-scale integration, and a datacenter element enabled to perform neural network processing comprises the wafer.

11. The apparatus of claim 7 , wherein the programmable processor comprises one or more hardware registers comprising at least one of the first and the second control resources, and at least one of the first and the second load control resource instructions is a load hardware register instruction.

12. The apparatus of claim 7 , wherein at least one of the first and the second control resources is memory-mapped, and at least one of the first and the second load control resource instructions is a memory store instruction.

13. An apparatus comprising:
a programmable processor enabled to execute instructions, the instructions comprising floating-point instructions, and first and second load control resource instructions;
a first control resource of the programmable processor enabled to receive an exponent bias re

Assignees

• Cerebras Systems Inc

Inventors

• James Michael Edwin
• Lie Sean
• Morrison Michael
• Arekapudi Srikanth
• Lauterbach Gary R

Classifications

• G06N3/045

  Combinations of networks · CPC title

• G06N3/044

  Recurrent networks, e.g. Hopfield networks · CPC title

• G06N3/0464

  Convolutional networks [CNN, ConvNet] · CPC title

• G06N3/09

  Supervised learning · CPC title

• G06N3/0495

  Quantised networks; Sparse networks; Compressed networks · CPC title