Artificial neural network training using flexible floating point tensors

Patent metadata

Field	Value
Publication number	US-12205035-B2
Application number	US-201816004243-A
Country	US
Kind code	B2
Filing date	Jun 8, 2018
Priority date	Jun 8, 2018
Publication date	Jan 21, 2025
Grant date	Jan 21, 2025

Abstract

Thus, the present disclosure is directed to systems and methods for training neural networks using a tensor that includes a plurality of FP16 values and a plurality of bits that define an exponent shared by some or all of the FP16 values included in the tensor. The FP16 values may include IEEE 754 format 16-bit floating point values and the tensor may include a plurality of bits defining the shared exponent. The tensor may include a shared exponent and FP16 values that include a variable bit-length mantissa and a variable bit-length exponent that may be dynamically set by processor circuitry. The tensor may include a shared exponent and FP16 values that include a variable bit-length mantissa; a variable bit-length exponent that may be dynamically set by processor circuitry; and a shared exponent switch set by the processor circuitry to selectively combine the FP16 value exponent with the shared exponent.

First claim

What is claimed:

1. A system for training a neural network, comprising:
processor circuitry;
a communications interface coupled to the processor circuitry, the communications interface couplable to the neural network; and
a storage device coupled to the processor circuitry, the storage device including machine readable instructions that, when executed by the processor circuitry, cause the processor circuitry to:
generate a neural network training tensor that includes:
a five-bit shared exponent stored in a memory, the five-bit shared exponent common to each of a plurality of 16-bit floating point values included in the neural network training tensor, each of the plurality of 16-bit floating point values including:
a first plurality of bits stored in the memory, the first plurality of bits to form a mantissa of the respective floating point value;
a second plurality of bits stored in the memory, the second plurality of bits to form an exponent portion of the respective floating point value; and
a one-bit switch stored in the memory, the one-bit switch to selectively combine the exponent portion of the respective 16-bit floating point value with the five-bit shared exponent; and

adjust, in response to prediction of an overflow condition or an underflow condition during training of the neural network, one or more of the 16-bit floating point values in the memory to avoid the predicted overflow or underflow.

2. The system of claim 1 , wherein each of the plurality of 16-bit floating point values included in the neural network training tensor includes:
a six bit mantissa provided by the first plurality of bits;
an eight bit exponent provided by the second plurality of bits; and
a one-bit sign.

3. The system of claim 1 , wherein the mantissa is a variable bit-length mantissa, the exponent portion is a variable bit-length exponent, and wherein the instructions further cause the processor circuitry to select, based on one or more neural network parameters:
a first number of bits to represent the variable bit-length mantissa; and
a second number of bits to represent the variable bit-length exponent.

4. The system of claim 3 , wherein the one or more neural network parameters include a trend indicative of at least one of: an underflow condition or an overflow condition in one or more of the 16-bit floating point values included in the neural network training tensor.

5. The system of claim 1 , wherein each of the plurality of 16-bit floating point values included in the neural network training tensor includes:
the mantissa provided by the first plurality of bits;
the exponent portion provided by the second plurality of bits; and
a one-bit sign.

6. A method of training a neural network, comprising:
generating, by processor circuitry, a neural network training tensor that includes:
a five-bit shared exponent stored in a memory, the five-bit shared exponent common to each of a plurality of 16-bit floating point values included in the neural network training tensor, each of the 16-bit floating point values including:
a first plurality of bits stored in the memory, the first plurality of bits to form a mantissa of the respective floating point value; and
a second plurality of bits stored in the memory, the second plurality of bits form an exponent portion of the respective floating point value; and
a one-bit switch stored in the memory, the one-bit switch to selectively combine the exponent portion of the respective 16-bit floating point value with the five-bit shared exponent;

adjusting, by the processor circuitry in response to prediction of an overflow condition or an underflow condition during training of the neural network, one or more of the 16-bit floating point values in the memory to avoid the predicted overflow or underflow; and
providing, by the processor circuitry, at least one of the plurality of 16-bit floating point values included in the neural network training tensor as an input to the neural network.

7. The method of claim 6 , wherein the generating of the neural network training tensor includes:
generating the neural network training tensor that includes the plurality of 16-bit floating point values and the five-bit shared exponent, wherein each of the plurality of 16-bit floating point values includes:
a six bit mantissa provided by the first plurality of bits;
an eight bit exponent provided by the second plurality of bits; and
a one-bit sign.

8. The method of claim 6 , wherein the generating of the neural network training tensor includes:
generating the neural network training tensor that includes the plurality of 16-bit floating point values and the five-bit shared exponent, wherein each of the plurality of 16-bit floating point values includes:
a variable bit-length mantissa provided by the first plurality of bits;
a variable bit-length exponent provided by the second plurality of bits; and
a one-bit sign.

9. The method of claim 8 , further including selecting, by the processor circuitry based on one or more neural network parameters:
a first number of bits to represent the variable bit-length mantissa; and
a second number of bits to represent the variable bit-length exponent.

10. The method of claim 9 , further including:
detecting, by the processor circuitry, a trend in one or more of the plurality of 16-bit floating point values included in the training tensor, the detected trend indicative of at least one of: an underflow condition or an overflow condition in one or more of the 16-bit floating point values included in the neural network training tensor.

11. The method of claim 9 , wherein the selecting of the first number of bits to represent the variable bit-length mantissa and the second number of bits to represent the variable bit-length exponent includes:
selecting, by the processor circuitry, a first number of bits included in the first plurality of bits and a second number of bits included in the second plurality of bits responsive to detecting a trend indicative of at least one of: an underflow condition or an overflow condition in one or more of the 16-bit floating point values included in the neural network training tensor.

12. The method of claim 6 , wherein the generating of the neural network training tensor includes:
generating the neural network training tensor that includes the plurality of 16-bit floating point values and the five-bit shared exponent, wherein each of the plurality of 16-bit floating point values includes:
the mantissa provided by the first plurality of bits;
the exponent portion provided by the second plurality of bits; and
a one-bit sign.

13. A neural network training system, comprising:
means for generating a neural network training tensor that includes:
a five-bit shared exponent stored in a memory, the five-bit shared exponent common to each of a plurality of 16-bit floating point values included in the neural network training tensor, each of the 16-bit floating point values including:
a first plurality of bits stored in the memory, the first plurality of bits to form a mantissa of the respective floating point value;
a second plurality of bits stored in the memory, the second plurality of bits to form an exponent portion of the respective floating point value; and
a one-bit switch stored in the memory, the one-bit switch to selectively combine the exponent portion of the respective 16-bit floating point value with the five-bit shared exponent;

means for adjusting, in response to prediction of an overflow condition or an underflow condition during training of a neural network, one or more of the 16-bit floating point values in the memor

Assignees

• Intel Corp

Inventors

• Nair Krishnakumar
• Yang Andrew
• Morris Brian

Classifications

• G06F9/30025

  Format conversion instructions, e.g. Floating-Point to Integer, decimal conversion · CPC title

• G06N3/063 Primary

  using electronic means · CPC title

• G06N3/0495

  Quantised networks; Sparse networks; Compressed networks · CPC title

• G06N3/09

  Supervised learning · CPC title

• G06F7/49942

  Significance control · CPC title