Hybrid floating point representation for deep learning acceleration

Patent metadata

Field	Value
Publication number	US-11620105-B2
Application number	US-202017128407-A
Country	US
Kind code	B2
Filing date	Dec 21, 2020
Priority date	Feb 6, 2019
Publication date	Apr 4, 2023
Grant date	Apr 4, 2023

Abstract

In an embodiment, a method includes configuring a specialized circuit for floating point computations using numbers represented by a hybrid format, wherein the hybrid format includes a first format and a second format. In the embodiment, the method includes operating the further configured specialized circuit to store an approximation of a numeric value in the first format during a forward pass for training a deep learning network. In the embodiment, the method includes operating the further configured specialized circuit to store an approximation of a second numeric value in the second format during a backward pass for training the deep learning network.

First claim

What is claimed is:

1. A processor comprising:
a specialized circuit configured for floating point computations using numbers represented by a hybrid format, wherein the hybrid format includes a first format and a second format,
wherein the specialized circuit is operable to store an approximation of a numeric value in the first format during a forward pass for training a deep learning network, and
wherein the specialized circuit is operable to store an approximation of a second numeric value in the second format during a backward pass for training the deep learning network; and
wherein the first format and the second format have the same bit width.

2. The processor of claim 1 , wherein the bit width of the first format and the second format is 8-bit.

3. The processor of claim 1 , wherein the specialized circuit is further configured to apportion the first format and the second format into a sign bit, exponent bits (e), and mantissa bits (p).

4. The processor of claim 3 , wherein an exponent bit of the first format is less than an exponent bit of the second format.

5. The processor of claim 3 , wherein the first format uses 8-bits, wherein the first format uses five bits as the exponent bits and two bits as the mantissa bits.

6. The processor of claim 3 , wherein the second format uses 8-bits, wherein the second format uses six bits as the exponent bits and one bit as the mantissa bit.

7. The processor of claim 3 , wherein the approximation of the numeric value is represented as a function of a multiple of a fraction, wherein the fraction is an inverse of a number of discrete values that can be represented using only the mantissa bits.

8. The processor of claim 1 , wherein the specialized circuit comprises a floating point unit (FPU).

9. The processor of claim 1 , wherein the specialized circuit is further configured to represent, in the hybrid format, infinity in the same manner as a “Not a Number” (NaN) value.

10. The processor of claim 1 , wherein the specialized circuit is further configured to disregard, in the hybrid format, the sign bit when representing zero, infinity, and NaN.

11. The processor of claim 1 , wherein the hybrid format reserves only two representations, one reserved representation to represent zero and another reserved representation to represent NaN.

12. The processor of claim 1 , wherein the hybrid format is a very low precision format (VLP format) comprising less than sixteen bits.

13. A Floating Point Unit (FPU) component of a data processing system comprising:
a specialized circuit configured for floating point computations using numbers represented by a hybrid format, wherein the hybrid format includes a first format and a second format,
wherein the specialized circuit is operable to store an approximation of a numeric value in the first format during a forward pass for training a deep learning network, and
wherein the specialized circuit is operable to store an approximation of a second numeric value in the second format during a backward pass for training the deep learning network; and
wherein the first format and the second format have the same bit width.

14. The FPU component of claim 13 , wherein the bit width of the first format and the second format is 8-bit.

15. The FPU component of claim 13 , wherein the specialized circuit is further configured to apportion the first format and the second format into a sign bit, exponent bits (e), and mantissa bits (p).

16. The FPU component of claim 15 , wherein an exponent bit of the first format is less than an exponent bit of the second format.

17. The FPU component of claim 15 , wherein the first format uses 8-bits, wherein the first format uses five bits as the exponent bits and two bits as the mantissa bits.

18. The FPU component of claim 15 , wherein the second format uses 8-bits, wherein the second format uses six bits as the exponent bits and one bit as the mantissa bit.

Assignees

• IBM

Inventors

• Wang Naigang
• Choi Jungwook
• Gopalakrishnan Kailash
• Agrawal Ankur
• Mueller Silvia Melitta

Classifications

• G06N3/063

  using electronic means · CPC title

• G06F7/483 Primary

  Computations with numbers represented by a non-linear combination of denominational numbers, e.g. rational numbers, logarithmic number system or floating-point numbers {(G06F7/4806, G06F7/4824, G06F7/49, G06F7/491, G06F7/544 take precedence)} · CPC title

• G06N3/084

  Backpropagation, e.g. using gradient descent · CPC title

• G06N3/045 Primary

  Combinations of networks · CPC title

• G06F9/30025

  Format conversion instructions, e.g. Floating-Point to Integer, decimal conversion · CPC title