Piecewise quantization for neural networks

Patent metadata

Field	Value
Publication number	US-11775611-B2
Application number	US-202016816247-A
Country	US
Kind code	B2
Filing date	Mar 11, 2020
Priority date	Nov 1, 2019
Publication date	Oct 3, 2023
Grant date	Oct 3, 2023

Abstract

In some embodiments, a method of quantizing an artificial neural network includes dividing a quantization range for a tensor of the artificial neural network into a first region and a second region, and quantizing values of the tensor in the first region separately from values of the tensor in the second region. In some embodiments, linear or nonlinear quantization are applied to values of the tensor in the first region and the second region. In some embodiments, the method includes locating a breakpoint between the first region and the second region by substantially minimizing an expected quantization error over at least a portion of the quantization range. In some embodiments, the expected quantization error is minimized by solving analytically and/or searching numerically.

First claim

The invention claimed is:

1. A system, comprising:
a weight buffer configured to store weight values and a first index value associated with each respective weight value, each first index value indicating a type of quantizing applied to a corresponding weight value, the weight values being arranged in at least one column of the weight buffer;
an activation buffer configured to store activation values, a second index value associated with each respective activation value, and a bit-range indicator associated with each respective activation value;
a selector circuit configured to select a weight value in each column of the weight buffer corresponding to the second index value of a current activation value;
a multiplier circuit corresponding to each respective column of the weight buffer, the multiplier circuit configured to multiply the current activation value and the weight value selected corresponding to the second index value of the current activation value;
a shift-register circuit coupled to an output of the multiplier circuit, the shift-register circuit being configured to shift the output of the multiplier circuit based on the bit-range indicator associated with the current activation value; and
an adder-tree circuit coupled to an output of the shift-register circuit, the adder-tree circuit being configured to be enabled or disabled to add the output of the shift-register circuit based on the first index value of the weight value selected in the column of the weight buffer.

2. The system of claim 1 , wherein the bit-range indicator indicates whether the activation value associated with the bit-range indicator comprises a most-significant nibble or a least-significant nibble of the activation value.

3. The system of claim 1 , wherein the first index value associated with a weight value indicates whether the weight value is within a predetermined weight-value range of multiple weight-value ranges spanning a distribution of the weight values.

4. The system of claim 3 , wherein the distribution of the weight values comprises a generally bell-shaped distribution or a normal distribution.

5. The system of claim 3 , wherein the weight values comprise quantized values.

6. The system of claim 5 , wherein the first index value associated with a weight value corresponds to a predetermined quantizing technique applied to the weight value.

7. The system of claim 6 , wherein the predetermined quantizing technique comprises at least one of a uniform quantizing technique, a linear quantizing technique, a nonlinear quantizing technique, an affine quantizing technique, and a non-affine quantizing technique.

8. The system of claim 5 , wherein a breakpoint between value ranges of the distribution of weight values is based on a minimum of an expected quantizing error of a quantizing technique used to quantize the weight values.

9. The system of claim 1 , wherein the activation values comprise quantized second values, each activation value being quantized using a predetermined quantizing technique based on the activation value being within a predetermined activation-value range of multiple activation value ranges spanning a distribution of the activation values.

10. The system of claim 9 , wherein the predetermined quantizing technique comprises at least one of a uniform quantizing technique, a linear quantizing technique, a nonlinear quantizing technique, an affine quantizing technique, and a non-affine quantizing technique.

11. A system, comprising:
a first buffer configured to store first values and a first index value associated with each respective first value, each first index value indicating a type of quantizing applied to a corresponding first value, the first values being arranged in at least one column of the first buffer;
a second buffer configured to store second values, a second index value associated with each respective second value, and a bit-range indicator associated with each respective second value;
a selector circuit configured to select a first value in each column of the first buffer corresponding to the second index value of a current second value;
a multiplier circuit corresponding to each respective column of the first buffer, each multiplier circuit configured to multiply the current second value and the first value selected corresponding to the second index value of the current second value;
a shift-register circuit coupled to an output of the respective multiplier circuit, the shift-register circuit being configured to shift the output of the multiplier circuit based on the bit-range indicator associated with the current second value; and
an adder-tree circuit coupled to an output of the shift-register circuit, the adder-tree circuit being configured to be enabled or disabled to add the output of a corresponding shift-register circuit based on the first index value of the first value selected in the column of the first buffer.

12. The system of claim 11 , wherein the bit-range indicator indicates whether the second value associated with the bit-range indicator comprises a most-significant nibble or a least-significant nibble of the second value.

13. The system of claim 11 , wherein the first index value associated with a first value indicates whether the first value is within a predetermined value range of multiple value ranges spanning a distribution of first values.

14. The system of claim 13 , wherein the distribution of first values comprises a generally bell-shaped distribution or a normal distribution.

15. The system of claim 13 , wherein the first values comprise quantized values.

16. The system of claim 15 , wherein the first index value associated with a first value corresponds to a predetermined quantizing technique applied to the first value.

17. The system of claim 16 , wherein the predetermined quantizing technique comprises at least one of a uniform quantizing technique, a linear quantizing technique, a nonlinear quantizing technique, an affine quantizing technique, and a non-affine quantizing technique.

18. The system of claim 17 , wherein a breakpoint between value ranges of the distribution of first values is based on a minimum of an expected quantizing error of a quantizing technique used to quantize the first values.

19. The system of claim 11 , wherein the second values comprise quantized second values, each second value being quantized using a predetermined quantizing technique based on the second value being within a predetermined second-value range of multiple second-value ranges spanning a distribution of the second values.

20. The system of claim 19 , wherein the predetermined quantizing technique comprises at least one of a uniform quantizing technique, a linear quantizing technique, a nonlinear quantizing technique, an affine quantizing technique, and a non-affine quantizing technique.

Assignees

• Samsung Electronics Co Ltd

Inventors

• Fang Jun
• Hassoun Joseph H
• Shafiee Ardestani Ali
• Abdelaziz Hamzah Ahmed Ali
• Georgiadis Georgios
• Chen Hui
• Thorsley David Philip Lloyd

Classifications

• G06N3/0464

  Convolutional networks [CNN, ConvNet] · CPC title

• G06N3/0495

  Quantised networks; Sparse networks; Compressed networks · CPC title

• G06F17/18 Primary

  for evaluating statistical data {, e.g. average values, frequency distributions, probability functions, regression analysis (forecasting specially adapted for a specific administrative, business or logistic context G06Q10/04)} · CPC title

• G06N3/02

  Neural networks · CPC title

• G06N3/04 Primary

  Architecture, e.g. interconnection topology · CPC title