Systems and methods for quantizing neural networks via periodic regularization functions

Patent metadata

Field	Value
Publication number	US-11468313-B1
Application number	US-201816006095-A
Country	US
Kind code	B1
Filing date	Jun 12, 2018
Priority date	Jun 12, 2018
Publication date	Oct 11, 2022
Grant date	Oct 11, 2022

Abstract

The disclosed computer-implemented method may include (1) identifying an artificial neural network comprising a set of nodes interconnected via a set of connections, and (2) training the artificial neural network by, for each connection in the set of connections, determining a quantized weight value associated with the connection. Determining the quantized weight value associated with the connection may include (1) associating a loss function with the connection, the loss function including a periodic regularization function that describes a relationship between an input value and a weight value of the connection, (2) determining a minimum of the associated loss function with respect to the weight value in accordance with the periodic regularization function, and (3) generating the quantized weight value associated with the connection based on the determined minimum of the loss function. Various other methods, systems, and computer-readable media are also disclosed.

First claim

What is claimed is:

1. A computer-implemented method comprising:
identifying an artificial neural network comprising a set of nodes interconnected via a set of connections; and
training the artificial neural network by, for each connection in the set of connections, determining a quantized weight value associated with the connection, wherein determining the quantized weight value associated with the connection comprises:
receiving data representative of a number of bits for a representation of the quantized weight value;
associating a loss function with the connection, the loss function comprising a periodic regularization function that describes a relationship between an input value and a weight value of the connection, the associating comprising determining a frequency for the periodic regularization function in accordance with a pre-determined relationship between the frequency and the received number of bits;
determining a minimum of the associated loss function with respect to the weight value in accordance with the periodic regularization function; and
generating the quantized weight value associated with the connection based on the determined minimum of the loss function.

2. The computer-implemented method of claim 1 , wherein the periodic regularization function comprises a trigonometric function.

3. The computer-implemented method of claim 2 , wherein the trigonometric function comprises at least one of:
a sine function; or
a cosine function.

4. The computer-implemented method of claim 1 , wherein the periodic regularization function comprises a sawtooth function.

5. The computer-implemented method of claim 1 , wherein the periodic regularization function comprises a triangular function.

6. The computer-implemented method of claim 1 , wherein the periodic regularization function comprises a scaling factor.

7. The computer-implemented method of claim 6 , wherein the scaling factor scales the periodic regularization function such that a result of the periodic regularization function is in an inclusive range from 0 to 1.

8. The computer-implemented method of claim 1 , wherein generating the quantized weight value associated with the connection based on the determined minimum of the loss function comprises:
selecting a subset of connections from the set of connections, the connection included in the selected subset of connections;
identifying a maximum absolute value of the determined minimums of the loss functions associated with the connections included in the subset of connections;
scaling the minimum of the loss function based on the identified maximum absolute value and a frequency;
rounding the scaled minimum of the loss function to a nearest integer value;
re-scaling the rounded and scaled minimum of the loss function based on the maximum absolute value and the frequency; and
designating the re-scaled minimum of the loss function as the quantized weight value associated with the connection.

9. The computer-implemented method of claim 1 , wherein the periodic regularization function comprises an amplitude factor.

10. The computer-implemented method of claim 1 , further comprising predicting an output value based on the input value via the trained artificial neural network.

11. The computer-implemented method of claim 1 , wherein the pre-determined relationship between the frequency and the received number of bits comprises a pre-determined function that relates the received number of bits to the frequency.

12. A system comprising:
an identifying module, stored in memory, that identifies an artificial neural network comprising a set of nodes interconnected via a set of connections; and
a training module, stored in memory, that trains the artificial neural network by, for each connection in the set of connections, determining a quantized weight value associated with the connection, wherein the training module determines the quantized weight value associated with the connection by:
receiving a number of bits for a representation of the quantized weight value; and
associating a loss function with the connection, the loss function comprising a periodic regularization function that describes a relationship between an input value and a weight value of the connection, the associating comprising determining a frequency for the periodic regularization function in accordance with a pre-determined relationship between the frequency and the received number of bits;
determining a minimum of the associated loss function with respect to the weight value in accordance with the periodic regularization function; and
generating the quantized weight value associated with the connection based on the determined minimum of the loss function; and

at least one physical processor that executes the identifying module and the training module.

13. The system of claim 12 , wherein the periodic regularization function comprises a trigonometric function.

14. The system of claim 13 , wherein the trigonometric function comprises at least one of:
a sine function; or
a cosine function.

15. The system of claim 12 , wherein the periodic regularization function comprises at least one of:
a sawtooth function; or
a triangular function.

16. The system of claim 12 , wherein the periodic regularization function comprises a scaling factor.

17. The system of claim 12 , wherein the training module generates the quantized weight value associated with the connection based on the determined minimum of the loss function by:
selecting a subset of connections from the set of connections, the connection included in the selected subset of connections;
identifying a maximum absolute value of the determined minimums of the loss functions associated with the connections included in the subset of connections;
scaling the minimum of the loss function based on the identified maximum absolute value and a frequency;
rounding the scaled minimum of the loss function to a nearest integer value;
re-scaling the rounded and scaled minimum of the loss function based on the maximum absolute value and the frequency; and
designating the re-scaled minimum of the loss function as the quantized weight value associated with the connection.

18. The system of claim 12 , wherein:
the system further comprises a predicting module that predicts an output value based on the input value via the trained artificial neural network; and
the physical processor further executes the predicting module.

19. The system of claim 12 , wherein the pre-determined relationship between the frequency and the received number of bits comprises a pre-determined function that relates the received number of bits to the frequency.

20. A non-transitory computer-readable medium comprising computer-readable instructions that, when executed by at least one processor of a computing system, cause the computing system to:
identify an artificial neural network comprising a set of nodes interconnected via a set of connections; and
train the artificial neural network by, for each connection in the set of connections, determining a quantized weight value associated with the connection, wherein determining the quantized weight value associated with the connection comprises:
receiving a number of bits for a representation of the quantized weight value;
associating a loss function with the connection, the loss function comprising a periodic regularization function that describes a relationship between an input value and a weight value of the connection, the associating compr

Assignees

• Meta Platforms Inc

Inventors

• Naumov Maxim
• Diril Abdulkadir Utku
• Park Jong Soo
• Ray Benjamin
• Jablonski Jedrzej
• Tulloch Andrew John

Classifications

• G06N5/01

  Dynamic search techniques; Heuristics; Dynamic trees; Branch-and-bound · CPC title

• G06N3/084 Primary

  Backpropagation, e.g. using gradient descent · CPC title

• G06N3/063

  using electronic means · CPC title

• G06N3/04

  Architecture, e.g. interconnection topology · CPC title

• G06N3/047

  Probabilistic or stochastic networks · CPC title