Facilitating neural network efficiency

Patent metadata

Field	Value
Publication number	US-11195096-B2
Application number	US-201715792733-A
Country	US
Kind code	B2
Filing date	Oct 24, 2017
Priority date	Oct 24, 2017
Publication date	Dec 7, 2021
Grant date	Dec 7, 2021

Abstract

Techniques that facilitate improving an efficiency of a neural network are described. In one embodiment, a system is provided that comprises a memory that stores computer-executable components and a processor that executes computer-executable components stored in the memory. In one implementation, the computer-executable components comprise an initialization component that selects an initial value of an output limit, wherein the output limit indicates a range for an output of an activation function of a neural network. The computer-executable components further comprise a training component that modifies the initial value of the output limit during training to a second value of the output limit, the second value of the output limit being provided as a parameter to the activation function. The computer-executable components further comprise an activation function component that determines the output of the activation function based on the second value of the output limit as the parameter.

First claim

What is claimed is:

1. A system, comprising:
a memory that stores computer executable components; and
a processor that executes the computer executable components stored in the memory, wherein the computer executable components comprise:
an initialization component that selects an initial value of an output limit, wherein the output limit indicates a range for an output of an activation function of a neural network;
a training component that modifies the initial value of the output limit during training to a second value of the output limit, the second value of the output limit being provided as a parameter to the activation function; and
an activation function component that determines the output of the activation function based on using the second value of the output limit as the parameter, wherein the training component utilizes a greater precision than a precision of the activation function component.

2. The system of claim 1 , wherein the training component utilizes a resolution parameter that has a greater precision than the precision of the activation function component, and wherein the training component utilizes a resolution slope parameter that indicates a slope within a sub-resolution range.

3. The system of claim 2 , wherein the training component increases a value of the resolution slope parameter toward infinity during training.

4. The system of claim 1 , wherein the computer executable components further comprise:
a clipping component that performs clipping during training with the training component to reduce accuracy degradation due to quantization.

5. The system of claim 1 , wherein the computer executable components further comprise:
a back-propagation component that performs back propagation during training with the training component.

6. The system of claim 1 , wherein the computer executable components further comprise:
an activation function selection component that determines to use a rectifier linear unit as the activation function in a case of full precision, and wherein cross entropy loss converges as the output limit increases.

7. The system of claim 1 , wherein the computer executable components further comprise:
an activation function selection component that determines that as the output limit increases, a loss function also increases with quantization, and determines to use an activation function type of the activation function that is other than a rectifier linear unit.

8. A computer-implemented method, comprising:
initializing, by a system operatively coupled to a processor, a value for an output limit, wherein the output limit comprises a range for an output of an activation function of a neural network, the value for the output limit being determined via training;
determining, by the system, the output of the activation function given the value of the output limit as a parameter to the activation function; and
determining, by the system, that as the output limit increases, a loss function also increases with quantization, and determining to use an activation function type of the activation function that is other than a rectifier linear unit.

9. The computer-implemented method of claim 8 , further comprising:
applying, by the system, a stochastic gradient descent approach during the training.

10. The computer-implemented method of claim 8 , further comprising:
determining, by the system, the value for the output limit based on performing the training with an initial value of the output limit.

11. The computer-implemented method of claim 8 , further comprising:
regularizing, by the system, the output limit during the training.

12. The computer-implemented method of claim 8 , further comprising:
clipping, by the system, during the training to reduce accuracy degradation due to quantization.

13. The computer-implemented method of claim 8 , further comprising:
employing, by the system, back propagation during the training.

14. The computer-implemented method of claim 8 , further comprising:
determining, by the system, to use a rectifier linear unit as the activation function in a case of full precision, and wherein cross entropy loss converges as the output limit increases.

15. A computer program product that facilitates training quantized activations for efficient implementation of deep learning, the computer program product comprising a computer readable storage medium having program instructions embodied therewith, the program instructions executable by a processor to cause the processor to at least:
initialize a value for an output limit, wherein the output limit comprises a range for an output of an activation function of a neural network, the value for the output limit being determined via training;
determine the output of the activation function given the value of the output limit as a parameter to the activation function; and
regularize the output limit during the training.

16. The computer program product of claim 15 , wherein the output limit is expressed as α, and wherein the activation function is expressed with equations comprising:

y
=

0.5
⁢

(


x


-



x
-
α



+
α

)

=

{

α
,

x
∈

[

α
,

+
∞

)

x
,

x
∈

[

0
,
α

)

Assignees

• IBM

Inventors

• Choi Jungwook
• Gopalakrishnan Kailash
• Sakr Charbel
• Venkataramani Swagath
• Wang Zhuo

Classifications

• G06N3/048

  Activation functions · CPC title

• G06N3/084 Primary

  Backpropagation, e.g. using gradient descent · CPC title

• G06N3/04

  Architecture, e.g. interconnection topology · CPC title

• G06N3/0464

  Convolutional networks [CNN, ConvNet] · CPC title

• G06N3/0495

  Quantised networks; Sparse networks; Compressed networks · CPC title