Method of accelerating training process of neural network and neural network device thereof

Patent metadata

Field	Value
Publication number	US-12205012-B2
Application number	US-201916550498-A
Country	US
Kind code	B2
Filing date	Aug 26, 2019
Priority date	Aug 24, 2018
Publication date	Jan 21, 2025
Grant date	Jan 21, 2025

Abstract

A method of accelerating a training process of a neural network includes acquiring activations used in the training process and a bit-vector corresponding to the activations, selecting activations requiring an operation from among the acquired activations by using the bit-vector, and performing backward propagation using the selected activations and filters corresponding to the selected activations.

First claim

What is claimed is:

1. A method of accelerating a training of a neural network through backpropagation process, the method comprising:
in the backward propagation process:
acquiring a bit-vector including bits indicating which of output activations of forward propagation operation of a layer of the neural network are zero or non-zero, the bit-vector having been generated in a process of a forward propagation of the neural network, with input activations of the layer in the backward propagation process respectively corresponding to the output activations;
selecting, based on bits of the bit-vector indicating which of the input activations have values of 0, only non-zero activations and corresponding filters respectively from among the input activations and forward propagation filters of the layer used in the forward propagation operation;
generating backward propagation filters of the layer by rearranging the selected corresponding filters, where there are fewer generated backward propagation filters than forward propagation filters;
generating zero padded activations by performing zero padding on the selected non-zero activations; and
performing the backward propagation using the zero padded activations and the backward propagation filters,
wherein the selecting of the non-zero activations comprises selecting the non-zero activations based on an interpreting of the bits included in the bit-vector for a total number of the selected non-zero activations based on a total number of multipliers available for the performance of the backward propagation, or until the total number of the selected non-zero activations equals the total number of multipliers.

2. The method of claim 1 , wherein the selecting of the non-zero activations includes selecting the non-zero activations until the total number of the selected non-zero activations is equal to the total number of multipliers of a single neural functional unit.

3. The method of claim 1 , further comprising repeating the respective selecting, the generating of the backward propagation filters, the generating of the zero padded activations, and the performing of the backward propagation until all of the bits included in the bit-vector have been interpreted.

4. The method of claim 1 , wherein the performing of the backward propagation comprises performing a multiplication and accumulation operation on the selected non-zero activations and the backward propagation filters.

5. The method of claim 4 , wherein the performing of the backward propagation comprises updating the input activations using a result of the multiplication and accumulation operation.

6. The method of claim 1 , further comprising updating the forward propagation filters, for use in another forward propagation of the neural network, using a result of the performing of the backward propagation.

7. A non-transitory computer-readable recording medium that, when executed by a processor, cause the processor to perform the method of claim 1 .

8. A neural network device comprising:
a memory; and
one or more processors configured to:
in a backward propagation process for training a neural network:
acquire a bit-vector including bits indicating which output activations of a forward propagation operation of a layer of the neural network are zero or non-zero, the bit-vector being generated in a process of the a forward propagation of the neural network, with the output activations of the forward propagation being input activations of the layer in the backward propagation process respectively corresponding to the output activations;
select, based on bits of the bit-vector indicating which of the input output activations have values of 0, only non-zero activations and corresponding filters respectively from among the input activations and forward propagation filters of the layer used in the forward propagation operation, with a result of the selection of the corresponding filters including a storage of the selected corresponding filters to the memory;
generate backward propagation filters of the layer by rearranging the selected corresponding filters, where there are fewer generated backward propagation filters than forward propagation filters;
generate zero padded activations through performance of zero padding on the selected non-zero activations; and
perform the backward propagation using the zero padded activations and the backward propagation filters,
wherein, for the selecting of the non-zero activations, the at least one processor is configured to select the non-zero activations based on an interpreting of the bits included in the bit-vector for a total number of the selected non-zero activations based on a total number of multipliers available for the performance of the backward propagation, or until the total number of the selected non-zero activations equals the total number of multipliers.

9. The neural network device of claim 8 , wherein, for the selecting of the non-zero activations, the one or more processors are further configured to select the non-zero activations until the total number of the selected non-zero activations is equal to the total number of multipliers of a single neural functional unit.

10. The neural network device of claim 8 , wherein the one or more processors are further configured to repeat the respective selection, the generation of the backward propagation filters, the generation of the zero padded activations, and the performance of the backward propagation until all of the bits included in the bit-vector have been interpreted.

11. The neural network device of claim 8 , wherein the one or more processors are further configured to perform a multiplication and accumulation operation on the selected non-zero activations and the backward propagation filters.

12. The neural network device of claim 11 , wherein the one or more processors are further configured to update the input activations using a result of the multiplication and accumulation operation.

13. The neural network device of claim 8 , wherein the one or more processors are further configured to update the forward propagation filters, for use in another forward propagation of the neural network, using a result of the performed backward propagation.

14. The neural network device of claim 13 , wherein the one or more processors are further configured to store the updated forward propagation filters in the memory.

15. The method of claim 1 , wherein respective values of all of the bits of the bit-vector that indicate the non-zero activations each have a value of 1.

16. The method of claim 1 , wherein the selecting of the non-zero activations, from among the input activations, includes selecting the non-zero activations in parallel with a sequenced interpreting of the bits of the bit-vector, such that a first corresponding activation is selected upon the interpreting of the bits indicating that a bit of the bit vector indicates that the first corresponding activation is non-zero, and such that a second corresponding activation is selected upon a subsequent operation of the interpreting of the bits indicating that another bit of the bit vector indicates that the second corresponding activation is non-zero.

17. The method of claim 16 , wherein the selecting of the non-zero activations in parallel with the interpreting of the bits of the bit-vector includes sequentially interpreting the bits of the bit-vector, and upon a result of each respective sequential interpreting of a respective bit of the bit-vector being that a corresponding activation is a non-zero activation, selecting the corresp

Assignees

• Samsung Electronics Co Ltd
• Seoul Nat Univ R&Db Foundation

Inventors

• Lee Seungwon
• Park Hanmin
• Lee Gunhee
• Kim Namhyung
• Yu Joonsang
• Choi Kiyoung

Classifications

• G06N3/0464

  Convolutional networks [CNN, ConvNet] · CPC title

• G06N3/0442

  characterised by memory or gating, e.g. long short-term memory [LSTM] or gated recurrent units [GRU] · CPC title

• G06N3/09

  Supervised learning · CPC title

• G06N3/0495

  Quantised networks; Sparse networks; Compressed networks · CPC title

• G06N3/084 Primary

  Backpropagation, e.g. using gradient descent · CPC title