Accelerating long short-term memory networks via selective pruning

Patent metadata

Field	Value
Publication number	US-10657426-B2
Application number	US-201815937558-A
Country	US
Kind code	B2
Filing date	Mar 27, 2018
Priority date	Jan 25, 2018
Publication date	May 19, 2020
Grant date	May 19, 2020

Abstract

A system and method for pruning. A neural network includes a plurality of long short-term memory cells, each of which includes an input having a weight matrix Wc, an input gate having a weight matrix Wi, a forget gate having a weight matrix Wf, and an output gate having a weight matrix Wo. In some embodiments, after initial training, one or more of the weight matrices Wi, Wf, and Wo are pruned, and the weight matrix Wc is left unchanged. The neural network is then retrained, the pruned weights being constrained to remain zero during retraining.

First claim

What is claimed is:

1. A method for performing machine learning in a first neural network comprising one or more long short-term memory cells, a first long short-term memory cell of the one or more long short-term memory cells comprising:
an input having a first weight matrix W c ;
an input gate having a first weight matrix W i ;
a forget gate having a first weight matrix W f ; and
an output gate having a first weight matrix W o ,
the method comprising:
maintaining the first weight matrix W c unchanged; and
pruning the first weight matrix W f , the pruning of the first weight matrix W f comprising:
calculating a standard deviation over a subset of elements of the first weight matrix W f ;
determining that a first element, at a first position in the first weight matrix W f , fails to meet a first threshold, the first threshold being based on a second threshold and the standard deviation;
setting to zero the first element, at the first position in the first weight matrix W f , based on determining that the first element, at the first position in the first weight matrix W f , fails to meet the first threshold;
determining that a second element, at a second position in the first weight matrix W f , meets the first threshold; and
maintaining the second element unchanged, based on determining that the second element, at the second position in the first weight matrix W f , meets the first threshold.

2. The method of claim 1 , wherein a second long short-term memory cell of the one or more long short-term memory cells comprises an input gate having a second weight matrix W f and wherein the calculating of the standard deviation comprises calculating a standard deviation over all elements of the first and second weight matrices W f .

3. The method of claim 1 , wherein the subset of elements of the first weight matrix W f includes all elements of the first weight matrix W f .

4. The method of claim 1 , wherein a second long short-term memory cell of the one or more long short-term memory cells comprises a second weight matrix W f and wherein the calculating of the standard deviation comprises calculating a standard deviation over all elements at the first position in the first and second weight matrices W f .

5. The method of claim 1 , further comprising pruning the first weight matrix W i .

6. The method of claim 1 , further comprising pruning the first weight matrix W o .

7. The method of claim 1 , wherein the first neural network further comprises a plurality of artificial neurons and a plurality of connections between the artificial neurons, each of the connections having a weight, and the method further comprises:
calculating a standard deviation over the weights of the connections;
setting to zero the weight of a connection of the plurality of connections when a magnitude of the weight is smaller than a product of a threshold and the standard deviation; and
leaving the weight of the connection unchanged, otherwise.

8. The method of claim 1 , further comprising retraining the first neural network, the retraining comprising maintaining the first element equal to zero.

9. The method of claim 8 , further comprising operating a second neural network, the second neural network having a weight matrix W f equal to the weight matrix W f of the first neural network, the operating comprising:
classifying an image with the second neural network, and
controlling a vehicle based on the classifying of the image.

10. A system for performing machine learning, the system comprising a first processing circuit, the first processing circuit being configured to:
instantiate a first neural network comprising one or more long short-term memory cells, a first long short-term memory cell of the one or more long short-term memory cells comprising:
an input having a first weight matrix W c ;
an input gate having a first weight matrix W i ;
a forget gate having a first weight matrix W f ; and
an output gate having a first weight matrix W o ,

maintain the first weight matrix W c unchanged; and
prune the first weight matrix W f , the pruning of the first weight matrix W f comprising:
calculating a standard deviation over a subset of elements of the plurality of weight matrices W f ;
determining that a first element, at a first position in the first weight matrix W f , fails to meet a first threshold, the first threshold being based on a second threshold and the standard deviation;
setting to zero the first element, at the first position in the first weight matrix W f , based on determining that the first element, at the first position in the first weight matrix W f , fails to meet the first threshold;
determining that a second element, at a second position in the first weight matrix W f , meets the first threshold; and
maintaining the second element unchanged, based on determining that the second element, at the second position in the first weight matrix W f , meets the first threshold.

11. The system of claim 10 , wherein a second long short-term memory cell of the one or more long short-term memory cells comprises an input gate having a second weight matrix W f and wherein the calculating of the standard deviation comprises calculating a standard deviation over all elements of the first and second weight matrices W f .

12. The system of claim 10 , wherein the subset of elements of the first weight matrix W f includes all elements of the first weight matrix W f .

13. The system of claim 10 , wherein a second long short-term memory cell of the one or more long short-term memory cells comprises a second weight matrix W f and wherein the calculating of the standard deviation comprises calculating a standard deviation over all elements at the first position in the first and second weight matrices W f .

14. The system of claim 10 , wherein the first processing circuit is further configured to prune the first weight matrix W i .

15. The system of claim 10 , wherein the first processing circuit is further configured to prune the first weight matrix W o .

16. The system of claim 10 , wherein the system further comprises a plurality of artificial neurons and a plurality of connections between the artificial neurons, each of the connections having a weight, and the first processing circuit is further configured to:
calculate a standard deviation over the weights of the connections;
set to zero the weight of a connection of the plurality of connections when a magnitude of the weight is smaller than a product of a threshold and the standard deviation; and
leave the weight of the connection unchanged, otherwise.

17. The system of claim 10 , wherein the first processing circuit is further configured to retrain the first neural network, the retraining comprising leaving the first element equal to zero.

18. The system of claim 17 , further comprising a second processing circuit configured to instantiate a second neural network, the second neural network having a weight matrix W f equal to the weight matrix W f of the first neural network, and to:
classify images with the second neural network, and
control a vehicle based on the classifying of the images.

19. A method for performing machine learning in a first neural network comprising one or more long short-term memory cells, a first long short-term memory cell of the one or more long short-term memory cells comprising:
an input having a first weight matrix W c ;
an input gate having a first weight matrix W i ;
a forget gate having a first weight matrix W f ; and
an o

Assignees

• Samsung Electronics Co Ltd

Inventors

• Georgiadis Georgios
• Deng Weiran

Classifications

• G06N3/08

  Learning methods · CPC title

• G06N3/082 Primary

  modifying the architecture, e.g. adding, deleting or silencing nodes or connections · CPC title

• G06K9/627

  Physics · mapped topic

• G06K9/66 Primary

  Physics · mapped topic

• G06V10/82

  using neural networks · CPC title