System and method for training neural networks with errors

Patent metadata

Field	Value
Publication number	US-11574194-B2
Application number	US-201916367078-A
Country	US
Kind code	B2
Filing date	Mar 27, 2019
Priority date	Mar 27, 2019
Publication date	Feb 7, 2023
Grant date	Feb 7, 2023

Abstract

A computing device includes one or more processors, random access memory (RAM), and a non-transitory computer-readable storage medium storing instructions for execution by the one or more processors. The computing device receives first data on which to train a neural network comprising at least one quantized layer and performs a set of training iterations to train weights for the neural network. Each training iteration of the set of training iterations includes stochastically writing values to the random access memory for a set of activations of the at least one quantized layer of the neural network using first write parameters corresponding to a first write error rate. The computing device stores trained values for the weights of the neural network. The trained neural network is configured to classify second data based on the stored values.

First claim

What is claimed is:

1. A method, comprising:
performing, at a computing device that includes one or more processors, a random access memory (RAM), and a non-transitory computer-readable storage medium including instructions for execution by the one or more processors, a set of operations including:
receiving first data on which to train a neural network comprising at least one quantized layer;
performing a set of training iterations to train weights for the neural network, each training iteration of the set of training iterations including stochastically writing activation values as a form of stochastic rounding, to the random access memory, for a set of activations of the at least one quantized layer of the neural network using first write parameters corresponding to a first write error rate; and
storing trained values for the weights of the neural network, wherein the trained neural network is configured to classify second data based on the stored trained values.

2. The method of claim 1 , wherein the RAM is magnetic RAM.

3. The method of claim 2 , wherein the first write parameters include a write current selected such that the computing device stochastically writes values to the random access memory at the first write error rate.

4. The method of claim 1 , wherein the first write parameters include a first write current to write a first value and a second write current to write a second value.

5. The method of claim 1 , wherein the first write error rate is greater than 0.5%.

6. The method of claim 5 , wherein the first write error rate is less than 20%.

7. The method of claim 1 , wherein the neural network comprises an XNOR neural network.

8. The method of claim 1 , wherein the neural network further includes one or more non-quantized layers.

9. The method of claim 1 , wherein each of the at least one quantized layer comprises a binary layer.

10. The method of claim 1 , wherein the neural network further comprises a second quantized layer and each training iteration of the set of training iterations includes stochastically writing values to the random access memory for a set of activations of the second quantized layer of the neural network using second write parameters corresponding to a second write error rate.

11. An electronic system comprising:
one or more processors;
a random access memory (RAM);
a non-transitory computer-readable storage medium including instructions for:
receiving first data on which to train a neural network comprising at least one quantized layer;
performing a set of training iterations to train weights for the neural network, each training iteration of the set of training iterations including stochastically writing activation values as a form of stochastic rounding to the random access memory for a set of activations of the at least one quantized layer of the neural network using first write parameters corresponding to a first write error rate; and
storing trained values for the weights of the neural network, wherein the
trained neural network is configured to classify second data based on the stored values.

12. The electronic system of claim 11 , wherein the electronic system comprises a chip.

13. The electronic system of claim 11 , wherein the RAM is magnetic RAM.

14. The electronic system of claim 11 , wherein the first write parameters include a first write current to write a first value and a second write current to write a second value.

15. The electronic system of claim 11 , wherein the first write error rate is greater than 0.5%.

16. The electronic system of claim 15 , wherein the first write error rate is less than 20%.

17. The electronic system of claim 11 , wherein the neural network comprises an XNOR neural network.

18. The electronic system of claim 11 , wherein the neural network further includes one or more non-quantized layers.

19. The electronic system of claim 11 , wherein each of the at least one quantized layer comprises a binary layer.

20. The electronic system of claim 11 , wherein the neural network further comprises a second quantized layer and each training iteration of the set of training iterations includes stochastically writing values to the random access memory for a set of activations of the second quantized layer of the neural network using second write parameters corresponding to a second write error rate.

Assignees

• Integrated Silicon Solution Cayman Inc

Inventors

• Tzoufras Michail

Classifications

• G06N3/09

  Supervised learning · CPC title

• G06N3/0495

  Quantised networks; Sparse networks; Compressed networks · CPC title

• G06N3/0464

  Convolutional networks [CNN, ConvNet] · CPC title

• G11C11/16

  using elements in which the storage effect is based on magnetic spin effect · CPC title

• G06F18/24

  Classification techniques · CPC title