Systems and methods for modifying neural networks for binary processing applications

Patent metadata

Field	Value
Publication number	US-11790241-B2
Application number	US-202017016130-A
Country	US
Kind code	B2
Filing date	Sep 9, 2020
Priority date	Sep 9, 2019
Publication date	Oct 17, 2023
Grant date	Oct 17, 2023

Abstract

In one embodiment, a method of simulating an operation of an artificial neural network on a binary neural network processor includes receiving a binary input vector for a layer including a probabilistic binary weight matrix and performing vector-matrix multiplication of the input vector with the probabilistic binary weight matrix, wherein the multiplication results are modified by simulated binary-neural-processing hardware noise, to generate a binary output vector, where the simulation is performed in the forward pass of a training algorithm for a neural network model for the binary-neural-processing hardware.

First claim

What is claimed is:

1. A method of training an artificial neural network based on simulated properties of a binary neural network processor, comprising:
receiving a binary input vector for a layer of the artificial neural network comprising a probabilistic binary weight matrix;
generating simulated binary-neural-processing hardware noise based on permutations of properties of the binary neural network processor;
performing vector-matrix multiplication of the input vector with the probabilistic binary weight matrix;
modifying the multiplication results based on the simulated binary-neural-processing hardware noise, to generate a binary output vector; and
forward propagating the binary output vector to one or more other layers of the artificial neural network.

2. The method of claim 1 , wherein the binary neural network processor is a compute-in-memory (CIM) device.

3. The method of claim 1 , wherein the probabilistic binary weight matrix comprises probabilities that the weights are 0 or 1.

4. The method of claim 1 , wherein the simulated binary-neural-processing hardware noise simulates process, voltage, and temperature (PVT) variations of the simulated binary-neural-processing hardware.

5. The method of claim 4 , wherein the thermal noise is varied for each multiplication.

6. The method of claim 4 , wherein:
the PVT variation noise includes capacitor variation noise;
generating the binary output vector comprises calculating a population count; and
capacitor variation noise depends on the population count.

7. The method of claim 1 , wherein:
generating the binary output vector comprises calculating a population count with noise ã;
ã=αa+β+γ;

a is a clean population count;
α represents capacitor variation noise;
β represents offset variation noise; and
γ represents thermal noise.

8. The method of claim 1 , wherein:
the layer is larger than a processing array of the binary neural network processor;
the binary neural network processor comprises analog-to-digital converters (ADCs);
performing the vector-matrix multiplication comprises splitting the layer and generating intermediate results digitized by the ADCs; and
the simulated binary-neural-processing hardware noise includes quantizing noise from digitizing the intermediate results by the ADCs.

9. The method of claim 1 , wherein:
performing the vector-matrix multiplication comprises using the binary neural network process in a sharing configuration; and
the simulated binary-neural-processing hardware noise includes correlated noise for the sharing configuration.

10. The method of claim 1 , wherein the simulated binary-neural-processing hardware noise is generated from a low-level circuit simulation of the binary neural network processor.

11. The method of claim 1 , wherein:
the artificial neural network comprises a first layer and a plurality of additional layers;
the probabilistic binary weight matrix may correspond to any of the plurality of additional layers; and
the probabilistic binary weight matrix may not correspond to the first layer.

12. An apparatus comprising a processor and a memory, the apparatus configured to train an artificial neural network based on simulated properties of a binary neural network processor, the training comprising:
receiving a binary input vector for a layer of the artificial neural network comprising a probabilistic binary weight matrix;
generating simulated binary-neural-processing hardware noise based on permutations of properties of the binary neural network processor;
performing vector-matrix multiplication of the input vector with the probabilistic binary weight matrix;
modifying the multiplication results based on the simulated binary-neural-processing hardware noise, to generate a binary output vector; and
forward propagating the binary output vector to one or more other layers of the artificial neural network.

13. The apparatus of claim 12 , wherein the binary neural network processor is a compute-in-memory (CIM) device.

14. The apparatus of claim 12 , wherein the probabilistic binary weight matrix comprises probabilities that the weights are 0 or 1.

15. The apparatus of claim 12 , wherein the simulated binary-neural-processing hardware noise simulates process, voltage, and temperature (PVT) variations of the simulated binary-neural-processing hardware.

16. The apparatus of claim 15 , wherein the thermal noise is varied for each multiplication.

17. The apparatus of claim 15 , wherein:
the PVT variation noise includes capacitor variation noise;
generating the binary output vector comprises calculating a population count; and
capacitor variation noise depends on the population count.

18. The apparatus of claim 12 , wherein:
generating the binary output vector comprises calculating a population count with noise ã;
ã=αa+β+γ;

a is a clean population count;
α represents capacitor variation noise;
β represents offset variation noise; and
γ represents thermal noise.

19. The apparatus of claim 12 , wherein:
the layer is larger than a processing array of the binary neural network processor;
the binary neural network processor comprises analog-to-digital converters (ADCs);
performing the vector-matrix multiplication comprises splitting the layer and generating intermediate results digitized by the ADCs; and
the simulated binary-neural-processing hardware noise includes quantizing noise from digitizing the intermediate results by the ADCs.

20. The apparatus of claim 12 , wherein:
performing the vector-matrix multiplication comprises using the binary neural network process in a sharing configuration; and
the simulated binary-neural-processing hardware noise includes correlated noise for the sharing configuration.

21. The apparatus of claim 12 , wherein the simulated binary-neural-processing hardware noise is generated from a low-level circuit simulation of the binary neural network processor.

22. The apparatus of claim 12 , wherein:
the artificial neural network comprises a first layer and a plurality of additional layers;
the probabilistic binary weight matrix may correspond to any of the plurality of additional layers; and
the probabilistic binary weight matrix may not correspond to the first layer.

23. A non-transitory computer-readable medium storing computer-executable code for operation of a neural network, comprising code to train an artificial neural network based on simulated properties of a binary neural network processor, the training comprising:
receiving a binary input vector for a layer of the artificial neural network comprising a probabilistic binary weight matrix;
generating simulated binary-neural-processing hardware noise based on permutations of properties of the binary neural network processor;
performing vector-matrix multiplication of the input vector with the probabilistic binary weight matrix;
modifying the multiplication results based on the simulated binary-neural-processing hardware noise, to generate a binary output vector; and
forward propagating the binary output vector to one or more other layers of the artificial neural network.

Assignees

• Qualcomm Inc

Inventors

• Reisser Matthias
• Pitre Saurabh Kedar
• Zhu Xiaochun
• Teague Edward Harrison
• Wang Zhongze
• Welling Max

Classifications

• G06N3/09

  Supervised learning · CPC title

• G06N3/0495

  Quantised networks; Sparse networks; Compressed networks · CPC title

• G06N3/0464

  Convolutional networks [CNN, ConvNet] · CPC title

• G06N3/10 Primary

  Interfaces, programming languages or software development kits, e.g. for simulating neural networks · CPC title

• G06F17/16

  Matrix or vector computation {, e.g. matrix-matrix or matrix-vector multiplication, matrix factorization (matrix transposition G06F7/78)} · CPC title