Recurrent neural networks with diagonal and programming fluctuation to find energy global minima

Patent metadata

Field	Value
Publication number	US-11599771-B2
Application number	US-201916260898-A
Country	US
Kind code	B2
Filing date	Jan 29, 2019
Priority date	Jan 29, 2019
Publication date	Mar 7, 2023
Grant date	Mar 7, 2023

Abstract

Recurrent neural networks, and methods therefor, are provided with diagonal and programming fluctuation to find energy global minima. The method may include storing the matrix of weights in memory cells of a crossbar array of a recursive neural network prior to operation of the recursive neural network; altering the weights according to a probability distribution; setting the weights to non-zero values in at least one of the memory cells in a diagonal of the memory cells in the crossbar array; and operating the recursive neural network.

First claim

What is claimed is:

1. An apparatus for processing a matrix of weights, the apparatus comprising:
a recursive neural network comprising a crossbar array, the crossbar array comprising a plurality of row lines, a plurality of column lines, and a plurality of memory cells each coupled between a respective combination of one of the row lines and one of the column lines, wherein:
the plurality of memory cells are configured to store the matrix of weights,
the column lines are configured to receive inputs according to an input vector, and
each row line is configured to deliver an output representing a dot-product of the input vector and the weights stored in the memory cells coupled to the row line; and a controller configured to:
prior to operation of the recursive neural network, receive the matrix of weights comprising zero values for the weights in the memory cells in a diagonal of the crossbar array, and set the weights to non-zero values in at least one of the memory cells in the diagonal of the crossbar array, and
adjust values of the weights in at least one of the memory cells in the diagonal of the crossbar array during operation of the recursive neural network,
wherein the non-zero values are set to introduce noise into the recursive neural network and to prevent convergence at a local minimum during operation of the recursive neural network.

2. The apparatus of claim 1 , wherein:
each of the memory cells in the diagonal of the crossbar array includes a memristor that fades to a zero value during operation of the recursive neural network.

3. The apparatus of claim 1 , wherein:
the controller is further configured to store the matrix of weights in the memory cells, and to alter the weights according to a probability distribution, prior to operation of the recursive neural network.

4. The apparatus of claim 3 , wherein:
the controller is further configured to adjust values of the weights stored in the memory cells of the crossbar array during operation of the recursive neural network.

5. The apparatus of claim 3 , wherein:
the controller is further configured to (i) record the altered values, and (ii) alter the weights stored in the memory cells of the crossbar array, according to the recorded values, during operation of the recursive neural network.

6. The apparatus of claim 3 , wherein:
the probability distribution is a Gaussian distribution.

7. The apparatus of claim 1 , wherein:
each of the memory cells includes a memristor.

8. The apparatus of claim 1 , wherein the recursive neural network further comprises:
a plurality of filters configured to generate a new input vector based on the outputs of the row lines.

9. An apparatus for processing a matrix of weights, the apparatus comprising:
a recursive neural network comprising a crossbar array, the crossbar array comprising a plurality of row lines, a plurality of column lines, and a plurality of memory cells each coupled between a respective combination of one of the row lines and one of the column lines, wherein:
the plurality of memory cells are configured to store the matrix of weights,
the column lines are configured to receive inputs according to an input vector, and
each row line is configured to deliver an output representing a dot-product of the input vector and the weights stored in the memory cells coupled to the row line; and a controller configured to:
store the matrix of weights in the memory cells, the matrix of weights representing a Hamiltonian matrix,
alter the weights in at least one of the memory cells in a diagonal of the crossbar array according to a probability distribution, prior to operation of the recursive neural network, and
adjust values of the weights in at least one of the memory cells in the diagonal of the crossbar array during operation of the recursive neural network,
wherein the weights are altered to introduce noise into the recursive neural network to converge at a global minimum during operation of the recursive neural network.

10. The apparatus of claim 9 , wherein:
the controller is further configured to (i) record the altered values, and (ii) alter the weights stored in the memory cells of the crossbar array, according to the recorded values, during operation of the recursive neural network.

11. The apparatus of claim 9 , wherein: the controller is further configured to set the weights to non-zero values in at least one of the memory cells in the diagonal of the crossbar array.

12. The apparatus of claim 11 , wherein:
the controller is further configured to adjust values of the weights in at least one of the memory cells in the diagonal of the crossbar array during operation of the recursive neural network.

13. The apparatus of claim 11 , wherein:
each of the memory cells in the diagonal of the crossbar array includes a memristor that fades to a zero value during operation of the recursive neural network.

14. The apparatus of claim 9 , wherein:
the probability distribution is a Gaussian distribution.

15. The apparatus of claim 9 , wherein:
each of the memory cells includes a memristor.

16. The apparatus of claim 9 , wherein the recursive neural network further comprises:
a plurality of filters configured to generate a new input vector based on the outputs of the row lines.

17. A method for processing a matrix of weights, the method comprising:
prior to operation of a recursive neural network,
storing the matrix of weights in memory cells of a crossbar array of a recursive neural network prior to operation of the recursive neural network, the matrix of weights comprising zero values for the weights in the memory cells in a diagonal of the crossbar array;
altering the weights according to a probability distribution;
setting the weights to non-zero values in at least one of the memory cells in the diagonal of the memory cells in the crossbar array;

operating the recursive neural network; and
adjusting values of the weights stored in at least one of the memory cells in the diagonal of the crossbar array during operation of the recursive neural network,
wherein the non-zero values are set to introduce noise into the recursive neural network and to prevent convergence at a local minimum during operation of the recursive neural network.

Assignees

• Hewlett Packard Entpr Dev Lp

Inventors

• Kumar Suhas
• Van Vaerenbergh Thomas
• Strachan John Paul

Classifications

• G06N3/0442

  characterised by memory or gating, e.g. long short-term memory [LSTM] or gated recurrent units [GRU] · CPC title

• G06N3/0495

  Quantised networks; Sparse networks; Compressed networks · CPC title

• G06N5/01

  Dynamic search techniques; Heuristics; Dynamic trees; Branch-and-bound · CPC title

• G06N3/044 Primary

  Recurrent networks, e.g. Hopfield networks · CPC title

• G06N3/065 Primary

  Analogue means · CPC title