Fixed, random, recurrent matrices for increased dimensionality in neural networks

Patent metadata

Field	Value
Publication number	US-12182719-B2
Application number	US-202016940857-A
Country	US
Kind code	B2
Filing date	Jul 28, 2020
Priority date	Jul 28, 2020
Publication date	Dec 31, 2024
Grant date	Dec 31, 2024

Abstract

A method of operating a neural network. The input layer of the network may have n input nodes connected to output nodes via a hidden layer. The hidden layer may include m hidden nodes. The n input nodes may connect to a subset of k nodes of the m hidden nodes via respective synaptic connections, to which training weights are associated, which form an n×k input matrix Win, whereas a subset of m−k nodes of the hidden layer are not connected by any node of the input layer. Running the network may include performing a first matrix vector multiplication between the input matrix Win and a vector of values obtained in output of the input nodes and a second matrix vector multiplication between a fixed matrix Wrec of fixed weights and a vector of values obtained in output of the m nodes of the hidden layer.

First claim

What is claimed is:

1. A computer-implemented method of operating a neural network, the method comprising:
training a neural network comprising successive layers, wherein:
the successive layers include an input layer, a hidden layer, and an output layer;
the input layer has n input nodes connected to output nodes of the output layer via the hidden layer;
the hidden layer including m hidden nodes, wherein the neural network is trained through connecting the n input nodes to a subset of k nodes of the m hidden nodes via respective synaptic connections to reduce number of learned parameters during the training, wherein each node in the subset of k nodes is associated with a training weight, which is used to form an n×k input matrix W in , wherein the training weights for the m hidden nodes are changeable during the training, and wherein a predetermined subset of m−k nodes of the hidden layer are not connected by any node of the input layer, wherein each node in the m−k nodes is associated with a fixed weight, wherein the fixed weights for the m−k nodes remain unchanged during the training; and
the neural network is set as a feed-forward neural network, the feed-forward neural network including a backpropagation learning algorithm, wherein the backpropagation learning algorithm comprises running the neural network so as to train the training weights of the n×k input matrix Win; and

coupling input data into input nodes of the input layer.

2. The computer-implemented method according to claim 1 , further comprising receiving a fixed matrix W rec of the fixed weights.

3. The computer-implemented method according to claim 1 , wherein output of the m hidden nodes is a vector.

4. The computer-implemented method according to claim 1 , wherein wherein the subset of k nodes are connected by the n input nodes of the input layer.

5. The computer-implemented method according to claim 4 , wherein output values y are obtained from input data x.

6. The computer-implemented method according to claim 1 , wherein said successive layers comprise several hidden layers, including said hidden layer, each configured as said hidden layer, whereby the input nodes are connected to the output nodes via the several hidden layers.

7. The computer-implemented method according to claim 1 , further comprising some of the fixed weights being random.

8. The computer-implemented method according to claim 7 , wherein a complementary subset of the fixed weights is random.

9. The computer-implemented method according to claim 7 , further comprising an echo state property.

10. The computer-implemented method according to claim 9 , wherein said fixed weights comprise a spectral radius that is strictly less than 1.

11. The computer-implemented method according to claim 1 , wherein
the neural network is run so as to train the training weights of the n×k input matrix W in , in a supervised manner.

12. The computer-implemented method according to claim 1 , wherein output values are inferred using a previous training.

13. The method according to claim 1 , wherein
said neural network is set as a multilayer perceptron or a convolutional neural network.

14. The method according to claim 1 , wherein
said neural network is set as an item-based autoencoder.

15. A computer program product for operating a neural network, the computer program product comprising a computer readable storage medium having program instructions embodied therewith, the program instructions executable by a processor, so as to cause the processor to:
train a neural network comprising successive layers, wherein:
the successive layers include an input layer, a hidden layer, and an output layer;
the input layer has n input nodes connected to output nodes of the output layer via the hidden layer;
the hidden layer including m hidden nodes;
and wherein the neural network is trained through connecting the n input nodes to a subset of k nodes of the m hidden nodes via respective synaptic connections to reduce number of learned parameters during the training, wherein each node in the subset of k nodes is associated with a training weight, which is used to form an n×k input matrix W in , wherein the training weights for the m hidden nodes are changeable during the training, and wherein a predetermined subset of m−k nodes of the hidden layer are not connected by any node of the input layer, wherein each node in the m−k nodes is associated with a fixed weight, wherein the fixed weights for the m−k nodes remain unchanged during the training; and the neural network is set as a feed-forward neural network, the feed-forward neural network including a backpropagation learning algorithm, wherein the backpropagation learning algorithm comprises running the neural network so as to train the training weights of the n×k input matrix Win; and

couple input data into input nodes of the input layer.

16. The computer program product of claim 15 , wherein input comprises a fixed matrix W rec of the fixed weights.

17. A neural network, comprising a processor operably connected to a memory, the memory containing program instructions to, when executed on the processor:
train a neural network comprising successive layers, wherein:
the successive layers include an input layer, a hidden layer, and an output layer;
the input layer has n input nodes connected to output nodes of the output layer via the hidden layer;
the hidden layer including m hidden nodes;
and wherein the neural network is trained through connecting the n input nodes to a subset of k nodes of the m hidden nodes via respective synaptic connections to reduce number of learned parameters during the training, wherein each node in the subset of k nodes is associated with a training weight, which is used to form an n×k input matrix W in , wherein the training weights for the m hidden nodes are changeable during the training, and wherein a predetermined subset of m−k nodes of the hidden layer are not connected by any node of the input layer, wherein each node in the m−k nodes is associated with a fixed weight, wherein the fixed weights for the m−k nodes remain unchanged during the training; and the neural network is set as a feed-forward neural network, the feed-forward neural network including a backpropagation learning algorithm, wherein the backpropagation learning algorithm comprises running the neural network so as to train the training weights of the n×k input matrix Win; and

couple input data into input nodes of the input layer.

18. The neural network of claim 17 , wherein the input comprises a fixed matrix W rec of the fixed weights.

19. The computer-implemented method according to claim 1 , wherein the backpropagation learning algorithm comprises only modifies the weights of the n×k input matrix W in .

20. The computer-implemented method according to claim 1 , wherein the backpropagation learning algorithm comprises does not modify the subset of m−k nodes of the hidden layer that are not connected by any node of the input layer.

Assignees

• IBM

Inventors

• Muller Lorenz K
• Stark Pascal
• Abel Stefan

Classifications

• G06N3/0495

  Quantised networks; Sparse networks; Compressed networks · CPC title

• G06N3/0455

  Auto-encoder networks; Encoder-decoder networks · CPC title

• G06N3/09

  Supervised learning · CPC title

• G06N3/0499

  Feedforward networks · CPC title

• G06N3/045

  Combinations of networks · CPC title