Computational efficiency improvements for artificial neural networks

Patent metadata

Field	Value
Publication number	US-11640522-B2
Application number	US-201916712954-A
Country	US
Kind code	B2
Filing date	Dec 12, 2019
Priority date	Dec 13, 2018
Publication date	May 2, 2023
Grant date	May 2, 2023

Abstract

An artificial neural network (ANN) generates a base expanded matrix that represents an output of a layer of the ANN, such as the output layer. Values in each row are grouped with respect to a set of network parameters in a previous layer, and a sum of the values in each row produces an output vector of activations. The ANN updates the values in at least one column of the expanded matrix according to parameter updates, which results in an updated expanded matrix or an update expanded matrix. An error or a total cost can be computed from the updated expanded matrix or the update expanded matrix. Nonlinear activation functions can be modeled as piecewise linear functions, and a change in an activation function's slope can be modeled as a linear update to an expanded matrix. Parameter updates can be constrained to a restricted value set in order to simplify update operations performed on the expanded matrices.

First claim

The invention claimed is:

1. An apparatus for an artificial neural network (ANN), comprising: one or more processors, coupled to a memory that includes instructions to execute operations of the one or more processors, configured to:
generate a base expanded matrix having a plurality of rows and a plurality of columns, the base expanded matrix representing an output of a layer of the ANN, wherein a sum of values in each row produces a base output vector of activations;
update values in at least one column of the base expanded matrix to produce an updated expanded matrix or an update expanded matrix; and
compute at least one of an error and a total cost from the updated expanded matrix or the update expanded matrix.

2. The apparatus of claim 1 , wherein the base expanded matrix comprises parameters from one or more previous layers.

3. The apparatus of claim 1 , wherein the at least one column is selected based on an update to a parameter in a previous layer of the ANN.

4. The apparatus of claim 1 , wherein the updated expanded matrix is computed from at least one of an additive update or a multiplicative update to the base expanded matrix.

5. The apparatus of claim 1 , wherein each column of the base expanded matrix comprises coefficients corresponding to one of a set of ANN parameters, or each column of the base expanded matrix comprises products of the coefficients with the one of the set of ANN parameters.

6. The apparatus of claim 1 , wherein each value in the base expanded matrix is computed numerically, and the update expanded matrix or the updated expanded matrix is computed by constraining one or more update values to a restricted set of values such that updates to each value in the base expanded matrix can comprise only one or more of shifting the base expanded matrix value's bits, changing the base expanded matrix value's sign bit, deleting the base expanded matrix value, and changing the base expanded matrix value's decimal point.

7. The apparatus of claim 1 , wherein the ANN comprises multiple electronic synapses connecting multiple electronic neurons.

8. The apparatus of claim 1 , further comprising instructions to execute operations of the one or more processors to sum values in each row of the updated expanded matrix to produce an updated output vector.

9. A computer program product, comprising a non-transitory computer-readable memory having computer-readable program code stored thereon, the computer-readable program code containing instructions executable by one or more processors in an artificial neural network (ANN) to:
generate a base expanded matrix having a plurality of rows and a plurality of columns, the base expanded matrix representing an output of a layer of the ANN, wherein a sum of values in each row produces a base output vector of activations;
update values in at least one column of the base expanded matrix to produce an updated expanded matrix or an update expanded matrix; and
compute at least one of an error and a total cost from the updated expanded matrix or the update expanded matrix.

10. The computer program product of claim 9 , wherein the base expanded matrix comprises parameters from one or more previous layers.

11. The computer program product of claim 9 , wherein the at least one column is selected based on an update to a parameter in a previous layer of the ANN.

12. The computer program product of claim 9 , wherein the updated expanded matrix is computed from at least one of an additive update or a multiplicative update to the base expanded matrix.

13. The computer program product of claim 9 , wherein each column of the base expanded matrix comprises coefficients corresponding to one of a set of ANN parameters, or each column of the base expanded matrix comprises products of the coefficients with the one of the set of ANN parameters.

14. The computer program product of claim 9 , wherein each value in the base expanded matrix is computed numerically, and the update expanded matrix or the updated expanded matrix is computed by constraining one or more update values to a restricted set of values such that updates to each value in the base expanded matrix can comprise only one or more of shifting the base expanded matrix value's bits, changing the base expanded matrix value's sign bit, deleting the base expanded matrix value, and changing the base expanded matrix value's decimal point.

15. The computer program product of claim 9 , wherein the ANN comprises multiple electronic synapses connecting multiple electronic neurons.

16. The computer program product of claim 9 , further comprising instructions executable by one or more processors to sum values in each row of the updated expanded matrix to produce an updated output vector.

17. A method for updating an artificial neural network (ANN), comprising:
generating a base expanded matrix having a plurality of rows and a plurality of columns, the base expanded matrix representing an output of a layer of the ANN, wherein a sum of values in each row produces a base output vector;
updating values in at least one column of the base expanded matrix to produce an updated expanded matrix or an update expanded matrix; and
computing at least one of an error and a total cost from the updated expanded matrix or the update expanded matrix.

18. The method of claim 17 , wherein the base expanded matrix comprises parameters from one or more previous layers.

19. The method of claim 17 , wherein the at least one column is selected based on an update to a parameter in a previous layer of the ANN.

20. The method of claim 17 , wherein the updated expanded matrix is computed from a multiplicative update of the base expanded matrix or an additive update produced by summing the base expanded matrix with the update expanded matrix.

21. The method of claim 17 , wherein each column of the base expanded matrix comprises coefficients corresponding to one of a set of ANN parameters, or each column of the base expanded matrix comprises products of the coefficients with the one of the set of ANN parameters.

22. The method of claim 17 , wherein each value in the base expanded matrix is computed numerically, and the update expanded matrix or the updated expanded matrix is computed by constraining one or more update values to a restricted set of values such that updates to each value in the base expanded matrix can comprise only one or more of shifting the base expanded matrix value's bits, changing the base expanded matrix value's sign bit, deleting the base expanded matrix value, and changing the base expanded matrix value's decimal point.

23. The method of claim 17 , wherein the ANN comprises multiple electronic synapses connecting multiple electronic neurons.

24. The method of claim 17 , further comprising summing values in each row of the updated expanded matrix to produce an updated output vector.

Assignees

• Genghiscomm Holdings Llc
• Tybalt Llc

Inventors

• Shattil Steve

Classifications

• G06N3/0495

  Quantised networks; Sparse networks; Compressed networks · CPC title

• G06N3/0464

  Convolutional networks [CNN, ConvNet] · CPC title

• G06N3/0455

  Auto-encoder networks; Encoder-decoder networks · CPC title

• G06N3/09

  Supervised learning · CPC title

• G06N3/0985

  Hyperparameter optimisation; Meta-learning; Learning-to-learn · CPC title