Efficient binary representations from neural networks

Patent metadata

Field	Value
Publication number	US-12277490-B2
Application number	US-202017270404-A
Country	US
Kind code	B2
Filing date	Apr 14, 2020
Priority date	Apr 14, 2020
Publication date	Apr 15, 2025
Grant date	Apr 15, 2025

Abstract

Persistent storage contains a representation of a neural network including an input layer, and output layer, and a hidden layer, wherein nodes of the hidden layer incorporate serialized activation functions, wherein the serialized activation functions for each of the nodes include a sigmoid function and a Beta function, wherein the sigmoid function is applied to weighted outputs from nodes of a previous layer of the neural network, wherein the Beta function is applied to a conductance hyper-parameter and respective outputs of the sigmoid function, and wherein outputs of the Beta function are provided to a subsequent layer of the neural network. One or more processors are configured to train the neural network until the outputs of the sigmoid function for the nodes of the hidden layer are substantially binary.

First claim

What is claimed is:

1. A system comprising:
persistent storage containing a representation of a neural-network-based encoder including an input layer and an output layer, wherein nodes of the output layer incorporate serialized activation functions, wherein the serialized activation functions for each of the nodes include a sigmoid function and a thresholding function, wherein the sigmoid function is applied to weighted outputs from nodes of a previous layer of the neural-network-based encoder, wherein the thresholding function is applied to outputs of the sigmoid function, wherein outputs of the thresholding function are binary, wherein the output layer was trained as a hidden layer of a neural-network-based auto-encoder, and wherein during training the thresholding function was replaced by a Beta function that was applied to a conductance hyper-parameter and respective outputs of the sigmoid function; and
one or more processors configured to:
introduce input to the input layer;
apply the serialized activation functions to the weighted outputs from the nodes of the previous layer; and
provide binary outputs from the output layer.

2. The system of claim 1 , wherein the previous layer is the input layer.

3. The system of claim 1 , wherein the neural-network-based auto-encoder was trained until the respective outputs of the sigmoid function for the nodes of the hidden layer were substantially binary, wherein being substantially binary comprises a first threshold percentage of the respective outputs of the sigmoid function for the nodes of the hidden layer being within a second threshold of 0 or within a third threshold of 1.

4. The system of claim 3 , wherein the first threshold percentage comprises at least 80% of the respective outputs being within a second threshold of 0 comprises being below 0.1, and being within a third threshold of 1 comprises being or above 0.9.

5. The system of claim 3 , wherein the first threshold percentage comprises at least 70% of the respective outputs being within a second threshold of 0 comprises being below 0.01, and being within a third threshold of 1 comprises being above 0.99.

6. The system of claim 1 , wherein, after training, an expected value of the outputs of the Beta function is within 1% of an expected value of the respective outputs of the sigmoid function for the nodes of the hidden layer.

7. The system of claim 1 , wherein parameters of the Beta function are: (i) the conductance hyper-parameter multiplied by the respective outputs of the sigmoid function, and (ii) the conductance hyper-parameter multiplied by a difference, wherein the difference is one minus the respective outputs of the sigmoid function.

8. The system of claim 1 , wherein the outputs of the Beta function are further from 0.5 than the respective outputs of the sigmoid function.

9. The system of claim 1 , wherein the hidden layer is one of a plurality of hidden layers in the neural-network-based auto-encoder.

10. A computer-implemented method comprising:
obtaining, by a computing system, a representation of a neural-network-based encoder including an input layer and an output layer, wherein nodes of the output layer incorporate serialized activation functions, wherein the serialized activation functions for each of the nodes include a sigmoid function and a thresholding function, wherein the sigmoid function is applied to weighted outputs from nodes of a previous layer of the neural-network-based encoder, wherein the thresholding function is applied to outputs of the sigmoid function, wherein outputs of the thresholding function are binary, wherein the output layer was trained as a hidden layer of a neural-network-based auto-encoder, and wherein during training the thresholding function was replaced by a Beta function that was applied to a conductance hyper-parameter and respective outputs of the sigmoid function;
introducing, by the computing system, input to the input layer;
applying, by the computing system, the serialized activation functions to the weighted outputs from the nodes of the previous layer; and
providing, by the computing system, binary outputs from the output layer.

11. The computer-implemented method of claim 10 , wherein the previous layer is the input layer.

12. The computer-implemented method of claim 10 , wherein the neural-network-based auto-encoder was trained until the respective outputs of the sigmoid function for the nodes of the hidden layer were substantially binary, wherein being substantially binary comprises a first threshold percentage of the respective outputs of the sigmoid function for the nodes of the hidden layer being within a second threshold of 0 or within a third threshold of 1.

13. The computer-implemented method of claim 12 , wherein the first threshold percentage comprises at least 80% of the respective outputs being within a second threshold of 0 comprises being below 0.1 and being within a third threshold of 1 comprises being above 0.9.

14. The computer-implemented method of claim 12 , wherein the first threshold percentage comprises at least 70% of the respective outputs being within a second threshold of 0 comprises being below 0.01 and being within a third threshold of 1 comprises being above 0.99.

15. The computer-implemented method of claim 10 , wherein, after training, an expected value of the outputs of the Beta function is within 1% of an expected value of the respective outputs of the sigmoid function for the nodes of the hidden layer.

16. The computer-implemented method of claim 10 , wherein parameters of the Beta function are: (i) the conductance hyper-parameter multiplied by the respective outputs of the sigmoid function, and (ii) the conductance hyper-parameter multiplied by a difference, wherein the difference is one minus the respective outputs of the sigmoid function.

17. The computer-implemented method of claim 10 , wherein the outputs of the Beta function are further from 0.5 than the respective outputs of the sigmoid function.

18. The computer-implemented method of claim 10 , wherein the hidden layer is one of a plurality of hidden layers in the neural-network-based auto-encoder.

19. An article of manufacture including a non-transitory computer-readable medium, having stored thereon program instructions that, upon execution by a computing device, cause the computing device to perform operations comprising:
obtaining a representation of a neural-network-based encoder including an input layer and an output layer, wherein nodes of the output layer incorporate serialized activation functions, wherein the serialized activation functions for each of the nodes include a sigmoid function and a thresholding function, wherein the sigmoid function is applied to weighted outputs from nodes of a previous layer of the neural-network-based encoder, wherein the thresholding function is applied to outputs of the sigmoid function, wherein outputs of the thresholding function are binary, wherein the output layer was trained as a hidden layer of a neural-network-based auto-encoder, and wherein during training the thresholding function was replaced by a Beta function that was applied to a conductance hyper-parameter and respective outputs of the sigmoid function;
introducing input to the input layer;
applying the serialized activation functions to the weighted outputs from the nodes of the previous layer; and
providing binary outputs from the output layer.

20. The article of manufacture of claim 19 , wherein the neural-network-based auto-encoder was trained until

Assignees

• Google Llc

Inventors

• Mcdonald Ryan
• Sarmento Luis

Classifications

• G06N3/0495

  Quantised networks; Sparse networks; Compressed networks · CPC title

• G06N3/09

  Supervised learning · CPC title

• G06N3/0455

  Auto-encoder networks; Encoder-decoder networks · CPC title

• G06N3/0499

  Feedforward networks · CPC title

• G06N3/088

  Non-supervised learning, e.g. competitive learning · CPC title