Circuit architecture with biased randomization

Patent metadata

Field	Value
Publication number	US-11250319-B1
Application number	US-201715714924-A
Country	US
Kind code	B1
Filing date	Sep 25, 2017
Priority date	Sep 25, 2017
Publication date	Feb 15, 2022
Grant date	Feb 15, 2022

Abstract

Disclosed herein are techniques for classifying data with a data processing circuit. In one embodiment, the data processing circuit includes a probabilistic circuit configurable to generate a decision at a pre-determined probability, and an output generation circuit including an output node and configured to receive input data and a weight, and generate output data at the output node for approximating a product of the input data and the weight. The generation of the output data includes propagating the weight to the output node according a first decision of the probabilistic circuit. The probabilistic circuit is configured to generate the first decision at a probability determined based on the input data.

First claim

What is claimed is:

1. An integrated circuit, comprising:
a probabilistic circuit configurable to generate a decision at a given probability;
an output generation circuit including an output node and configured to:
receive input data and a weight; and
generate output data at the output node for approximating a product of the input data and the weight;

wherein the generation of the output data includes propagating the weight to the output node according to a first decision of the probabilistic circuit; and
wherein the probabilistic circuit is configured to generate the first decision at a probability determined based on the input data.

2. The integrated circuit of claim 1 , wherein the probabilistic circuit comprises:
a pseudo-random number generator configured to generate a first number;
a threshold generator configured to generate a second number based on the input data; and
a comparator configured to generate the decision by comparing the first number and the second number.

3. The integrated circuit of claim 2 , wherein the pseudo-random number generator is configured to generate the first number within a range;
wherein the threshold generator is configured to generate the second number as a fraction of the range;
wherein the fraction is determined based on the input data.

4. The integrated circuit of claim 1 , wherein the output generation circuit is further configured to:
generate intermediate data based on accumulating the propagated weight; and
generate the output data by inputting the intermediate data to a first activation function.

5. The integrated circuit of claim 4 , wherein the first activation function is represented as a mapping table.

6. The integrated circuit of claim 4 , wherein the input data comprises an input sample associated with a first time stamp and an output of the integrated circuit associated with a second time stamp;
wherein the second time stamp precedes the first time stamp.

7. The integrated circuit of claim 6 , wherein the output generation circuit is further configured to:
receive a first set of weights associated with a forget factor;
selectively propagate, based on the first decision, the first set of weights to a first accumulator to generate first accumulator data; and
generate, based on the first accumulator data and a second activation function, the forget factor.

8. The integrated circuit of claim 7 , wherein the output generation circuit is further configured to:
receive a second set of weights associated with an input factor;
selectively propagate, based on the first decision, the second set of weights to a second accumulator to generate second accumulator data; and
generate, based on the second accumulator data and a third activation function, the input factor.

9. The integrated circuit of claim 8 , wherein the output generation circuit is further configured to:
receive a third set of weights associated with a candidate state;
selectively propagate, based on the first decision, the third set of weights to a third accumulator to generate third accumulator data; and
generate, based on the third accumulator data and a fourth activation function, the candidate state.

10. The integrated circuit of claim 9 ,
wherein the probabilistic circuit is configured to generate a second decision at a second probability determined based on the forget factor and the input factor;
wherein the output generation circuit is further configured to:
receive a previous state associated with the second time stamp;
selectively propagate, based on the second decision, the candidate state and the previous state to a fourth accumulator to generate fourth accumulator data;
generate the output data based on the fourth accumulator data; and
associate the output data with the first time stamp.

11. A method, comprising:
receiving, by an output generation circuit, input data and a weight, wherein the output generation circuit includes an output node;
generating, by the output generation circuit, output data at the output node for approximating a product of the input data and the weight; and
generating, by a probabilistic circuit, a first decision at a probability determined based on the input data, wherein the generation of the output data by the output generation circuit includes propagating the weight to the output node according to the first decision of the probabilistic circuit, and wherein the output generation circuit and the probabilistic circuit are elements of an integrated circuit.

12. The method of claim 11 , further comprising:
generating, by the output generation circuit, intermediate data based on accumulating the propagated weight; and
generating, by the output generation circuit, the output data by inputting the intermediate data to a first activation function.

13. The method of claim 12 , wherein the first activation function is represented as a mapping table.

14. The method of claim 12 , wherein the input data comprises an input sample associated with a first time stamp and an output of the integrated circuit associated with a second time stamp, and wherein the second time stamp precedes the first time stamp.

15. The method of claim 14 , further comprising:
receiving, by the output generation circuit, a first set of weights associated with a forget factor;
selectively propagating, by the output generation circuit, based on the first decision, the first set of weights to a first accumulator to generate first accumulator data; and
generating, by the output generation circuit, based on the first accumulator data and a second activation function, the forget factor.

16. An integrated circuit for implementing a neural network, the integrated circuit comprising:
random decision generator circuits to generate a decision at a probability that corresponds to a configuration value;
a historical state processor configured to:
generate a forget factor based on probabilistic propagations of a first set of weights, the probabilistic propagations of the first set of weights being performed based on decisions of the random decision generator circuits with configuration values comprising a current input sample and a previous classifier output from the integrated circuit, and
approximate scaling of a prior internal state by the forget factor based on probabilistic propagations of the prior internal state, the probabilistic propagations of the prior internal state being performed based on decisions of the random decision generator circuits with configuration values comprising the forget factor;

a candidate internal state generator configured to generate a candidate state based on probabilistic propagations of a second set of weights, the probabilistic propagations of the second set of weights being performed based on decisions of the random decision generator circuits with configuration values comprising the current input sample and the previous classifier output;
an internal state processor configured to:
generate an input factor based on probabilistic propagations of a third set of weights, the probabilistic propagations of the third set of weights being performed based on decisions of the random decision generator circuits with configuration values comprising the current input sample and the previous classifier output, and
generate a current internal state based on probabilistic propagations of the candidate internal state generator and based on the approximate scaling of the prior internal state provided by the historical state processor, the probabilistic propagations of the candidate state being performed based on d

Assignees

• Amazon Tech Inc

Inventors

• Huang Randy
• Diamant Ron

Classifications

• G06N7/01

  Probabilistic graphical models, e.g. probabilistic networks · CPC title

• G06N3/044

  Recurrent networks, e.g. Hopfield networks · CPC title

• G06N3/047

  Probabilistic or stochastic networks · CPC title

• G06N3/048

  Activation functions · CPC title

• G06N3/063 Primary

  using electronic means · CPC title