Systems and methods for employing predication in computational models

Patent metadata

Field	Value
Publication number	US-11264011-B2
Application number	US-202016749328-A
Country	US
Kind code	B2
Filing date	Jan 22, 2020
Priority date	Dec 29, 2017
Publication date	Mar 1, 2022
Grant date	Mar 1, 2022

Abstract

The disclosed method may include (1) determining whether a next operation of a plurality of operations of an artificial neural network (ANN) is dependent upon a Boolean predication value based on a representative value for a weight or an input of a node of the ANN, (2) based on the next operation not being dependent on the Boolean predication value, allowing the next operation to update a state of the ANN, and (3) based on the next operation being dependent on the Boolean predication value, performing at least one of (a) allowing, based on the Boolean predication value being a first value, the next operation to update the state of the ANN, and (b) preventing, based on the Boolean predication value being a second value different from the first value, the next operation from updating the state of the ANN. Various other methods and systems are also disclosed.

First claim

What is claimed is:

1. A method comprising:
setting, using at least one hardware processor, as a result of a previous operation of a plurality of operations of an artificial neural network (ANN), a Boolean predication value to a first value or a second value different from the first value, wherein the previous operation comprises comparing a representative value for one of a weight or an input of a node of the ANN to a comparison value;
determining whether a next operation of the plurality of operations of the ANN is dependent upon the Boolean predication value;
allowing, based on the next operation not being dependent on the Boolean predication value, the next operation to update a state of the ANN; and
based on the next operation being dependent on the Boolean predication value:
allowing, based on the Boolean predication value being the first value, the next operation to update the state of the ANN; and
preventing, based on the Boolean predication value being the second value, the next operation from updating the state of the ANN.

2. The method of claim 1 , wherein:
the representative value of the one of a weight or an input of the node comprises an actual value of a weight of the node; and
the next operation comprises a multiply operation of the actual value of the weight and a value of an input associated with the weight.

3. The method of claim 1 , wherein:
the representative value of the one of a weight or an input of the node comprises an actual value of an input of the node; and
the next operation comprises a multiply operation of the actual value of the input and a value of a weight associated with the input.

4. The method of claim 1 , wherein:
the representative value of the one of a weight or an input of the node comprises an absolute value of a weight of the node; and
the next operation comprises a multiply operation of an actual value of the weight and a value of an input associated with the weight.

5. The method of claim 1 , wherein:
the representative value of the one of a weight or an input of the node comprises an absolute value of an input of the node; and
the next operation comprises a multiply operation of an actual value of the input and a value of a weight associated with the input.

6. The method of claim 1 , wherein:
the representative value of the one of a weight or an input of the node comprises an actual value of a weight of the node; and
the next operation comprises a summing operation including a product of the actual value of the weight and a value of an input associated with the weight that is part of an accumulate operation.

7. The method of claim 1 , wherein:
the representative value of the one of a weight or an input of the node comprises an actual value of an input of the node; and
the next operation comprises a summing operation including a product of the actual value of the input and a value of a weight associated with the input that is part of an accumulate operation.

8. The method of claim 1 , wherein:
the representative value of the one of a weight or an input of the node comprises an absolute value of a weight of the node; and
the next operation comprises a summing operation including a product of an actual value of the weight and a value of an input associated with the weight that is part of an accumulate operation.

9. The method of claim 1 , wherein:
the representative value of the one of a weight or an input of the node comprises an absolute value of an input of the node; and
the next operation comprises a summing operation including a product of an actual value of the input and a value of a weight associated with the input that is part of an accumulate operation.

10. The method of claim 1 , wherein the comparison value is zero.

11. The method of claim 10 , wherein:
the Boolean predication value is set to the first value when the representative value is not equal to the comparison value; and
the Boolean predication value is set to the second value when the representative value is equal to the comparison value.

12. The method of claim 1 , wherein the comparison value is non-zero.

13. The method of claim 12 , wherein:
the Boolean predication value is set to the first value when the representative value is greater than or equal to the comparison value; and
the Boolean predication value is set to the second value when the representative value is less than the comparison value.

14. The method of claim 1 , wherein the input of the node comprises an output of an activation operation of another node of the ANN.

15. The method of claim 1 , further comprising:
performing the next operation prior to evaluating the Boolean predication value;
saving, based on the next operation being dependent on the Boolean predication value, a result of performing the next operation without updating the state of the ANN; and
updating, based on the Boolean predication value being the first value, the state of the ANN using the result of performing the next operation.

16. The method of claim 1 , further comprising:
performing, based on the next operation being dependent on the Boolean predication value, the next operation after determining the Boolean predication value is the first value; and
skipping, based on the next operation being dependent on the Boolean predication value, the next operation after determining the Boolean predication value is the second value.

17. The method of claim 1 , further comprising:
based on another operation of the plurality of operations being dependent on the Boolean predication value:
allowing, based on the Boolean predication value being the second value, the other operation to update the state of the ANN; and
preventing, based on the Boolean predication value being the first value, the other operation from updating the state of the ANN.

18. The method of claim 17 , wherein performance of the next operation and the other operation at least partially overlap in time.

19. A hardware processing element comprising:
an operation unit that performs a plurality of operations associated with nodes of an artificial neural network (ANN); and
a predication unit that:
sets, as a result of a previous operation of the plurality of operations, a Boolean predication value to a first value or a second value different from the first value, wherein the previous operation comprises comparing a representative value for one of a weight or an input of a node of the ANN to a comparison value;
determines whether a next operation of the plurality of operations is dependent upon the Boolean predication value;
based on the next operation not being dependent on the Boolean predication value, allows the next operation to update a state of the ANN; and
based on the next operation being dependent on the Boolean predication value:
allows, based on the Boolean predication value being the first value, the next operation to update the state of the ANN; and
prevents, based on the Boolean predication value being the second value, the next operation from updating the state of the ANN.

20. A system comprising:
a plurality of hardware processing elements that process nodes of an artificial neural network (ANN), wherein at least one of the hardware processing elements comprises:
an operation unit that performs a plurality of operations associated with the nodes of the ANN; and
a predication unit that:
sets, as a result of a previous operation of the plurality of operations, a Boolean predication value to a first value or a second valu

Assignees

• Facebook Inc

Inventors

• Rotem Nadav
• Diril Abdulkadir Utku
• Smelyanskiy Mikhail
• Park Jong Soo
• Reed James Kenneth

Classifications

• G06N3/048

  Activation functions · CPC title

• G06N3/044

  Recurrent networks, e.g. Hopfield networks · CPC title

• G06N3/063 Primary

  using electronic means · CPC title

• G06N3/0499

  Feedforward networks · CPC title

• G10L15/16 Primary

  using artificial neural networks · CPC title