Lookup table activation functions for neural networks

Patent metadata

Field	Value
Publication number	US-12189599-B2
Application number	US-202017028920-A
Country	US
Kind code	B2
Filing date	Sep 22, 2020
Priority date	Jun 19, 2020
Publication date	Jan 7, 2025
Grant date	Jan 7, 2025

Abstract

The subject technology provides a framework for evaluating activation functions of a neural network using lookup tables. In order to provide lookup table based activation functions with a desired precision within hardware constraints for the lookup tables, multiple lookup tables for each activation function can be provided. Each of the multiple lookup tables may correspond to a respective subrange of input values, within a full range of input values for the activation function.

First claim

What is claimed is:

1. A method of computing an output of a neural network, comprising:
obtaining an input value for an activation function associated with a node of the neural network;
obtaining, based on the input value, a lookup table value corresponding to each of a plurality of lookup tables that correspond to the activation function associated with the node of the neural network; and
determining a value of the activation function for the input value by combining the obtained lookup table values corresponding to the plurality of lookup tables.

2. The method of claim 1 , wherein the activation function has a corresponding range of input values, and wherein each of the plurality of lookup tables corresponds to a subrange of the range of input values.

3. The method of claim 1 , further comprising evaluating the neural network, using the determined value of the activation function, to obtain the output.

4. The method of claim 1 , wherein obtaining the lookup table value corresponding to each of the plurality of lookup tables based on the input value comprises:
interpolating between at least two lookup table entries from a first one of the plurality of lookup tables to obtain a first one of the lookup table values.

5. The method of claim 4 , wherein obtaining the lookup table value corresponding to each of the plurality of lookup tables based on the input value further comprises:
obtaining a saturation value for a second one of the plurality of lookup tables as a second one of the lookup table values.

6. The method of claim 5 , wherein combining the obtained lookup table values comprises adding the first one of the lookup table values and the second one of the lookup table values.

7. The method of claim 1 , wherein the activation function is a symmetric function of a range of input values, and wherein obtaining the lookup table value corresponding to at least one of the plurality of lookup tables comprises applying a symmetry function to a lookup table entry in the at least one of the plurality of lookup tables.

8. A device, comprising:
a memory storing a plurality of lookup tables corresponding to an activation function associated with a node of a neural network; and
a neural processor configured to:
obtain an input value for the activation function associated with the node of the neural network;
obtain, based on the input value, a lookup table value corresponding to each of a plurality of lookup tables that correspond to the activation function associated with the node of the neural network; and
determine a value of the activation function for the input value by combining the obtained lookup table values corresponding to the plurality of lookup tables.

9. The device of claim 8 , wherein the neural processor is configured to obtain the lookup table value corresponding to each of the plurality of lookup tables based on the input value by:
interpolating between at least two lookup table entries from a first one of the plurality of lookup tables to obtain a first one of the lookup table values; and
obtaining a saturation value for a second one of the plurality of lookup tables as a second one of the lookup table values.

10. The device of claim 9 , wherein the neural processor is configured to combine the obtained lookup table values by adding the first one of the lookup table values and the second one of the lookup table values.

11. The device of claim 8 , wherein the activation function has a corresponding range of input values, and wherein each of the plurality of lookup tables corresponds to a subrange of the range of input values.

12. The device of claim 8 , wherein the neural processor is further configured to evaluate the neural network, using the determined value of the activation function, to obtain the output.

13. A non-transitory computer-readable medium comprising instructions, which when executed by a computing device, cause the computing device to perform operations comprising:
obtaining, by a neural processor, an input value for an activation function associated with a node of the neural network;
obtaining, by the neural processor based on the input value, a lookup table value corresponding to each of a plurality of lookup tables that correspond to the activation function associated with the node of the neural network; and
determining, by the neural processor, a value of the activation function for the input value by combining the obtained lookup table values corresponding to the plurality of lookup tables.

14. The non-transitory computer-readable medium of claim 13 , wherein the activation function has a corresponding range of input values, and wherein each of the plurality of lookup tables corresponds to a subrange of the range of input values.

15. The non-transitory computer-readable medium of claim 14 , wherein the subrange of the range of input values for each of the plurality of lookups table are evenly spaced within the subrange for that lookup table.

16. The non-transitory computer-readable medium of claim 13 , wherein the operations further comprise: evaluating the neural network, using the determined value of the activation function, to obtain the output.

17. The non-transitory computer-readable medium of claim 13 , wherein obtaining the lookup table value corresponding to each of the plurality of lookup tables based on the input value comprises:
interpolating between at least two lookup table entries from a first one of the plurality of lookup tables to obtain a first one of the lookup table values.

18. The non-transitory computer-readable medium of claim 17 , wherein obtaining the lookup table value corresponding to each of the plurality of lookup tables based on the input value further comprises:
obtaining a saturation value for a second one of the plurality of lookup tables as a second one of the lookup table values.

19. The non-transitory computer-readable medium of claim 18 , wherein combining the obtained lookup table values comprises adding the first one of the lookup table values and the second one of the lookup table values.

20. The non-transitory computer-readable medium of claim 13 , wherein the activation function is a symmetric function of a range of input values, and wherein obtaining the lookup table value corresponding to at least one of the plurality of lookup tables comprises applying a symmetry function to a lookup table entry in the at least one of the plurality of lookup tables.

Assignees

• Apple Inc

Inventors

• Antony Albert
• Rossi Francesco
• Tartavel Guillaume
• Shi Xiaojin
• Zuliani Marco

Classifications

• G06N3/0499

  Feedforward networks · CPC title

• G06N3/04

  Architecture, e.g. interconnection topology · CPC title

• G06N3/048

  Activation functions · CPC title

• G06F16/2282 Primary

  Tablespace storage structures; Management thereof · CPC title

• G06N3/063 Primary

  using electronic means · CPC title