Deep neural network architecture using piecewise linear approximation

Patent metadata

Field	Value
Publication number	US-11775805-B2
Application number	US-201816023441-A
Country	US
Kind code	B2
Filing date	Jun 29, 2018
Priority date	Jun 29, 2018
Publication date	Oct 3, 2023
Grant date	Oct 3, 2023

Abstract

A log circuit for piecewise linear approximation is disclosed. The log circuit identifies an input associated with a logarithm operation to be performed using piecewise linear approximation. The log circuit then identifies a range that the input falls within from various ranges associated with piecewise linear approximation (PLA) equations for the logarithm operation, where the identified range corresponds to one of the PLA equations. The log circuit computes a result of the corresponding PLA equation based on the respective operands of the equation. The log circuit then returns an output associated with the logarithm operation, which is based at least partially on the result of the PLA equation.

First claim

What is claimed is:

1. An activation function circuit, comprising:
decode circuitry to decode a first instruction to perform a first activation function, wherein the first activation function is one of a plurality of activation functions implemented on the activation function circuit, and wherein the first activation function includes an exponent operation and a division operation;
exponent circuitry to perform the exponent operation using piecewise linear approximation;
log circuitry to perform logarithm operations on a numerator and a denominator of the division operation using piecewise linear approximation;
subtractor circuitry to perform a subtraction operation on results of the logarithm operations;
antilog circuitry to perform an antilogarithm operation on a result of the subtraction operation using piecewise linear approximation; and
output circuitry to output a result of the first activation function based on a result of the antilogarithm operation.

2. The activation function circuit of claim 1 , wherein:
the first activation function further includes a multiplication operation;
the log circuitry is further to perform logarithm operations on operands of the multiplication operation using piecewise linear approximation;
the activation function circuit further comprises adder circuitry to perform an addition operation on results of the logarithm operations on the operands of the multiplication operation; and
the antilog circuitry is further to perform an antilogarithm operation on a result of the addition operation using piecewise linear approximation, wherein a result of the antilogarithm operation is a result of the multiplication operation.

3. The activation function circuit of claim 1 , wherein:
the exponent operation has a base of 2; and
the exponent circuitry to perform the exponent operation using piecewise linear approximation is further to perform the exponent operation using an antilogarithm base 2 operation, wherein the antilogarithm base 2 operation is performed using piecewise linear approximation.

4. The activation function circuit of claim 1 , wherein the first activation function is:
a sigmoid function;
a hyperbolic tangent function; or
a swish function.

5. The activation function circuit of claim 1 , wherein the plurality of activation functions include:
a sigmoid function;
a hyperbolic tangent function;
a swish function; and
a rectified linear unit function.

6. The activation function circuit of claim 5 , wherein:
at least one of the sigmoid function, the hyperbolic tangent function, or the swish function is defined using one or more exponent operations that exclusively have a base of 2, wherein the one or more exponent operations include the exponent operation; and
the exponent circuitry to perform the exponent operation using piecewise linear approximation is further to perform the one or more exponent operations using one or more antilogarithm base 2 operations, wherein the one or more antilogarithm base 2 operations are performed using piecewise linear approximation.

7. The activation function circuit of claim 1 , wherein the first instruction includes an opcode identifying the first activation function from the plurality of activation functions implemented on the activation function circuit.

8. The activation function circuit of claim 1 , wherein the plurality of activation functions are associated with one or more artificial neural networks.

9. At least one non-transitory machine-accessible storage medium having instructions stored thereon, wherein the instructions, when implemented or executed on processing circuitry comprising exponent circuitry, log circuitry, antilog circuitry, and subtractor circuitry, cause the processing circuitry to:
receive a first instruction to perform a first activation function, wherein the first activation function is one of a plurality of activation functions implemented on the processing circuitry, and wherein the first activation function includes an exponent operation and a division operation;
perform, using the exponent circuitry, the exponent operation using piecewise linear approximation;
perform, using the log circuitry, logarithm operations on a numerator and a denominator of the division operation using piecewise linear approximation;
perform, using the subtractor circuitry, a subtraction operation on results of the logarithm operations;
perform, using the antilog circuitry, an antilogarithm operation on a result of the subtraction operation using piecewise linear approximation; and
output a result of the first activation function based on a result of the antilogarithm operation.

10. The storage medium of claim 9 , wherein:
the first activation function further includes a multiplication operation;
the processing circuitry further comprises adder circuitry; and
the instructions further cause the processing circuitry to:
perform, using the log circuitry, logarithm operations on operands of the multiplication operation using piecewise linear approximation;
perform, using the adder circuitry, an addition operation on results of the logarithm operations on the operands of the multiplication operation;
perform, using the antilog circuitry, an antilogarithm operation on a result of the addition operation using piecewise linear approximation; and
return a result of the multiplication operation based on a result of the antilogarithm operation on the result of the addition operation.

11. The storage medium of claim 9 , wherein:
the exponent operation has a base of 2; and
the instructions that cause the processing circuitry to perform, using the exponent circuitry, the exponent operation using piecewise linear approximation further cause the processing circuitry to:
perform, using the exponent circuitry, the exponent operation using an antilogarithm base 2 operation, wherein the antilogarithm base 2 operation is performed using piecewise linear approximation.

12. The storage medium of claim 9 , wherein:
the first activation function is defined using one or more exponent operations that exclusively have a base of 2, wherein the one or more exponent operations include the exponent operation; and
the instructions that cause the processing circuitry to perform, using the exponent circuitry, the exponent operation using piecewise linear approximation further cause the processing circuitry to:
perform, using the exponent circuitry, the one or more exponent operations using one or more antilogarithm base 2 operations, wherein the one or more antilogarithm base 2 operations are performed using piecewise linear approximation.

13. The storage medium of claim 9 , wherein the first activation function is:
a sigmoid function;
a hyperbolic tangent function; or
a swish function.

14. The storage medium of claim 9 , wherein the plurality of activation functions include:
a sigmoid function;
a hyperbolic tangent function;
a swish function; and
a rectified linear unit function.

15. The storage medium of claim 9 , wherein the plurality of activation functions are associated with one or more artificial neural networks.

16. A system, comprising:
a processor; and
an activation function circuit, comprising:
decode circuitry to decode a first instruction to perform a first activation function, wherein the first activation function is one of a plurality of activation functions implemented on the activation function circuit, and wherein the first activation function includes an exponent operation and a division operation;
exponent circuitry to perform the exponent operation using pi

Assignees

• Intel Corp
• Intel Coroporation

Inventors

• Pillai Kamlesh
• Kalsi Gurpreet S
• Mishra Amit

Classifications

• G06N3/0464

  Convolutional networks [CNN, ConvNet] · CPC title

• G06N3/09

  Supervised learning · CPC title

• G06N3/0499

  Feedforward networks · CPC title

• G06F7/556 Primary

  Logarithmic or exponential functions · CPC title

• G06N3/063 Primary

  using electronic means · CPC title