Techniques for conformance testing computational operations

Patent metadata

Field	Value
Publication number	US-11704231-B2
Application number	US-201916523732-A
Country	US
Kind code	B2
Filing date	Jul 26, 2019
Priority date	Jul 26, 2019
Publication date	Jul 18, 2023
Grant date	Jul 18, 2023

Abstract

Examples described herein generally relate to performing conformance testing of a computational operation. A reference result including one or more reference intermediate products and a reference accumulator output at a first level of precision can be generated for the computational operation and based on one or more inputs. A hardware result can similarly be created using hardware at a second level of precision. The reference result can be compared to the hardware result to determine a variance value. A conformance result can be output based on whether the variance value is within a threshold range.

First claim

What is claimed is:

1. A computer-implemented method for performing conformance testing of a machine learning (ML) computational operation configured to be implemented in a graphics processing unit (GPU) wherein the ML computational operation is performed by an algorithm specific to a hardware device, comprising:
generating, for the ML computational operation and based on one or more inputs, a reference result including one or more reference intermediate products and a reference accumulator output at a first level of precision, wherein the one or more reference intermediate products of the ML computational operation are within one or more ranges including a range where all values within the range are within a same single floating point exponent;
generating, for the ML computational operation and based on specifying, to the GPU, the one or more inputs and a second level of precision, a result for the hardware device including one or more hardware intermediate products and a hardware accumulator output using the GPU at the second level of precision; and
outputting a conformance result based on whether a variance between the reference result and the result is within a threshold range.

2. The computer-implemented method of claim 1 , wherein determining the one or more inputs comprises determining the one or more inputs as in the range of 1.0 and a square root of 2.0.

3. The computer-implemented method of claim 1 , wherein the one or more ranges include a first range with an exponent of zero, a second range with a minimum exponent, and a third range with a maximum exponent, and wherein generating the reference result, generating the result, and comparing the reference result to the generated result are performed for each of the one or more ranges.

4. The computer-implemented method of claim 1 , wherein the conformance result includes an indication of whether the generated result passes a conformance test.

5. The computer-implemented method of claim 1 , wherein generating the reference result, generating the result, and comparing the reference result to the generated result are performed for multiple configurations for the ML computational operation to output conformance results for each of the configurations.

6. The computer-implemented method of claim 5 , wherein the multiple configurations include:
a first configuration having a single precision to represent values of the one or more inputs, a single precision to represent values of the one or more hardware intermediate products, a single precision to represent values of an accumulator that generates the hardware accumulator output, and a single precision to represent values of the hardware accumulator output;
a second configuration having a half precision to represent values of the one or more inputs, a half precision to represent values of the one or more hardware intermediate products, a half precision to represent values of an accumulator that generates the hardware accumulator output, and a half precision to represent values of the hardware accumulator output;
a third configuration having a half precision to represent values of the one or more inputs, a half precision to represent values of the one or more hardware intermediate products, a full precision to represent values of an accumulator that generates the hardware accumulator output, and a half precision to represent values of the hardware accumulator output; and
a fourth configuration having a half precision to represent values of the one or more inputs, a full precision to represent values of the one or more hardware intermediate products, a full precision to represent values of an accumulator that generates the hardware accumulator output, and a half precision to represent values of the hardware accumulator output.

7. The computer-implemented method of claim 1 , wherein comparing the reference result to the generated result comprises comparing each of the one or more reference intermediate products to a corresponding one of the one or more hardware intermediate products and comparing the reference accumulator output to the hardware accumulator output to determine associated error values.

8. The computer-implemented method of claim 1 , wherein the ML computational operation includes a convolution operation.

9. A computing device for performing conformance testing of a machine learning (ML) computational operation configured to be implemented in a graphics processing unit (GPU) wherein the ML computational operation is performed by an algorithm specific to a hardware device, comprising:
a memory storing one or more parameters or instructions for developing an application; and
at least one processor coupled to the memory, wherein the at least one processor is configured to:
generate, for the ML computational operation and based on one or more inputs, a reference result including one or more reference intermediate products and a reference accumulator output at a first level of precision, wherein the one or more reference intermediate products of the ML computational operation are within one or more ranges including a range where all values within the range are within a same single floating point exponent;
generate, for the ML computational operation and based on specifying, to the GPU, the one or more inputs and a second level of precision, a result for the hardware device including one or more hardware intermediate products and a hardware accumulator output using the GPU at the second level of precision; and
output a conformance result based on whether a variance between the reference result and the result is within a threshold range.

10. The computing device of claim 9 , wherein the at least one processor is configured to determine the one or more inputs as in the range of 1.0 and a square root of 2.0.

11. The computing device of claim 9 , wherein the one or more ranges include a first range with an exponent of zero, a second range with a minimum exponent, and a third range with a maximum exponent, and wherein the at least one processor is configured to generate the reference result, generate the result, and compare the reference result to the generated result, for each of the one or more ranges.

12. The computing device of claim 9 , wherein the conformance result includes an indication of whether the generated result passes a conformance test.

13. The computing device of claim 9 , wherein the at least one processor is configured to generate the reference result, generate the result, and compare the reference result to the generated result, for multiple configurations for the computational operation to output conformance results for each of the configurations.

14. The computing device of claim 13 , wherein the multiple configurations include:
a first configuration having a single precision to represent values of the one or more inputs, a single precision to represent values of the one or more hardware intermediate products, a single precision to represent values of an accumulator that generates the hardware accumulator output, and a single precision to represent values of the hardware accumulator output;
a second configuration having a half precision to represent values of the one or more inputs, a half precision to represent values of the one or more hardware intermediate products, a half precision to represent values of an accumulator that generates the hardware accumulator output, and a half precision to represent values of the hardware accumulator output;
a third configuration having a half precision to represent values of the one or more inputs, a half precision to represent values of the one or more hardware intermediate

Assignees

• Microsoft Technology Licensing Llc

Inventors

• House Jr Barton Robert

Classifications

• G06F11/3688 Primary

  for test execution, e.g. scheduling of test suites · CPC title

• G06F11/3692

  for test results analysis · CPC title

• G06F17/15

  Correlation function computation {including computation of convolution operations (arithmetic circuits for sum of products per se, e.g. multiply-accumulators G06F7/5443; digital filters, e.g. FIR, IIR, adaptive filters H03H17/00)} · CPC title

• G06N20/00

  Machine learning · CPC title

• G06T1/20

  Processor architectures; Processor configuration, e.g. pipelining · CPC title