Determining a test confidence metric for a testing application

Patent metadata

Field	Value
Publication number	US-10558559-B2
Application number	US-201715655095-A
Country	US
Kind code	B2
Filing date	Jul 20, 2017
Priority date	Jul 25, 2016
Publication date	Feb 11, 2020
Grant date	Feb 11, 2020

Abstract

Techniques for calculating a test confidence metric (TCM) are disclosed. Calculating the TCM involves obtaining test results of a testing application. Calculating the TCM is based on confidence scores corresponding respectively to the test results. Calculate a confidence score for a particular test result involves identifying a failure reason for the test result, determining a weight corresponding to the failure reason, and calculating the confidence score based on the weight.

First claim

What is claimed is:

1. A non-transitory computer readable medium comprising instructions which, when executed by one or more hardware processors, cause performance of operations comprising:
obtaining a first plurality of test results of a first testing application;
calculating a first test confidence metric for the first testing application based on a first plurality of confidence scores corresponding respectively to the first plurality of test results, wherein calculating a first confidence score in the first plurality of confidence scores comprises:
(a) identifying a first failure reason for a first test result in the first plurality of test results, wherein the first failure reason indicates a problem in the first testing application;
(b) determining a first weight corresponding to the first failure reason, wherein the first weight indicates a lower confidence than a second weight for a second failure reason indicating a problem in functional code tested by the first testing application; and
(c) calculating the first confidence score based on the first weight; and

generating, based at least in part on the first test confidence metric, a graphical user interface (GUI) that is indicative of whether the first plurality of test results are relevant to the functional code tested by the first testing application.

2. The medium of claim 1 , wherein the operations further comprise:
determining a baseline confidence score per test result in the first plurality of test results,
wherein calculating the first confidence score comprises modifying the baseline score based on the first weight.

3. The medium of claim 2 , wherein the operations further comprise:
determining a plurality of time-based damping factors corresponding respectively to the first plurality of test results,
wherein calculating the first confidence score is further based on a corresponding time-based damping factor of the plurality of time-based damping factors.

4. The medium of claim 3 , wherein the plurality of time-based damping factors is based on a temporal ordering of the first plurality of test results.

5. The medium of claim 3 , wherein for each test result in the plurality of test results, a corresponding time-based damping factor in the plurality of time-based damping factors is based on recentness of the test result.

6. The medium of claim 1 , wherein calculating a second confidence score in the plurality of confidence scores comprises:
(a) determining that a second test result in the first plurality of test results indicates a pass condition of the first testing application,
(b) determining a third weight corresponding to the pass condition, and
(c) calculating the second confidence score based on the third weight.

7. The medium of claim 1 , wherein the operations further comprise:
obtaining a second plurality of test results of a second testing application; and
calculating a second test confidence metric for the second testing application based on a second plurality of confidence scores corresponding respectively to the second plurality of test results, wherein calculating a second confidence score in the second plurality of confidence scores comprises:
(a) identifying the second failure reason for a second test result in the second plurality of test results,
(b) determining the second weight corresponding to the second failure reason, and
(c) calculating the second confidence score based on the second weight,

wherein the second test confidence metric is different from the first test confidence metric.

8. The medium of claim 1 , wherein the GUI indicates, based at least in part on the first test confidence metric, a trend in confidence in the first testing application over time.

9. The medium of claim 8 , wherein the GUI comprises a distribution chart of a plurality of test confidence metrics.

10. The medium of claim 1 , wherein the operations further comprise:
assigning a confidence category to the first testing application based on the first test confidence metric.

11. The medium of claim 1 , wherein the first test confidence metric is calculated based on a sum of the first plurality of confidence scores.

12. The medium of claim 11 , wherein the first test confidence metric is calculated by dividing the sum by a maximum possible value.

13. The medium of claim 1 , wherein the obtaining the first plurality of test results comprises:
determining a maximum number of test results to be used for calculating the first test confidence metric; and
selecting a number of test results not exceeding the maximum number of test results from a plurality of available test results.

14. The medium of claim 1 , wherein obtaining the first plurality of test results comprises querying a quality assurance database.

15. The medium of claim 1 , wherein determining the first weight comprises selecting a weight corresponding to a problem in the first testing application.

16. The medium of claim 1 , wherein determining the second weight comprises selecting a weight corresponding to a problem in functional code tested by the first testing application.

17. The medium of claim 1 , wherein the GUI includes a heat map that depicts a visual indicator based on what percentage of the first plurality of test results correspond to the first failure reason.

18. The medium of claim 1 , wherein the GUI includes a control for selecting which test confidence metrics to visualize based on groupings of a plurality of testing applications.

19. A system comprising:
one or more hardware processors; and
one or more one or more non-transitory computer-readable media storing instructions, which when executed by the one or more hardware processors, cause execution of operations comprising:
obtaining a first plurality of test results of a first testing application;
calculating a first test confidence metric for the first testing application based on a first plurality of confidence scores corresponding respectively to the first plurality of test results, wherein calculating a first confidence score in the first plurality of confidence scores comprises:
(a) identifying a first failure reason for a first test result in the first plurality of test results, wherein the first failure reason indicates a problem in the first testing application;
(b) determining a first weight corresponding to the first failure reason, wherein the first weight indicates a lower confidence than a second weight for a second failure reason indicating a problem in functional code tested by the first testing application; and
(c) calculating the first confidence score based on the first weight; and

generating, based at least in part on the first test confidence metric, a graphical user interface (GUI) that is indicative of whether the first plurality of test results are relevant to the functional code tested by the first testing application.

20. A method comprising:
obtaining a first plurality of test results of a first testing application;
calculating a first test confidence metric for the first testing application based on a first plurality of confidence scores corresponding respectively to the first plurality of test results, wherein calculating a first confidence score in the first plurality of confidence scores comprises:
(a) identifying a first failure reason for a first test result in the first plurality of test results, wherein the first failure reason indicates a problem in the first testing application;
(b) determining a first weight corresponding to the first failure

Assignees

• Oracle Int Corp

Inventors

• Agarwal Mayank
• Vadlamani Jagannadha Prasad Srinivas
• Mui Wendy

Classifications

• G06F11/3692 Primary

  for test results analysis · CPC title

• G06F3/048

  Interaction techniques based on graphical user interfaces [GUI] · CPC title

• G06F11/3676

  for coverage analysis · CPC title