Performance regression testing in presence of noise

Patent metadata

Field	Value
Publication number	US-9400732-B2
Application number	US-201414282084-A
Country	US
Kind code	B2
Filing date	May 20, 2014
Priority date	May 20, 2014
Publication date	Jul 26, 2016
Grant date	Jul 26, 2016

Abstract

A method and system includes calculating a performance metric for each of a plurality of builds of a software application in view of a respective performance test result associated with each of the plurality of builds, calculating a respective difference in performance metrics for each pair of consecutive builds of the plurality of builds, determining a largest performance drop in view of respective differences in the performance metrics among the pairs of consecutive builds of the plurality of builds, wherein the largest performance drop is associated with a first pair of consecutive builds comprising a first build and a second build, determining, by a processing device, a confidence level associated with the largest performance drop in view of performance test results associated with the first build and the second build, in response to determining that the confidence level is greater than or equal to a threshold, identifying one of the first build or the second build as a problematic build of the software application.

First claim

What is claimed is:

1. A method comprising:
calculating a performance metric for each of a plurality of builds of a software application in view of a respective performance test result associated with each of the plurality of builds;
calculating a respective difference in performance metrics for each pair of consecutive builds of the plurality of builds;
determining a largest performance drop in view of respective differences in the performance metrics among the pairs of consecutive builds of the plurality of builds, wherein the largest performance drop is associated with a first pair of consecutive builds comprising a first build and a second build;
determining, by a processing device, a confidence level associated with the largest performance drop in view of performance test results associated with the first build and the second build; and
in response to determining that the confidence level is greater than or equal to a threshold, identifying one of the first build or the second build as a problematic build of the software application.

2. The method of claim 1 , further comprising:
in response to determining that the confidence level is less than the threshold,
executing a performance test on a selected build of the software application to generate a further performance test result for the selected build; calculating a further performance metric for the selected build in view of the further performance test result;
adding the selected build into the plurality of builds;
calculating a respective difference in performance metrics for each pair of consecutive builds of the plurality of builds;
determining an updated largest performance drop in view of respective differences in the performance metrics among the pairs of consecutive builds of the plurality of builds, wherein the updated largest performance drop is associated with a second pair comprising a third build and a fourth build;
determining a second confidence level associated with the updated largest performance drop in view of performance test results associated with the third build and the fourth build;
determining that the second confidence level is greater than or equal to the threshold; and
identifying one of the third build or the fourth build as a problematic build of the software application.

3. The method of claim 2 , wherein the selected build of the software application is an untested build between the first build and the second build.

4. The method of claim 2 , wherein the selected build of the software application is selected from one of the first build or the second build in view of a number of performance test results of the first build and a number of performance test results of the second build.

5. The method of claim 1 , wherein the plurality of builds are part of all builds of the software application, and wherein the plurality of builds are ordered chronologically according to progress of software application development.

6. The method of claim 1 , wherein the performance test result comprises at least one of a time to execute part of the software application, an amount of data transmitted on a network device over a specific time period, or an amount of computational resources consumed.

7. The method of claim 1 , wherein the performance metric associated with each of the plurality of builds comprises an average of a plurality of samples of the performance test result associated with each of the plurality of builds.

8. The method of claim 1 , wherein determining the largest performance drop comprises identifying a largest difference among all differences in the performance metrics among the pairs of consecutive builds.

9. The method of claim 1 , wherein determining the confidence level associated with the largest performance drop comprises performing a T-test in view of the test results associated with the first build and the second build, and wherein performing the T-test comprises calculating a respective average of the test results associated with the first build and the second build, and a respective standard deviation of the test results associated with the first build and the second build.

10. The method of claim 1 , wherein the threshold is at 95%.

11. A non-transitory machine-readable storage medium storing instructions which, when executed, a processing device to:
calculate a performance metric for each of a plurality of builds of a software application in view of a respective performance test result associated with each of the plurality of builds;
calculate a respective difference in performance metrics for each pair of consecutive builds of the plurality of builds;
determine a largest performance drop in view of respective differences in the performance metrics among the pairs of consecutive builds of the plurality of builds, wherein the largest performance drop is associated with a first pair of consecutive builds comprising a first build and a second build;
determine, by a processing device, a confidence level associated with the largest performance drop in view of performance test results associated with the first build and the second build; and
in response to determining that the confidence level is greater than or equal to a threshold, identify one of the first build or the second build as a problematic build of the software application.

12. The machine-readable storage medium of claim 11 , wherein the processing device is further to:
in response to determining that the confidence level is less than the threshold,
execute a performance test on a selected build of the software application to generate a further performance test result for the selected build;
calculate a further performance metric for the selected build in view of the further performance test result;
add the selected build into the plurality of builds;
calculate a respective difference in performance metrics for each pair of consecutive builds of the plurality of builds;
determine an updated largest performance drop in view of respective differences in the performance metrics among the pairs of consecutive builds of the plurality of builds, wherein the updated largest performance drop is associated with a second pair comprising a third build and a fourth build;
determine a second confidence level associated with the updated largest performance drop in view of performance test results associated with the third build and the fourth build;
determine that the second confidence level is greater than or equal to the threshold; and
identify one of the third build or the fourth build as a problematic build of the software application.

13. The machine-readable storage medium of claim 12 , wherein the selected build of the software application is an untested build between the first build and the second build.

14. The machine-readable storage medium of claim 12 , wherein the selected build of the software application is selected from one of the first build or the second build in view of a number of performance test results of the first build and a number of performance test results of the second build.

15. The machine-readable storage medium of claim 11 , wherein the plurality of builds are part of all builds of the software application, and wherein the plurality of builds are ordered chronologically according to progress of software application development.

16. The machine-readable storage medium of claim 11 , wherein the performance metric associated with each of the plurality of builds comprises an average of a plurality of samples of the performance test result associated with each of the plurality of builds.

Assignees

• Red Hat Israel Ltd

Inventors

• Hod Ronen

Classifications

• G06F11/3003 Primary

  Monitoring arrangements specially adapted to the computing system or computing system component being monitored · CPC title

• G06F11/34

  Recording or statistical evaluation of computer activity, e.g. of down time, of input/output operation {; Recording or statistical evaluation of user activity, e.g. usability assessment} · CPC title

• G06F11/3409 Primary

  for performance assessment · CPC title

• G06F2201/865

  Monitoring of software · CPC title

• G06F11/3428

  Benchmarking · CPC title