Method of detecting false test alarms using test step failure analysis

Patent metadata

Field	Value
Publication number	US-9710364-B2
Application number	US-201514845891-A
Country	US
Kind code	B2
Filing date	Sep 4, 2015
Priority date	Sep 4, 2015
Publication date	Jul 18, 2017
Grant date	Jul 18, 2017

Abstract

Identifying false test alarms to a developer. A code build is executed in a test system that includes computing functionality and computing infrastructure that is able to execute the build. Executing the code build includes running a plurality of system and integration tests on the code build. As a result of executing the code build, a system and integration test failure is identified. One or more characteristics of the system and integration test failure are identified. The characteristics of the system and integration test failure are compared to characteristics of a set of historical previous known false test alarms. False test alarms are failures caused by a factor other than a factor for which a test is being run. Based on the act of comparing, information is provided to a developer with respect to if the system and integration test failure is potentially a false test alarm.

First claim

What is claimed is:

1. A system for identifying potential false test alarms to a developer for a code base under test the system comprising:
a test system that includes computing functionality and computing infrastructure that is able to execute a code build, including running a plurality of system and integration tests on the code build to identify system and integration test failures and to identify one or more characteristics of the system and integration test failures;
a data store, wherein the data store comprises a set of association rules used for identifying potential false test alarms, wherein the association rules include characteristics of previous test failures that can be compared to test failures to identify the test failures as potentially being false test alarms; and
a comparator coupled to the test system and the data store, wherein the comparator is configured to receive identification of characteristics of system and integration test failures from the test system and to compare the system and integration test failure characteristics to the characteristics in the association rules in the data store, to identify potential false test alarms, and
wherein the comparator is configured to identify test failures as potential false test alarms to a user based on comparing the system and integration test failure characteristics to the characteristics of the association rules in the data store;
wherein the comparator is configured to compare identified failing test steps in a test case to the association rules.

2. The system of claim 1 , wherein the comparator is configured to provide a statistical analysis of comparisons made by the comparator.

3. The system of claim 1 , wherein the association rules are configured to provide information based on a confidence and a support value, wherein the confidence value is the relative number of observed integration test failures with characteristics that matched the characteristics of previous test failures and that were determined to be true false test alarms verses the total number of observed integration test failures with characteristics that matched the characteristics of previous test failures, and the support value is the proportion of test case executions for which all characteristics of a given test failure were satisfied.

4. The system of claim 1 , wherein the comparator is configured to provide a ranking of test failures based on the comparisons to the false test alarms.

5. The system of claim 4 , wherein the ranking is based on the statistical significance of the comparison.

6. The system of claim 1 , wherein the comparator is configured to identify potential false test alarms to a user only when a statistical threshold is met.

7. The system of claim 1 , further comprising a learning engine coupled to the data store, wherein the learning engine is configured to identify new association rules and to add the new association rules to the data store.

8. In a computer code testing environment, a method of identifying potential false test alarms to a developer for a code base under test the method comprising:
executing a code build in a test system that includes computing functionality and computing infrastructure that is able to execute the build wherein executing the code build in the test system comprises running a plurality of system and integration tests on the code build;
as a result of executing the code build, identifying a system and integration test failure; identifying one or more characteristics of the system and integration test failure;
comparing the characteristics of the system and integration test failure to characteristics of a set of historical previous known false test alarms that can be compared to test failures to identify the test failures as potentially being false test alarms; and
based on the act of comparing the characteristics of the system and integration test failure to the characteristics of the set of historical previous known false test alarms,
providing information to a developer with respect to if the system and integration test failure is potentially a false test alarm;
wherein the comparator is configured to compare identified failing test steps in a test case to the association rules.

9. The method of claim 8 , wherein the compared characteristics are identified failing test steps in a test case such that comparing comprises attempting match failed test steps of a test case for the system and integration test failure to failed test steps of a test case for a historical previous known false test alarm.

10. The method of claim 8 , wherein providing information comprises providing a statistical analysis showing the likelihood that the test failures are false alarms.

11. The method of claim 8 , wherein providing information comprises providing information based on a confidence and a support value showing the likelihood that the test failures are false alarms.

12. The method of claim 8 , further comprising providing a ranking of test failures based on the comparisons to the false test alarms and ranked according to the likelihood of a test failure being a false test alarm.

13. The method of claim 12 , wherein the ranking is based on the statistical significance of the comparison.

14. The method of claim 8 , wherein providing information is only performed when a statistical threshold is met.

15. The method of claim 8 , further comprising receiving input regarding additional false test alarms, and as a result adding new false test alarms to the set of historical previous known false test alarms to allow new rules to be learned for evaluating subsequent system and integration test failures.

16. A system for identifying potential false test alarms to a developer for a code base under test the system comprising: one or more processors; and
one or more computer-readable media, wherein the one or more computer-readable media comprise computer-executable instructions that can be executed by at least one of the one or more processors to cause the following to be performed:
executing a code build in a test system that includes computing functionality and computing infrastructure that is able to execute the build wherein executing the code build in the test system comprises running a plurality of system and integration tests on the code build;
as a result of executing the code build, identifying a system and integration test failure;
identifying one or more characteristics of the system and integration test failure;
comparing the characteristics of the system and integration test failure to characteristics of a set of historical previous known false test alarms that can be compared to test failures to identify the test failures as potentially being false test alarms; and
based on the act of comparing the characteristics of the system and integration test failure to the characteristics of the set of historical previous known false test alarms,
providing information to a developer with respect to if the system and integration test failure is potentially a false test alarm;
wherein the comparator is configured to compare identified failing test steps in a test case to the association rules.

17. The system of claim 16 , wherein the compared characteristics are identified failing test steps in a test case such that comparing comprises attempting match failed test steps of a test case for the system and integration test failure to failed test steps of a test case for a historical previous known false test alarm.

18. The system of claim 16 , wherein providing information compri

Assignees

• Microsoft Technology Licensing Llc
• Micron Tech Licensing Llc

Inventors

• Herzig Kim Sebastian
• Nagappan Nachiappan

Classifications

• G06F11/3692 Primary

  for test results analysis · CPC title

• G06F11/3672 Primary

  Test management · CPC title