Method of detecting false test alarms using test step failure analysis

Patent metadata

Field	Value
Publication number	US-10235277-B2
Application number	US-201715625636-A
Country	US
Kind code	B2
Filing date	Jun 16, 2017
Priority date	Sep 4, 2015
Publication date	Mar 19, 2019
Grant date	Mar 19, 2019

Abstract

Identifying false test alarms to a developer. A code build is executed in a test system that includes computing functionality and computing infrastructure that is able to execute the build. Executing the code build includes running a plurality of system and integration tests on the code build. As a result of executing the code build, a system and integration test failure is identified. One or more characteristics of the system and integration test failure are identified. The characteristics of the system and integration test failure are compared to characteristics of a set of historical previous known false test alarms. False test alarms are failures caused by a factor other than a factor for which a test is being run. Based on the act of comparing, information is provided to a developer with respect to if the system and integration test failure is potentially a false test alarm.

First claim

What is claimed is:

1. A system for identifying potential false test alarms to a developer for a code base under test the system comprising:
a test system that includes computing functionality and computing infrastructure that is able to execute a code build, including running a plurality of system and integration tests on the code build to identify system and integration test failures and to identify one or more characteristics of the system and integration test failures;
a data store, wherein the data store comprises a set of association rules used for identifying potential false test alarms, wherein each association rule in the association rules identifies characteristics of a test case, including system and integration test steps, that results in a false test alarm when system and integration test steps in test cases having the identified characteristics are executed by the test system; and
a comparator coupled to the test system and the data store, wherein the comparator is configured to receive identification of characteristics of system and integration test failures from the test system and to compare the system and integration test failure characteristics to the association rules in the data store to identify potential false test alarms; and
wherein the comparator is configured to generate a notification to a user identifying a test failure as a potential false test alarm when a potential false test alarm is identified.

2. The system of claim 1 , wherein the comparator is configured to compare identified failing test steps in a test case to the association rules.

3. The system of claim 1 , wherein the comparator is configured to provide a statistical analysis of comparisons made by the comparator.

4. The system of claim 1 , wherein the comparator is configured to provide information based on a confidence and a support value.

5. The system of claim 1 , wherein the comparator is configured to provide a ranking of test failures based on the comparisons to the false test alarms.

6. The system of claim 5 , wherein the ranking is based on the statistical significance of the comparison.

7. The system of claim 1 , wherein the comparator is configured to identify potential false test alarms to a user only when a statistical threshold is met.

8. The system of claim 1 , further comprising a learning engine coupled to the data store, wherein the learning engine is configured to identify new association rules and to add the new association rules to the data store.

9. In a computer code testing environment, a method of identifying potential false test alarms to a developer for a code base under test the method comprising:
executing a code build in a test system that includes computing functionality and computing infrastructure that is able to execute the build wherein executing the code build in the test system comprises running a test case, including a plurality of system and integration tests, including a plurality of test steps, on the code build;
as a result of executing the code build and running the test case, identifying one or more system and integration test failures, including identifying failures of one or more test steps from the test case;
identifying one or more characteristics of the system and integration test failures;
comparing the characteristics of the system and integration test failures to characteristics of a test case known to cause false test alarms when run on an executing code build;
identifying that the characteristics of the system and integration test failures match the characteristics of the test case known to cause false test alarms when run on an executing code build; and
based on the act of identifying that the characteristics of the system and integration test failures match the characteristics of the test case known to cause false test alarms when run on an executing code build, providing information to a developer indicating that the system and integration test failure is potentially a false test alarm.

10. The method of claim 9 , wherein the compared characteristics are identified failing test steps in a test case such that comparing comprises attempting match failed test steps of a test case for the system and integration test failure to failed test steps of a test case for a historical previous known false test alarm.

11. The method of claim 9 , wherein providing information comprises providing a statistical analysis.

12. The method of claim 9 , wherein providing information comprises providing information based on a confidence and a support value.

13. The method of claim 9 , further comprising providing a ranking of test failures based on the comparisons to the false test alarms.

14. The method of claim 13 , wherein the ranking is based on the statistical significance of the comparison.

15. The method of claim 9 , wherein providing information is only performed when a statistical threshold is met.

16. The method of claim 9 , further comprising receiving input regarding additional false test alarms, and as a result adding new false test alarms to the set of historical previous known false test alarms to allow new rules to be learned for evaluating subsequent system and integration test failures.

17. A system for identifying potential false test alarms to a developer for a code base under test the system comprising:
one or more processors; and
one or more computer-readable storage media, wherein the one or more computer-readable storage media comprise computer-executable instructions that can be executed by at least one of the one or more processors to cause the following to be performed:
executing a code build in a test system that includes computing functionality and computing infrastructure that is able to execute the build wherein executing the code build in the test system comprises running a test case, including a plurality of system and integration tests, including a plurality of test steps, on the code build;
as a result of executing the code build and running the test case, identifying one or more system and integration test failures, including identifying failures of one or more test steps from the test case;
identifying one or more characteristics of the system and integration test failures;
comparing the characteristics of the system and integration test failures to characteristics of a test case known to cause false test alarms when run on an executing code build;
identifying that the characteristics of the system and integration test failures match the characteristics of the test case known to cause false test alarms when run on an executing code build; and
based on the act of identifying that the characteristics of the system and integration test failures match the characteristics of the test case known to cause false test alarms when run on an executing code build, providing information to a developer indicating that the system and integration test failure is potentially a false test alarm.

18. The system of claim 17 , wherein the compared characteristics are identified failing test steps in a test case such that comparing comprises attempting match failed test steps of a test case for the system and integration test failure to failed test steps of a test case for a historical previous known false test alarm.

19. The system of claim 17 , wherein providing information comprises providing a statistical analysis.

20. The system of claim 17 , wherein providing information comprises providing information based on a confidence and a support value.

Assignees

• Microsoft Technology Licensing Llc

Inventors

• Herzig Kim Sebastian
• Nagappan Nachiappan

Classifications

• G06F11/3672 Primary

  Test management · CPC title

• G06F11/3692 Primary

  for test results analysis · CPC title