Method of Detecting False Test Alarms Using Test Step Failure Analysis

Patent metadata

Field	Value
Publication number	US-2017068612-A1
Application number	US-201514845891-A
Country	US
Kind code	A1
Filing date	Sep 4, 2015
Priority date	Sep 4, 2015
Publication date	Mar 9, 2017
Grant date	—

Abstract

Identifying false test alarms to a developer. A code build is executed in a test system that includes computing functionality and computing infrastructure that is able to execute the build. Executing the code build includes running a plurality of system and integration tests on the code build. As a result of executing the code build, a system and integration test failure is identified. One or more characteristics of the system and integration test failure are identified. The characteristics of the system and integration test failure are compared to characteristics of a set of historical previous known false test alarms. False test alarms are failures caused by a factor other than a factor for which a test is being run. Based on the act of comparing, information is provided to a developer with respect to if the system and integration test failure is potentially a false test alarm.

First claim

What is claimed is:

1 . A system for identifying potential false test alarms to a developer for a code base under test the system comprising:
a test system that includes computing functionality and computing infrastructure that is able to execute a code build, including running a plurality of system and integration tests on the code build to identify system and integration test failures and to identify one or more characteristics of the system and integration test failures; a data store, wherein the data store comprises a set of association rules used for identifying potential false test alarms; and a comparator coupled to the test system and the data store, wherein the comparator is configured to receive identification of characteristics of system and integration test failures from the test system and to compare the system and integration test failure characteristics to the association rules in the data store to identify potential false test alarms, and wherein the comparator is configured to identify potential false test alarms to a user.

2 . The system of claim 1 , wherein the comparator is configured to compare identified failing test steps in a test case to the association rules.

3 . The system of claim 1 , wherein the comparator is configured to provide a statistical analysis of comparisons made by the comparator.

4 . The system of claim 1 , wherein the comparator is configured to provide information based on a confidence and a support value.

5 . The system of claim 1 , wherein the comparator is configured to provide a ranking of test failures based on the comparisons to the false test alarms.

6 . The system of claim 5 , wherein the ranking is based on the statistical significance of the comparison.

7 . The system of claim 1 , wherein the comparator is configured to identify potential false test alarms to a user only when a statistical threshold is met.

8 . The system of claim 1 , further comprising a learning engine coupled to the data store, wherein the learning engine is configured to identify new association rules and to add the new association rules to the data store.

9 . In a computer code testing environment, a method of identifying potential false test alarms to a developer for a code base under test the method comprising:
executing a code build in a test system that includes computing functionality and computing infrastructure that is able to execute the build wherein executing the code build in the test system comprises running a plurality of system and integration tests on the code build; as a result of executing the code build, identifying a system and integration test failure; identifying one or more characteristics of the system and integration test failure; comparing the characteristics of the system and integration test failure to characteristics of a set of historical previous known false test alarms; and based on the act of comparing, providing information to a developer with respect to if the system and integration test failure is potentially a false test alarm.

10 . The method of claim 9 , wherein the compared characteristics are identified failing test steps in a test case such that comparing comprises attempting match failed test steps of a test case for the system and integration test failure to failed test steps of a test case for a historical previous known false test alarm.

11 . The method of claim 9 , wherein providing information comprises providing a statistical analysis.

12 . The method of claim 9 , wherein providing information comprises providing information based on a confidence and a support value.

13 . The method of claim 9 , further comprising providing a ranking of test failures based on the comparisons to the false test alarms

14 . The method of claim 13 , wherein the ranking is based on the statistical significance of the comparison.

15 . The method of claim 9 , wherein providing information is only performed when a statistical threshold is met.

16 . The method of claim 9 , further comprising receiving input regarding additional false test alarms, and as a result adding new false test alarms to the set of historical previous known false test alarms to allow new rules to be learned for evaluating subsequent system and integration test failures.

17 . A system for identifying potential false test alarms to a developer for a code base under test the system comprising:
one or more processors; and one or more computer-readable media, wherein the one or more computer-readable media comprise computer-executable instructions that can be executed by at least one of the one or more processors to cause the following to be performed:
executing a code build in a test system that includes computing functionality and computing infrastructure that is able to execute the build wherein executing the code build in the test system comprises running a plurality of system and integration tests on the code build;
as a result of executing the code build, identifying a system and integration test failure;
identifying one or more characteristics of the system and integration test failure;
comparing the characteristics of the system and integration test failure to characteristics of a set of historical previous known false test alarms; and
based on the act of comparing, providing information to a developer with respect to if the system and integration test failure is potentially a false test alarm.

18 . The system of claim 17 , wherein the compared characteristics are identified failing test steps in a test case such that comparing comprises attempting match failed test steps of a test case for the system and integration test failure to failed test steps of a test case for a historical previous known false test alarm.

19 . The system of claim 17 , wherein providing information comprises providing a statistical analysis.

20 . The system of claim 17 , wherein providing information comprises providing information based on a confidence and a support value.

Assignees

• Microsoft Technology Licensing Llc

Inventors

• Herzig Kim Sebastian
• Nagappan Nachiappan

Classifications

• G06F11/3692 Primary

  for test results analysis · CPC title

• G06F11/3672 Primary

  Test management · CPC title