Software test automation systems and methods

Patent metadata

Field	Value
Publication number	US-9448916-B2
Application number	US-201213420923-A
Country	US
Kind code	B2
Filing date	Mar 15, 2012
Priority date	Jul 21, 2011
Publication date	Sep 20, 2016
Grant date	Sep 20, 2016

Abstract

Described are a system and method for performing an automated quality assessment on a software program under test. A test automation system executes a test on a software program Data related to the test is automatically collected. The data includes first information determined by the test automation system in response to executing the test. The data further includes second information related to the test and received from a source other than the test automation system. The first information is analyzed. A quality assessment of the software program is generated from the analyzed first information and from the second information.

First claim

What is claimed is:

1. A computer-implemented method for performing an automated quality assessment on a software program under test, comprising:
executing by a test automation system a test on a software program;
automatically collecting data related to the test, the data including first information determined by the test automation system in response to executing the test, the data further including information about the test and received from a source other than the test automation system;
analyzing the first information, including analyzing a type of test being exercised;
generating a quality assessment of the software program from the analyzed first information and from the second information, the quality assessment including analysis data about components of the software program relative to each other, wherein the data further includes the type of test performed on a particular component of the software component, and wherein the quality assessment distinguishes a first result generated in response to the type of test performed on the particular component in a first test environment included in a first combination of the first and second information from a second result generated in response to the type of test performed on the particular component in a second test environment included in a second combination of the first and second information, wherein the particular component of the body of software code in the first test environment is part of a same software release as the first component of the body of software code in the second test environment, wherein the first and second results each includes a test result and a performance result, wherein the test result includes an indication of whether the test of the software program passes, and wherein the performance result includes an indicator of a comparison between the test and historical test data regardless of whether the test result indicates that the software program passes;
generating a regression result, which identifies whether the test is failing now, but the test previously passed, by comparing the test result of the particular component and one or more previous tests to indicate whether a test failure is a regression, the regression result different than the test result and the performance result;
generating a product quality table that displays an amount of code coverage for the component having the test result, and a percentage of the code coverage passing the test and a percentage of the code coverage failing the test, and that generates a correlation token associated with a thread of execution of the test, wherein the code coverage is tagged with the correlation token for reconstructing a code path of the component having the test result;
and preventing a false positive reading of the first and second results by displaying both the performance results and the test results of both the first and second results and comparing a performance of the test against the historical test data, wherein when the test result of the first result and the test result of the second result are the same result, the performance result of the first result is either the same as or different than the performance result of the second result depending on corresponding differences between the first test environment and the second test environment.

2. The computer-implemented method of claim 1 further comprising:
determining the quality assessment from the test results of the first and second results.

3. The computer-implemented method of claim 2 , wherein the test result includes at least one of a test pass result and a test fail result.

4. The computer-implemented method of claim 3 , wherein a software defect is automatically generated or annotated from the test fail result.

5. The computer-implemented method of claim 1 , wherein the quality assessment is generated as a quality assessment report, and wherein the quality assessment report includes at least one of:
a result identifying thoroughly tested and properly functioning areas of the software program;
a result identifying thoroughly tested and inadequate functioning areas of the software program;
a result identifying untested areas of the software program;
and a trend analysis report.

6. The computer implemented method of claim 1 , wherein the second information is received from at least one of a quality management test tool and a test automation result history database.

7. The computer implemented method of claim 6 , wherein the second information includes test artifacts related to the software program provided by the quality management test tool.

8. The computer-implemented method of claim 1 , wherein analyzing the first information includes determining at least one of code paths being exercised and platform configuration information related to the test.

9. The computer-implemented method of claim 8 , wherein the type of test includes at least one of a boundary conditions test, a stress test, a GUI test, an http protocol test, a SOAP interface test, a CLI test.

10. The computer-implemented method of claim 1 , further comprising:
receiving code coverage analysis data;
and determining from the code coverage analysis data a degree to which source code of the software program has been tested.

11. A computer-implemented method for performing an automated quality assessment on a software program under test, comprising:
executing a test on a body of software code using a test automation tool;
dynamically determining information about the test being exercised from a source other than the test automation tool, the information including data indicating whether the test covers at least a first component of the body of software code;
analyzing the information about the test being exercised;
generating a first record of the test, the first record including an identifier corresponding to the first component, information about the test, and at least one result of the test related to the first component;
distinguishing, by the automated quality assessment, a first result of the at least one result of the test performed on the first component of the body of software code in a first test environment from a second result of the at least one result of the test performed on the first component of the body of software code in a second test environment that is different from the first test environment, the first component of the body of software code in the first test environment being part of a same software release as the first component of the body of software code in the second test environment, wherein the first and second results each includes a test result and a performance result, wherein the test result includes an indication of whether the test of the software program passes, and wherein the performance result includes an indicator of a comparison between the test and historical test data regardless of whether the test result indicates that the software program passes;
generating a regression result, which identifies whether the test is failing now, but the test previously passed, by comparing the test result of the particular component and one or more previous tests to indicate whether a test failure is a regression, the regression result different than the test result and the performance result;
generating a product quality table that displays an amount of code coverage for the component having the test result, and a percentage of the code coverage passing the test and a percentage of the code coverage failing the test, and that generates a correlation token associated with a thread of execution of the test, wherein the code coverage is tagged with the correlation token for reconstructing a co

Assignees

• Browne Michael E
• Emejulu Nnaemeka Iriele
• Lavery Andrew Jason
• Liu Ye
• Maldari Mario Alessandro
• IBM

Inventors

• Browne Michael E
• Emejulu Nnaemeka Iriele
• Lavery Andrew Jason
• Liu Ye
• Maldari Mario Alessandro

Classifications

• G06F11/3692 Primary

  for test results analysis · CPC title

• G06F11/3604

  Analysis of software for verifying properties of programs (testing of software G06F11/3668) · CPC title

• G06F11/3676 Primary

  for coverage analysis · CPC title