Dynamic testing based on automated impact analysis

Patent metadata

Field	Value
Publication number	US-10417119-B2
Application number	US-201615141477-A
Country	US
Kind code	B2
Filing date	Apr 28, 2016
Priority date	Mar 11, 2016
Publication date	Sep 17, 2019
Grant date	Sep 17, 2019

Abstract

A method for automated software testing may include mapping test cases to lines in files in a codebase. Each test case covers one or more lines in one or more files and has corresponding test results. The method may further include obtaining a change list including one or more changes. Each change specifies a changed line in a changed file. The method may further include determining impacted test cases, based on the mapping and the change list, prioritizing the impacted test cases based, in part, on test results corresponding to each impacted test case, executing, based on the prioritization, one or more impacted test cases to obtain executed test cases, and obtaining, for each executed test case, new test results.

First claim

What is claimed is:

1. A method for automated software testing, comprising:
instrumenting a codebase to extract a plurality of lines in a plurality files that are executed when executing the codebase;
in response to executing the codebase, mapping a plurality of test cases to the plurality of lines,
wherein each test case of the plurality of test cases covers one or more lines in the plurality of lines and has a corresponding plurality of test results;

obtaining a change list comprising one or more changes, each change specifying a changed line in a changed file of the plurality of files;
determining, based on the mapping and the change list, a plurality of impacted test cases of the plurality of test cases by:
attempting to identify, for each change of the change list, one of the plurality of test cases that covers the changed line specified by the change;
failing to identify a test case that covers the changed line specified by a first change of the change list; and
identifying one of the plurality of test cases that covers an unchanged line in the changed file specified by the first change and fails to cover the changed line;

prioritizing the plurality of impacted test cases to favor frequently failing test cases based, in part, on a plurality of test results corresponding to each impacted test case of the plurality of impacted test cases,
wherein the plurality of test results comprises a failure count and a measure of the utilization of a resource of a computer system during execution of the corresponding impacted test case, and
wherein the prioritization is further based on a first pre-determined weighting factor of the measure of the utilization and a second pre-determined weighting factor of the failure count;

executing, based on the prioritization, one or more impacted test cases of the plurality of impacted test cases to obtain a plurality of executed test cases; and
obtaining, for each executed test case of the plurality of executed test cases, a new plurality of test results.

2. The method of claim 1 , further comprising:
storing, in a database, the mapping and a plurality of test results corresponding to each test case of the plurality of test cases; and
updating, for each executed test case, the database with the new plurality of test results.

3. The method of claim 1 ,
wherein each test result of the plurality of test results comprises an execution time, and
wherein the prioritization is further based on a third pre-determined weighting factor of the execution time.

4. The method of claim 1 , wherein the determination is further based on a precision of coverage, relative to a change in the change list, of an impacted test case of the plurality of impacted test cases.

5. The method of claim 1 , wherein the prioritization is further based on a precision of coverage, relative to a change in the change list, of an impacted test case of the plurality of impacted test cases.

6. The method of claim 1 , further comprising:
issuing an alert identifying one or more lines in one or more files of the plurality of files based on one or more test results corresponding to a test case of the plurality of test cases.

7. A system for automated software testing, comprising:
mapper circuitry configured for: in response to executing a codebase, mapping a plurality of test cases to a plurality of lines in a plurality files in the codebase, wherein each test case of the plurality of test cases covers one or more lines in the plurality of lines and has a corresponding plurality of test results;
change list analyzer circuitry configured for:
obtaining a change list comprising one or more changes, each change specifying a changed line in a changed file of the plurality of files, and
determining, based on the mapping and the change list, a plurality of impacted test cases of the plurality of test cases by:
attempting to identify, for each change of the change list, one of the plurality of test cases that covers the changed line specified by the change
failing to identify a test case that covers the changed line specified by a first change of the change list; and
identifying one of the plurality of test cases that covers an unchanged line in the changed file specified by the first change and fails to cover the changed line;

test sequencer circuitry configured for:
prioritizing the plurality of impacted test cases to favor frequently failing test cases based, in part, on a plurality of test results corresponding to each impacted test case of the plurality of impacted test cases,
wherein the plurality of test results comprises a failure count and a measure of the utilization of a resource of a computer system during execution of the corresponding impacted test case, and
wherein the prioritization is further based on a first pre-determined weighting factor of the measure of the utilization and a second pre-determined weighting factor of the failure count;

test executive circuitry configured for:
instrumenting the codebase to extract the plurality of lines in the plurality files that are executed when executing the codebase;
executing, based on the prioritization, one or more impacted test cases of the plurality of impacted test cases to obtain a plurality of executed test cases, and
obtaining, for each executed test case of the plurality of executed test cases, a new plurality of test results; and

repository circuitry configured for storing at least the codebase and the plurality of test cases.

8. The system of claim 7 ,
wherein the repository circuitry further comprises a database storing the mapping and a plurality of test results corresponding to each test case of the plurality of test cases, and
wherein the test executive circuitry further comprises updating, for each executed test case, the database with each new plurality of test results.

9. The system of claim 7 ,
wherein each test result of the plurality of test results comprises a failure count and an execution time, and
wherein the prioritization is further based on a third pre-determined weighting factor of the execution time.

10. The system of claim 7 , wherein the determination is further based on a precision of coverage, relative to a change in the change list, of an impacted test case of the plurality of impacted test cases.

11. The system of claim 7 , wherein the prioritization is further based on a precision of coverage, relative to a change in the change list, of an impacted test case of the plurality of impacted test cases.

12. The system of claim 7 , further comprising:
failure analyzer circuitry configured for issuing an alert identifying one or more lines in one or more files of the plurality of files based on one or more test results corresponding to a test case of the plurality of test cases.

13. A non-transitory computer readable medium comprising instructions that, when executed by a hardware processor, perform a method for automated software testing, the method comprising:
instrumenting a codebase to extract a plurality of lines in a plurality files that are executed when executing the codebase;
in response to executing the codebase, mapping a plurality of test cases to the plurality of lines,
wherein each test case of the plurality of test cases covers one or more lines in the plurality of lines and has a corresponding plurality of test results;

obtaining a change list comprising one or more changes, each change specifying a changed line in a changed file of the plurality of files;
determining, based on the mapping and the change list, a plurality of impacted test cases of the plurality of test cases by:
attempting to identif

Assignees

• Francis Sachin
• Gupta Divya
• Datta Ayush
• Thomas Vijay
• Intuit Inc

Inventors

• Francis Sachin
• Gupta Divya
• Datta Ayush
• Thomas Vijay

Classifications

• G06F11/3672 Primary

  Test management · CPC title

• G06F11/3692

  for test results analysis · CPC title

• G06F11/368

  for test version control, e.g. updating test cases to a new software version · CPC title

• G06F11/3688 Primary

  for test execution, e.g. scheduling of test suites · CPC title