Mutation testing within continuous integration systems

Patent metadata

Field	Value
Publication number	US-12072790-B1
Application number	US-202117224904-A
Country	US
Kind code	B1
Filing date	Apr 7, 2021
Priority date	Apr 7, 2021
Publication date	Aug 27, 2024
Grant date	Aug 27, 2024

Abstract

Techniques described herein relate to implementing mutation testing of software applications associated with continuous integration (CI) systems. A mutation test system may determine one or more portions of modified source code within an application codebase. Mutated applications may be generated based on the modified source code, and a mutation test system may determine subsets application test suites for execution based on the portions of the modified source code and/or other factors. In various examples, the mutation test system may use mappings between portions of source code and test subsets, and/or machine-learned models or heuristics-based techniques to determine subsets of test suites based on discreet source code modifications. Mutation testing can be performed by executing the determined test subsets on the mutated applications, and the results may be used by the CI system to control the integration of the code changes into the shared source code repository and/or automated testing of the application build.

First claim

What is claimed is:

1. A computer-implemented method, comprising:
receiving, by a computer system, a request to integrate a source code change into a codebase associated with an application; determining an application test suite associated with the application;
determining a first line of source code associated with the requested source code change;
analyzing, by the computer system, the first line of source code to determine a first source code attribute associated with the first line of source code, wherein the first source code attribute includes at least one of (a) a type of programming statement in the first line of source code, (b) a type of operator in the first line of source code, (c) an indication of a type of function invoked by the first line of source code, (d) an indication of whether the first line of source code is within a conditional code block, or (e) an indication of whether the first line of source code is within a programming loop;
determining a first test subset of the application test suite, based at least in part on the first source code attribute associated with the first line of source code, wherein determining the first test subset comprises inputting the first source code attribute into a trained model configured to output data identifying a subset of the test suite;
mutating, by the computer system, the source code change into a mutated source code change;
building, by the computer system, a mutated application by compiling one or more software classes including the mutated source code change; and
executing, by the computer system, the first test subset of the application test suite on the mutated application.

2. The computer-implemented method of claim 1 , further comprising:
determining that the source code change modifies a second line of source code within the codebase, wherein the second line of source code is within a different function from the first line of source code;
determining a second set of tests associated with the second line of source code, wherein the second set of tests is a second subset of the test suite different from the first subset; and
executing the second set of tests on the mutated application.

3. The computer-implemented method of claim 1 , further comprising:
initiating an operation to integrate the source code change into the codebase, based at least in part on test results associated with executing the first test subset.

4. The computer-implemented method of claim 1 , wherein determining the first test subset for execution on the mutated application comprises:
retrieving test results associated with a previous execution of the test suite, on a previous mutated application associated with the first line of source code; and
determining, within the test results associated with a previous execution of the test suite, one or more passing tests; and
excluding the one or more passing tests from the first subset of the test suite associated with the first line of source code.

5. The computer-implemented method of claim 1 , wherein determining the first test subset for execution on the mutated application comprises:
accessing a mapping storing associations between different subsets of the test suite, and corresponding lines of source code within the codebase.

6. The computer-implemented method of claim 1 , wherein:
mutating the source code change is based on a set of code mutation rules; and
determining the first test subset for execution on the mutated application, is based at least in part on the set of code mutation rules.

7. The computer-implemented method of claim 1 , further comprising:
receiving a mutation test confidence level associated with the request to integrate the source code change; and
determining the first test subset for execution on the mutated application, based at least in part on the mutation test confidence level.

8. One or more non-transitory computer-readable media storing instructions executable by a processor, wherein the instructions, when executed by the processor, cause the processor to perform operations comprising:
receiving a request to integrate a source code change into a codebase associated with an application;
determining an application test suite associated with the application;
determining a first line of source code associated with the requested source code change;
analyzing the first line of source code to determine a first source code attribute associated with the first line of source code, wherein the first source code attribute includes at least one of (a) a type of programming statement in the first line of source code, (b) a type of operator in the first line of source code, (c) an indication of a type of function invoked by the first line of source code, (d) an indication of whether the first line of source code is within a conditional code block, or (e) an indication of whether the first line of source code is within a programming loop;
determining a first test subset of the application test suite, based at least in part on the first source code attribute associated with the first line of source code, wherein determining the first test subset comprises inputting the first source code attribute into a trained model configured to output data identifying a subset of the test suite;
mutating the source code change into a mutated source code change;
building a mutated application based on the mutated source code change; and
executing the first test subset of the application test suite on the mutated application.

9. The one or more non-transitory computer-readable media of claim 8 , the operations further comprising:
determining that the source code change affects a second line of source code within the codebase, wherein the second line of source code is within a different function from the first line of source code;
determining a second set of tests associated with the second line of source code, wherein the second set of tests is a second subset of the test suite different from the first subset; and
executing the second set of tests on the mutated application.

10. The one or more non-transitory computer-readable media of claim 8 , wherein determining the first test subset comprises:
accessing a mapping storing associations between different subsets of the test suite, and at least one of corresponding lines of source code or corresponding functions within the codebase.

11. The one or more non-transitory computer-readable media of claim 8 , wherein:
mutating the source code change is based on a set of code mutation rules; and
determining the first test subset is based at least in part on the set of code mutation rules.

12. A computer-implemented method, comprising:
receiving, by a computer system, a request to integrate a source code change into a codebase associated with an application; determining an application test suite associated with the application;
determining a first line of source code associated with the requested source code change;
analyzing, by the computer system, the first line of source code to determine a first source code attribute associated with the first line of source code, wherein the first source code attribute includes at least one of (a) a type of programming statement in the first line of source code, (b) a type of operator in the first line of source code, (c) an indication of a type of function invoked by the first line of source code, (d) an indication of whether the first line of source code is within a conditional code block, or (e) an indication of whether the first line of source code is within a programming loop;
determining a first test subset of the application test suite, based at least in par

Assignees

• State Farm Mutual Automobile Insurance Co

Inventors

• Pearson Andrew L
• Shepherd Nate

Classifications

• G06F11/3684

  for test design, e.g. generating new test cases · CPC title

• G06F11/3692

  for test results analysis · CPC title

• G06F11/3676

  for coverage analysis · CPC title

• G06F11/3688 Primary

  for test execution, e.g. scheduling of test suites · CPC title

• G06F11/368 Primary

  for test version control, e.g. updating test cases to a new software version · CPC title