Retrying failed test cases in software testing using parallel threads

Patent metadata

Field	Value
Publication number	US-11720482-B1
Application number	US-202117333637-A
Country	US
Kind code	B1
Filing date	May 28, 2021
Priority date	Feb 23, 2021
Publication date	Aug 8, 2023
Grant date	Aug 8, 2023

Abstract

Test cases written to test a software application can be dynamically distributed among different sets of test cases that can be executed simultaneously in different parallel threads, thereby speeding up testing relative to executing the test cases sequentially in a single thread. Although executing the test cases in parallel threads can cause the test cases to execute in a different order than intended by developers, any test cases that fail due to database conflicts or other errors as a result of the parallelization can be retried at least once. Re-executing failed test cases at a later time can lead to the full set of test cases ultimately succeeding, despite errors that may have been caused by the parallelization.

First claim

What is claimed is:

1. A computer-implemented method, comprising:
identifying, by one or more processors of a computing device, a plurality of test cases that test interactions between a software application and a database;
distributing, by the one or more processors, individual test cases, of the plurality of test cases, among a plurality of parallel threads on the computing device;
executing, by the one or more processors, the plurality of test cases in the plurality of parallel threads;
identifying, by the one or more processors, failed test cases, of the plurality of test cases, based on the executing;
generating, by the one or more processors, at least one test failure log, the at least one test failure log indicating the failed test cases and failure codes corresponding to the failed test cases, wherein the failure codes indicate causes of failures of the failed test cases;
determining, by the one or more processors, and based on the failure codes, an execution order for re-execution of the failed test cases, wherein:
the execution order is arranged to reduce likelihoods of the causes of the failures, associated with the failure codes, reoccurring during the re-execution of the failed test cases;

re-executing, by the one or more processors, the failed test cases based on the execution order;
collecting, by the one or more processors, a plurality of test results associated with the test cases distributed to the plurality of parallel threads, wherein the plurality of test results indicates whether the plurality of test cases ultimately succeeded during at least one of the executing or the re-executing; and
aggregating, by the one or more processors, the plurality of test results.

2. The computer-implemented method of claim 1 , wherein the plurality of test cases comprises a set of classes, individual classes of the set of classes comprise one or more methods, and the individual test cases are distributed at one or more of a class level or a method level.

3. The computer-implemented method of claim 1 , wherein:
the failed test cases comprise:
first test cases associated with a first failure code of the failure codes, and
second test cases associated with a second failure code of the failure codes,

the execution order causes the one or more processors to:
re-execute the first test cases substantially immediately following failure of the first test cases based on the first failure code, and
re-execute the second test cases on a delayed basis after a period of time following failure of the second test cases based on the second failure code.

4. The computer-implemented method of claim 1 , further comprising:
assigning, by the one or more processors, the failed test cases to at least one retry test set; and
re-executing, by the one or more processors, the failed test cases in the at least one retry test set based on the execution order.

5. The computer-implemented method of claim 4 , wherein:
the at least one retry test set includes a single retry test set, and
re-executing the failed test cases comprises executing the failed test cases, in the single retry test set, in sequence in a single thread based on the execution order.

6. The computer-implemented method of claim 4 , wherein:
the at least one retry test set includes a plurality of retry test sets associated with the plurality of parallel threads, and
the method further comprises assigning, by the one or more processors, the failed test cases among the plurality of retry test sets.

7. The computer-implemented method of claim 6 , wherein determining the execution order comprises determining, by the one or more processors, different execution orders associated with different retry test sets, of the plurality of retry test sets, based on one or more of the failure codes or an original structure of the plurality of test cases.

8. The computer-implemented method of claim 7 , wherein determining the different execution orders comprises identifying, by the one or more processors, relationships between test cases in individual retry test sets, of the plurality of retry test sets, using a machine learning model.

9. A computing device, comprising:
one or more processors;
memory storing computer-executable instructions that, when executed by the one or more processors, cause the computing device to perform operations comprising:
identifying a plurality of test cases that test interactions between a software application and a database;
distributing individual test cases, of the plurality of test cases, among a plurality of parallel threads on the computing device;
executing the plurality of test cases in the plurality of parallel threads;
identifying failed test cases, of the plurality of test cases, based on the executing;
identifying failure codes associated with the failed test cases, wherein the failure codes indicate causes of failures of the failed test cases;
determining, based on the failure codes, an execution order for re-execution of the failed test cases, wherein the execution order is arranged to reduce likelihoods of the causes of the failures, associated with the failure codes, reoccurring during the re-execution of the failed test cases;
re-executing the failed test cases based on the execution order; and
aggregating test results associated with the executing and the re-executing.

10. The computing device of claim 9 , wherein the plurality of test cases comprises a set of classes, individual classes of the set of classes comprise one or more methods, and the individual test cases are distributed at one or more of a class level or a method level.

11. The computing device of claim 9 , wherein:
a failure code, of the failure codes, indicates that a particular failed test case failed due to a record-locking error or a table-locking error associated with the database, and
the execution order indicates a substantially immediate re-execution of the particular failed test case, following failure of the particular failed test case, based on the failure code.

12. The computing device of claim 9 , wherein:
a failure code, of the failure codes, indicates that a particular failed test case failed due to a missing data error associated with the database, and
re-executing the failed test cases comprises:
adding the particular failed test case to a test failure log; and
re-executing the particular failed test case, based on the execution order, as part of a retry test set that includes other failed test cases.

13. The computing device of claim 9 , wherein re-executing the failed test cases is performed as part of a retry test set executed in a single thread.

14. The computing device of claim 9 , wherein re-executing the failed test cases comprises:
distributing the failed test cases among a plurality of retry test sets associated with the plurality of parallel threads; and
executing the plurality of retry test sets in the plurality of parallel threads.

15. One or more non-transitory computer-readable media storing computer-executable instructions that, when executed by one or more processors, cause the one or more processors to perform operations comprising:
identifying a plurality of test cases that test interactions between a software application and a database;
distributing individual test cases of the plurality of test cases among a plurality of test sets;
executing different test sets of the plurality of test sets simultaneously in different parallel threads;
determining failed test cases, of the plurality of test cases, based on the executing;
determining failure codes corresponding to the failed te

Assignees

• State Farm Mutual Automobile Insurance Co

Inventors

• Chauhan Shaktiraj
• Shepherd Nate

Classifications

• G06F11/3698

  Environments for analysis, debugging or testing of software · CPC title

• G06N20/00

  Machine learning · CPC title

• G06F11/3688 Primary

  for test execution, e.g. scheduling of test suites · CPC title

• G06F11/3692

  for test results analysis · CPC title

• G06F11/3696

  Methods or tools to render software testable · CPC title