Retrying failed test cases in software testing using parallel threads

Patent metadata

Field	Value
Publication number	US-2023333973-A1
Application number	US-202318336757-A
Country	US
Kind code	A1
Filing date	Jun 16, 2023
Priority date	Feb 23, 2021
Publication date	Oct 19, 2023
Grant date	—

Abstract

Test cases written to test a software application can be dynamically distributed among different sets of test cases that can be executed simultaneously in different parallel threads, thereby speeding up testing relative to executing the test cases sequentially in a single thread. Although executing the test cases in parallel threads can cause the test cases to execute in a different order than intended by developers, any test cases that fail due to database conflicts or other errors as a result of the parallelization can be retried at least once. Re-executing failed test cases at a later time can lead to the full set of test cases ultimately succeeding, despite errors that may have been caused by the parallelization.

First claim

What is claimed is:

1 . A computer-implemented method, comprising:
distributing, by one or more processors, a group of test cases among different test sets, wherein the group of test cases is configured to test interactions between a software application and a database; executing, by the one or more processors, the different test sets in parallel; identifying, by the one or more processors, failed test cases that failed during execution of the different test sets in parallel; determining, by the one or more processors, failure codes that indicate causes of failures of the failed test cases; determining, by the one or more processors, and based on the failure codes, an execution order for re-execution of the failed test cases; and re-executing, by the one or more processors, the failed test cases based on the execution order.

2 . The computer-implemented method of claim 1 , wherein:
executing the different test sets in parallel is associated with a first likelihood of the causes of the failures, and re-executing the failed test cases based on the execution order is associated with a second likelihood, less than the first likelihood, of the causes of the failures reoccurring during re-execution of the failed test cases.

3 . The computer-implemented method of claim 1 , wherein:
the execution order is determined based at least in part by a machine learning model, and the machine learning model is trained, based on historical data indicating successful reexecutions of historical failed test cases associated with the failure codes, to determine an arrangement of the execution order that is predicted to reduce likelihoods of the causes of the failures reoccurring during re-execution of the failed test cases.

4 . The computer-implemented method of claim 1 , wherein:
the failed test cases comprise:
one or more first test cases associated with a first failure code of the failure codes, and
one or more second test cases associated with a second failure code of the failure codes, and
the execution order causes the one or more processors to:
re-execute the one or more first test cases substantially immediately following failure of the one or more first test cases, based on the first failure code, and
re-execute the one or more second test cases after a period of time following failure of the one or more second test cases, based on the second failure code.

5 . The computer-implemented method of claim 4 , wherein:
the first failure code indicates that database locking errors caused the one or more first test cases to fail, and the execution order is arranged, based on the first failure code, to cause re-execution of the one or more first test cases substantially immediately following the failure of the one or more first test cases.

6 . The computer-implemented method of claim 4 , wherein:
the second failure code indicates that missing data errors caused the one or more second test cases to fail, and the execution order is arranged, based on the second failure code, to delay re-execution of the one or more second test cases for the period of time, following the failure of the one or more second test cases, to reduce likelihoods of the missing data errors re-occuring during re-execution of the one or more second test cases.

7 . The computer-implemented method of claim 1 , wherein re-executing the failed test cases based on the execution order comprises:
distributing, by the one or more processors, the failed test cases among different retry test sets; and executing, by the one or more processors, the different retry test sets in parallel.

8 . The computer-implemented method of claim 7 , wherein determining the execution order comprises determining different execution orders associated with the different retry test sets.

9 . The computer-implemented method of claim 1 , wherein:
the group of test cases was created based on an expectation of individual test cases, of the group of test cases, being executed in a sequential execution order, and the failures of one or more of the failed test cases are caused at least in part by executing the different test sets in parallel, rather than executing the individual test cases in the sequential execution order.

10 . The computer-implemented method of claim 1 , further comprising:
collecting, by the one or more processors, test results associated with the group of test cases, wherein the test results indicate whether individual test cases succeeded during at least one of:
execution of the different test sets in parallel, or
re-execution of the failed test cases; and
combining, by the one or more processors, the test results into an aggregated test result report.

11 . A computing device, comprising:
one or more processors; memory storing computer-executable instructions that, when executed by the one or more processors, cause the one or more processors to perform operations comprising:
distributing a group of test cases, configured to test interactions between a software application and a database, among at least a first test set and a second test set;
executing the first test set and the second test set in parallel;
identifying failed test cases that failed during execution of the first test set and the second test set in parallel;
determining, based on failure codes indicating causes of failures of the failed test cases, an execution order for re-execution of the failed test cases; and
re-executing the failed test cases based on the execution order.

12 . The computing device of claim 11 , wherein the execution order is determined to reduce likelihoods of the causes of the failures, associated with the failure codes, reoccurring during the re-execution of the failed test cases.

13 . The computing device of claim 11 , wherein:
the failed test cases comprise:
one or more first test cases associated with a first failure code of the failure codes, and
one or more second test cases associated with a second failure code of the failure codes, and
the execution order causes the one or more processors to:
re-execute the one or more first test cases substantially immediately following failure of the one or more first test cases, based on the first failure code, and
re-execute the one or more second test cases after a period of time following failure of the one or more second test cases, based on the second failure code.

14 . The computing device of claim 11 , wherein re-executing the failed test cases based on the execution order comprises:
distributing the failed test cases among different retry test sets; and executing the different retry test sets in parallel.

15 . The computing device of claim 11 , wherein:
the group of test cases was created based on an expectation of individual test cases, of the group of test cases, being executed in a sequential execution order, and the failures of one or more of the failed test cases are caused at least in part by executing the first test set and the second test set in parallel, rather than executing the individual test cases in the sequential execution order.

16 . One or more non-transitory computer-readable media storing computer-executable instructions that, when executed by one or more processors, cause the one or more processors to perform operations comprising:
distributing a group of test cases, configured to test interactions between a software application and a database, among different test sets; executing the different test sets in parallel; identifying failed test cases that failed during execution of the different test

Assignees

• State Farm Mutual Automobile Insurance Co

Inventors

• Chauhan Shaktiraj
• Shepherd Nate

Classifications

• G06F11/3698

  Environments for analysis, debugging or testing of software · CPC title

• G06F11/3688 Primary

  for test execution, e.g. scheduling of test suites · CPC title

• G06F11/3692

  for test results analysis · CPC title

• G06F11/3696

  Methods or tools to render software testable · CPC title

• G06N20/00

  Machine learning · CPC title