Systems and methods for software regression detection

Patent metadata

Field	Value
Publication number	US-2020371902-A1
Application number	US-201916508097-A
Country	US
Kind code	A1
Filing date	Jul 10, 2019
Priority date	May 21, 2019
Publication date	Nov 26, 2020
Grant date	—

Abstract

The disclosed Test Case Prioritization (TCP) system includes a TCP server that is designed to generate a TCP Model, which stores identified relationships between software files and test cases based on test results. The TCP server groups the test cases by track and/or build, and clusters the test cases that correspond to failed test results within each group. The TCP server also determines which software files have been modified since a previous testing of the software files. The TCP server then correlates the clusters of test cases and the modified software files to construct the TCP model. Once the TCP model has been generated, the TCP server can use the TCP model to provide useful information during software development and testing.

First claim

1 . A Test Case Prioritization (TCP) system, comprising:
at least one memory configured to store software files and test cases; and at least one processor configured to execute instructions stored in the at least one memory to cause the TCP system to perform operations comprising:
executing the software files according to the test cases to generate test results in the at least one memory, wherein each of the test results is associated with both a respective test case and a respective software file;
grouping and clustering the test cases into test case clusters;
identifying the software files that have been modified since a previous preflight verification; and
correlating the test case clusters with the modified software files to generate a Test Case Prioritization (TCP) model.

2 . The TCP system of claim 1 , wherein the software files and the test cases are stored in a database that is stored in the at least one memory.

3 . The TCP system of claim 1 , wherein grouping and clustering the test cases into test case clusters comprises:
grouping the test results by track, by build, or a combination thereof; and providing the grouped test results to a machine-learning (ML)-based component of the TCP system, wherein the ML-based component is configured to cluster failed test results in the grouped test results to generate the test case clusters.

4 . The TCP system of claim 3 , wherein correlating the test case clusters with the modified software files comprises:
grouping the modified software files by track, by build, or a combination thereof; and providing the grouped modified software files to the ML-based component of the TCP system, wherein the ML-based component is configured to correlate the test case clusters and the grouped modified software files to generate the TCP model.

5 . The TCP system of claim 4 , wherein the ML-based component is an artificial neural network (ANN).

6 . The TCP system of claim 5 , wherein the ANN comprises a restricted Boltzmann machine.

7 . The TCP system of claim 1 , wherein the at least one processor is configured to execute the instructions stored in the at least one memory to cause the TCP system to perform operations comprising:
providing, as input to the generated TCP model, one or more of the software files; receiving, as output from the generated TCP model, a minimum set of the test cases to be applied to verify the one or more software files.

8 . The TCP system of claim 7 , wherein the minimum set of the test cases includes a respective test case from each test case cluster associated with each of the one or more software files.

9 . The TCP system of claim 8 , wherein the at least one processor is configured to execute the instructions stored in the at least one memory to cause the TCP system to perform operations comprising:
executing the one or more software files according to the minimum set of test cases to generate a minimum set of test results; and in response to the minimum set of test results comprising only successes, providing an indication of a successful preflight verification.

10 . The TCP system of claim 9 , wherein the at least one processor is configured to execute the instructions stored in the at least one memory to cause the TCP system to perform operations comprising:
in response to the minimum set of test results comprising one or more failures:
providing the one or more failed test results as input to the generated TCP model; and
receiving, as output from the generated TCP model, a second set of the test cases to be applied to verify the one or more software files based on the one or more failed test results.

11 . The TCP system of claim 1 , wherein the at least one processor is configured to execute the instructions stored in the at least one memory to cause the TCP system to perform operations comprising:
providing one or more failed test results as input to the generated TCP model; receiving, as output from the generated TCP model, one or more of the software files that are correlated with the one or more failed test results when the generated TCP model correlates the one or more failed test results with the one or more software files; and receiving, as the output from the generated TCP model, an indication that the one or more failed test results are flappers when the generated TCP model does not correlate the one or more failed test results with the one or more software files.

12 . The TCP system of claim 1 , wherein the at least one processor is configured to execute the instructions stored in the at least one memory to cause the TCP system to perform operations comprising:
providing one of the software files as input to the generated TCP model; receiving, as output from the generated TCP model, a value indicating a likelihood that modifying the software file will result in a regression based on a number of the test cases that are correlated with the modified software file in the generated TCP model.

13 . A method of generating a Test Case Prioritization (TCP) model, comprising:
executing software files according to test cases to generate test results, wherein each of the test results is associated with both a respective test case and a respective software file; grouping and clustering the test cases into test case clusters; identifying the software files that have been modified since a previous preflight verification; and correlating the test case clusters with the modified software files to generate the TCP model.

14 . The method of claim 13 , comprising:
retrieving a previous version of the software files from a memory of a database; and wherein identifying the software files that have been modified since the previous preflight verification is based on a comparison between the software files and the previous version of the software files.

15 . The method of claim 13 , wherein grouping and clustering the test cases into test case clusters comprises:
grouping the test results by track, by build, or a combination thereof; and providing the grouped test results to a machine-learning (ML)-based component, wherein the ML-based component is configured to cluster failed test results in the grouped test results to generate the test case clusters.

16 . The method of claim 13 , comprising providing one or more failed test results as input to the generated TCP model;
receiving, as output from the generated TCP model, one or more of the software files that are correlated with the one or more failed test results when the generated TCP model correlates the one or more failed test results with the one or more software files; and receiving, as the output from the generated TCP model, an indication that the one or more failed test results are flappers when the generated TCP model does not correlate the one or more failed test results with the one or more software files.

17 . A tangible, non-transitory, machine-readable medium, comprising machine-readable instructions, wherein the machine-readable instructions, when executed by one or more processors cause the one or more processors to:
executing software files according to test cases to generate test results, wherein each of the test results is associated with both a respective test case and a respective software file; grouping and clustering the test cases into test case clusters; identifying the software files that have been modified since a previous preflight verification; correlating the test case clusters with the modified software files to generate a Test Case Prioritization (TCP) model; providing, as

Assignees

• Servicenow Inc

Inventors

• Mangione-Tran Carmine
• Kathuria Raman

Classifications

• G06N3/044

  Recurrent networks, e.g. Hopfield networks · CPC title

• G06N3/047

  Probabilistic or stochastic networks · CPC title

• G06F11/3688 Primary

  for test execution, e.g. scheduling of test suites · CPC title

• G06F11/3692

  for test results analysis · CPC title

• G06N20/00

  Machine learning · CPC title