Automatic regression identification

Patent metadata

Field	Value
Publication number	US-10509719-B2
Application number	US-201515758284-A
Country	US
Kind code	B2
Filing date	Sep 8, 2015
Priority date	Sep 8, 2015
Publication date	Dec 17, 2019
Grant date	Dec 17, 2019

Abstract

Example implementations relate to automatically identifying regressions. Some implementations may include a data capture engine to capture data points during test executions of the application under test. The data points may include, for example, test action data and application action data. Additionally, some implementations may include a data correlation engine to correlate each of the data points with a particular test execution of the test executions, and each of the data points may be correlated based on a sequence of events that occurred during the particular test execution. Furthermore, some implementations may also include a regression identification engine to automatically identify, based on the correlated data points, a regression between a first version of the application under test and a second version of the application under test.

First claim

We claim:

1. A system for automatic regression identification comprising:
a processor; and
a non-transitory machine-readable storage medium storing instructions that are executable to cause the processor to:
capture data points during test executions of an application under test, the data points including test action data and application action data;
correlate each of the data points with a particular test execution of the test executions based on a sequence of events that occurred during the particular test execution, wherein to correlate each of the data points with the particular test execution, the instructions are executable to cause the processor to:
categorize each of the data points into one of a plurality of types, including a first type;
assign the data points categorized into the first type as backbone nodes of a directed acyclic graph; and
connect remaining data points that are not categorized into the first type as data nodes to the backbone nodes via directed edges based on timestamps associated with the remaining data points; and

automatically identify, based on the correlated data points, a regression between a first version of the application under test and a second version of the application under test.

2. The system of claim 1 , wherein the regression is identified based on a similarity between: (1) first tests executed related to the first version and second tests executed related to the second version; and (2) first application behaviors in response to the first tests and second application behaviors in response to the second tests.

3. The system of claim 1 , wherein, to identify the regression, the instructions are executable to cause the processor to:
identify a defect of the application under test, the defect occurring during a first test execution of the test executions;
identify, based on the correlated data points, a second test execution of the test executions, the second test execution having a similar test identification to a test identification of the first test execution; and
determine the regression by comparing, based on the correlated data points, the first test execution with the second test execution, the first test execution including a first set of test steps and a first set of application behavior, and the second test execution including a second set of test steps and a second set of application behavior.

4. The system of claim 3 , wherein the comparing of the first test execution with the second test execution is based on a generalized data sequence.

5. The system of claim 3 , wherein, to determine the regression by comparing, based on the correlated data points, the first test execution with the second test execution, the instructions are executable to cause the processor to:
determine whether the first set of test steps are a same as the second set of test steps;
determine, using a similarity algorithm, whether the first set of application behavior and the second set of application behavior are similar; and
determine whether the defect occurs during the second test execution.

6. The system of claim 5 , wherein the instructions are executable to cause the processor to automatically identify the regression in response to determinations that the first set of test steps are the same as the second set of test steps, the first set of application behavior is similar to the second set of application behavior, and the defect does not occur during the second test execution.

7. The system of claim 6 , wherein the first test execution occurs after the second test execution.

8. The system of claim 1 , wherein the instructions are executable to cause the processor to:
assign metadata as vertices connected to the backbone nodes; and
assign tags to the directed edges to identify a type of relation between the vertices.

9. The system of claim 8 , wherein to identify the regression, the instructions are executable to cause the processor to:
receive an indication of a defect of the first version of the application under test, the defect occurring during a first test execution of the test executions, the first test execution being associated with a first set of backbone nodes of the backbone nodes of the directed acyclic graph;
query the directed acyclic graph to identify a second set of backbone nodes that are similar to the first set, the second set of backbone nodes being associated with a second test execution of the second version of the application under test;
compare data nodes, metadata, and tags corresponding to the first test execution to data nodes, metadata, and tags corresponding to the second test execution; and
automatically identify the regression by determining, based on the comparison, that (1) the defect occurred during the first test execution and did not occur during the second test execution; and (2) there was no difference in behavior of the application under test during the first test execution and the second test execution.

10. A non-transitory machine-readable storage medium including instructions which, when executed by a processor, cause the processor to:
receive an indication of a defect of an application under test, the defect occurring during a first test execution of a plurality of test executions, the first test execution being associated with a first test identification;
query a correlated data structure to identify an initial set of test executions of the plurality of test executions, the initial set of test executions having a same test identification as the first test identification;
narrow, based on the correlated data structure, the initial set of test executions to generate a narrowed set of test executions, the narrowed set of test executions being test executions that experience similar application behavior in response to the first test execution but not including the defect; and
automatically identify each version of the application under test corresponding to the narrowed set of test executions as a regression from a version of the application under test tested during the first test execution.

11. The non-transitory machine-readable storage medium of claim 10 , wherein the correlated data structure includes data points that have been correlated with a particular test execution of the plurality of test executions, each of the data points being correlated based on a sequence of events that occurred during the particular test execution.

12. The non-transitory machine-readable storage medium of claim 10 , wherein the indication of the defect is received via an interaction with a regression interface.

13. A computer-implemented method comprising:
receiving an indication of a defect of an application under test, the defect occurring during a first test execution of a plurality of test executions of the application under test;
accessing, by a processor, data from a correlated data structure related to a plurality of test executions, the correlated data structure including a generalized data sequence corresponding to a sequence of test steps leading up to the defect;
identifying, by the processor, using the data from the correlated data structure:
a first set of versions of the application under test where the generalized data sequence and the defect occurred;
a second set of versions of the application under test where the generalized data sequence occurred but the defect did not occur;
a third set of versions of the application under test where the generalized data sequence occurred only for backbone nodes of the correlated data structure; and
a fourth set of versions of the application under test where the generalized data sequence did not occur; and

causing

Assignees

• Entit Software Llc
• Micro Focus Llc

Inventors

• Shani Inbar
• Cohen Ayal
• Burg Yaron

Classifications

• G06F11/3692

  for test results analysis · CPC title

• G06F11/3688 Primary

  for test execution, e.g. scheduling of test suites · CPC title

• G06F8/71

  Version control (security arrangements therefor G06F21/57); Configuration management · CPC title

• G06F16/9024

  Graphs; Linked lists (G06F16/9027 takes precedence) · CPC title