Selecting a set of test configurations associated with a particular coverage strength using a constraint solver

Patent metadata

Field	Value
Publication number	US-10248550-B2
Application number	US-201715715145-A
Country	US
Kind code	B2
Filing date	Sep 25, 2017
Priority date	Dec 16, 2016
Publication date	Apr 2, 2019
Grant date	Apr 2, 2019

Abstract

Techniques for selecting test configurations associated with a particular coverage strength using a constraint solver are disclosed. A set of parameters are configurable for conducting a test on a particular target application. A data model generator identifies one or more candidate test configurations based on the set of parameters. The data model generator determines a set of interactions based on a desired coverage strength. The data model generator specifies a selection variable indicating the candidate test configurations that are selected for testing the particular target application. The data model generator specifies constraint(s) minimizing the number of selected test configurations. The data model generator specifies constraint(s) requiring that each interaction be covered by at least one selected test configuration. The data model, including the selection variable and the constraints, are input to a constraint solver to obtain a set of selected test configurations associated with the desired coverage strength.

First claim

What is claimed is:

1. A non-transitory computer readable medium comprising instructions which, when executed by one or more hardware processors, causes performance of a set of operations, comprising:
identifying a set of test parameters for testing a target application;
identifying, for each test parameter of the set of test parameters, a respective set of candidate values for the test parameter;
identifying a set of candidate test configurations, each associated with a respective combination of candidate values for the set of test parameters;
identifying a desired coverage strength;
identifying subgroups of test parameters, from the set of test parameters, each of the subgroups of test parameters including a number of test parameters that is equal to the desired coverage strength;
identifying a set of interactions, wherein:
each interaction is associated with one of the subgroups of test parameters;
each interaction is associated with a respective combination of candidate values for the associated subgroup of test parameters; and
the set of interactions covers every possible combination of candidate values for each of the subgroups of parameters;

specifying a selection variable, indicating which of the set of candidate test configurations have been selected;
specifying a set of one or more constraints that:
minimizes a cost of testing the target application using a subset of the set of candidate test configurations that is selected based on the selection variable; and
requires each of the set of interactions to be covered by at least one of the subset of the set of candidate test configurations that is selected based on the selection variable;

storing a data model, applicable to a constraint solver, comprising:
the selection variable; and
the set of constraints;

wherein the target application is executed using at least one of the subset of the set of candidate test configurations that is selected based on the selection variable.

2. The medium of claim 1 , wherein:
the operations further comprise: specifying a plurality of coverage variables, wherein each of the plurality of coverage variables respectively corresponds to the set of interactions, and each of the plurality of coverage variables indicates which of the set of candidate test configurations cover the corresponding interaction; and
specifying the set of one or more constraints comprises: requiring that the dot product of (a) the selection variable and (b) each of the plurality of coverage variables be equal to or greater than one.

3. The medium of claim 1 , wherein:
the operations further comprise: obtaining a cost variable, indicating a cost associated with each of the set of candidate test configurations; and
the set of one or more constraints comprises: minimizing the dot product of (a) the selection variable and (b) the cost variable.

4. The medium of claim 1 , wherein:
the selection variable comprises a set of elements, respectively corresponding to the set of candidate test configurations, wherein each of the set of elements assumes a first value if the corresponding candidate test configuration is selected or assumes a second value if the corresponding candidate test configuration is not selected; and
the set of one or more constraints comprises: minimizing the summation of values assumed by the set of elements of the selection variable.

5. The medium of claim 1 , wherein the constraint solver comprises at least one of: a constraint programming solver and a mathematical programming solver.

6. The medium of claim 1 , wherein the set of candidate test configurations covers every possible combination of candidate values for the set of parameters.

7. The medium of claim 1 , wherein the set of candidate test configurations is only a subset of a complete set of candidate test configurations that covers every possible combination of candidate values for the set of parameters.

8. The medium of claim 1 , wherein the operations further comprise:
selecting one or more additional candidate test configurations, which were not included in the set of candidate test configurations, based on a column generation method; and
storing a new data model, applicable to the constraint solver, based on both (a) the set of candidate test configurations and (b) the additional candidate test configurations.

9. The medium of claim 1 , wherein the set of candidate test configurations includes a particular test configuration, and the plurality of coverage variables indicates that the particular test configuration covers every interaction.

10. The medium of claim 1 , wherein:
executing the target application using a first candidate test configuration, of the set of candidate test configurations, is completed in a shorter duration of time than executing the target application using a second candidate test configuration, of the set of candidate test configurations;
the operations further comprise: obtaining a cost corresponding to each of the set of candidate test configurations, wherein a first cost corresponding to the first candidate test configuration is less than a second cost corresponding to the second candidate test configuration.

11. The medium of claim 1 , wherein:
a first candidate test configuration, of the set of candidate test configurations, is associated with a larger number of known problems than a second candidate test configuration, of the set of candidate test configurations;
the operations further comprise: obtaining a cost corresponding to each of the set of candidate test configurations, wherein a first cost corresponding to the first candidate test configuration is less than a second cost corresponding to the second candidate test configuration.

12. The medium of claim 1 , wherein:
a first candidate test configuration, of the set of candidate test configurations, is associated with a larger number of end users than a second candidate test configuration, of the set of candidate test configurations;
the operations further comprise: obtaining a cost corresponding to each of the set of candidate test configurations, wherein a first cost corresponding to the first candidate test configuration is less than a second cost corresponding to the second candidate test configuration.

13. The medium of claim 1 , wherein:
a first candidate test configuration, of the set of candidate test configurations, is associated with an invalid combination of candidate values for the set of parameters;
a second candidate test configuration, of the set of candidate test configurations, is associated with a valid combination of candidate values for the set of parameters;
the operations further comprise: obtaining a cost corresponding to each of the set of candidate test configurations, wherein a first cost corresponding to the first candidate test configuration is greater than a second cost corresponding to the second candidate test configuration.

14. The medium of claim 1 , wherein the operations further comprise:
applying the data model to the constraint solver.

15. The medium of claim 1 , wherein the operations further comprise:
iterating a test on the target application using the subset of the set of candidate test configurations that is selected based on the selection variable, while refraining from iterating the test on the target application using any candidate test configuration that is not selected based on the selection variable.

16. The medium of claim 1 , wherein the target application comprises at least one of: a software application and a hardware design.

17. The medium of claim

Assignees

• Oracle Int Corp

Inventors

• Kadioglu Serdar
• Sebbah Samir

Classifications

• G06F11/3672 Primary

  Test management · CPC title

• G06F11/3688

  for test execution, e.g. scheduling of test suites · CPC title

• G06F11/368 Primary

  for test version control, e.g. updating test cases to a new software version · CPC title