Automated machine learning test system

Patent metadata

Field	Value
Publication number	US-11775878-B2
Application number	US-202117523607-A
Country	US
Kind code	B2
Filing date	Nov 10, 2021
Priority date	Dec 22, 2020
Publication date	Oct 3, 2023
Grant date	Oct 3, 2023

Abstract

A computing device selects new test configurations for testing software. Software under test is executed with first test configurations to generate a test result for each test configuration. Each test configuration includes a value for each test parameter where each test parameter is an input to the software under test. A predictive model is trained using each test configuration of the first test configurations in association with the test result generated for each test configuration based on an objective function value. The predictive model is executed with second test configurations to predict the test result for each test configuration of the second test configurations. Test configurations are selected from the second test configurations based on the predicted test results to define third test configurations. The software under test is executed with the defined third test configurations to generate the test result for each test configuration of the third test configurations.

First claim

What is claimed is:

1. A non-transitory computer-readable medium having stored thereon computer-readable instructions that when executed by a computing device cause the computing device to:
(A) execute software under test with a first plurality of test configurations to generate a test result for each test configuration of the first plurality of test configurations, wherein each test configuration of the first plurality of test configurations includes a value for each test parameter of a plurality of test parameters, wherein each test parameter of the plurality of test parameters is an input to the software under test, wherein the test result includes at least one of a severity code, a memory consumption value, or an execution time value, wherein the severity code indicates whether or not execution of the software under test failed using a respective test configuration, wherein the memory consumption value indicates an amount of computer memory used for execution of the software under test using the respective test configuration, wherein the execution time value indicates an amount of time used for execution of the software under test using the respective test configuration;
(B) train a predictive model using each test configuration of the first plurality of test configurations and a target variable value that is the test result generated for each test configuration of the first plurality of test configurations based on an objective function value;
(C) execute the trained predictive model with a second plurality of test configurations to predict the test result for each test configuration of the second plurality of test configurations, wherein each test configuration of the second plurality of test configurations includes the value for each test parameter of the plurality of test parameters;
(D) select a predefined number of test configurations from the second plurality of test configurations based on the predicted test result for each test configuration of the second plurality of test configurations, wherein the selected predefined number of test configurations define a third plurality of test configurations, wherein the third plurality of test configurations are selected to maximize the memory consumption value or the execution time value or to result in failed execution of the software under test;
(E) execute the software under test with the defined third plurality of test configurations to generate the test result for each test configuration of the third plurality of test configurations, wherein each test configuration of the third plurality of test configurations includes the value for each test parameter of the plurality of test parameters;
(AA) retrain the predictive model using each test configuration of the first plurality of test configurations in association with the test result generated for each test configuration of the first plurality of test configurations and using each test configuration of the third plurality of test configurations in association with the test result generated for each test configuration of the third plurality of test configurations;
(BB) update the second plurality of test configurations by removing the third plurality of test configurations from the second plurality of test configurations;
(CC) execute the predictive model retrained in (AA) with the updated second plurality of test configurations to predict the test result for each test configuration of the second plurality of test configurations;
(DD) select the predefined number of test configurations from the updated second plurality of test configurations based on the predicted test result for each test configuration of the second plurality of test configurations in (CC), wherein the selected predefined number of test configurations define a fourth plurality of test configurations, wherein each test configuration of the fourth plurality of test configurations includes the value for each test parameter of the plurality of test parameters, wherein each test configuration of the fourth plurality of test configurations is unique relative to other test configurations of the fourth plurality of test configurations and is unique relative to the first plurality of test configurations and relative to the third plurality of test configurations;
(EE) execute the software under test with the fourth plurality of test configurations to generate the test result for each test configuration of the fourth plurality of test configurations; and
(F) output the generated test result for each test configuration of the first plurality of test configurations, for each test configuration of the third plurality of test configurations, and for each test configuration of the fourth plurality of test configurations to identify errors in the software under test.

2. The non-transitory computer-readable medium of claim 1 , wherein a model type of the predictive model is a random forest model type.

3. The non-transitory computer-readable medium of claim 1 , wherein a model type of the predictive model is a Bayesian optimization model type.

4. The non-transitory computer-readable medium of claim 1 , wherein after (E), the computer-readable instructions further cause the computing device to:
determine an importance value for each test parameter of the plurality of test parameters using a second predictive model trained using each test configuration of the first plurality of test configurations in association with the test result generated for each test configuration of the first plurality of test configurations, and using each test configuration of the third plurality of test configurations in association with the test result generated for each test configuration of the third plurality of test configurations; and
output the determined importance value for each test parameter of the plurality of test parameters.

5. The non-transitory computer-readable medium of claim 4 , wherein a model type of the second predictive model is a random forest model type.

6. The non-transitory computer-readable medium of claim 1 , wherein each test configuration of the first plurality of test configurations is unique relative to other test configurations of the first plurality of test configurations.

7. The non-transitory computer-readable medium of claim 1 , wherein each test configuration of the third plurality of test configurations is unique relative to other test configurations of the third plurality of test configurations and is unique relative to the first plurality of test configurations.

8. The non-transitory computer-readable medium of claim 4 , wherein a highest determined importance value for a test parameter of the plurality of test parameters indicates that the test parameter contributed most to the test result generated for each test configuration of the first plurality of test configurations.

9. The non-transitory computer-readable medium of claim 1 , wherein before (A), the computer-readable instructions further cause the computing device to define a plurality of test configurations based on random selection of a value for each test parameter of the plurality of test parameters, wherein the value is selected between a minimum value and a maximum value defined for each test parameter of the plurality of test parameters, wherein the first plurality of test configurations are randomly selected from the defined plurality of test configurations.

10. The non-transitory computer-readable medium of claim 9 , wherein the second plurality of test configurations are the defined plurality of test configurations after excluding the selected first plurality of test configurations.

11. The non-transitory computer-readable medium of claim 1 , wherein before (A), th

Assignees

• Sas Inst Inc

Inventors

• Gao Yan
• Griffin Joshua David
• Lin Yu-Min
• Pederson Bengt Wisen
• Tharrington Jr Ricky Dee
• Tan Pei-Yi
• Wright Raymond Eugene

Classifications

• G06N20/20 Primary

  Ensemble learning · CPC title

• G06N7/01

  Probabilistic graphical models, e.g. probabilistic networks · CPC title

• G06F11/3688

  for test execution, e.g. scheduling of test suites · CPC title

• G06N5/01

  Dynamic search techniques; Heuristics; Dynamic trees; Branch-and-bound · CPC title