Validation sets for machine learning algorithms

Patent metadata

Field	Value
Publication number	US-10754764-B2
Application number	US-201916692172-A
Country	US
Kind code	B2
Filing date	Nov 22, 2019
Priority date	Apr 22, 2018
Publication date	Aug 25, 2020
Grant date	Aug 25, 2020

Abstract

A computing device receives data comprising inputs representing a respective option for each of factors in each of test cases. The data comprises a response of the system for each of the test cases. The computing device receives a request requesting an evaluation of the data for generating a model (e.g. a machine learning algorithm) to predict responses based on the factors. The computing device obtains different group identifiers for each of groups for distributing the test cases for the system (e.g., groups of a K-fold cross-validation). The computing device for each of validation(s): generates a data set comprising a respective data element for each of the test cases of the plurality of test cases; and controls assignment of a group identifier of the different group identifiers to each of the respective data elements. The computing device outputs an indication of one or more generated data sets for the validation(s).

First claim

What is claimed is:

1. A computer-program product tangibly embodied in a non-transitory machine-readable storage medium, the computer-program product including instructions operable to cause a computing device to:
receive data for a plurality of test cases for testing a system, wherein the data comprises inputs, wherein the inputs represent a respective option for each of a plurality of factors in each of respective test cases of the plurality of test cases, and the data comprises a response of the system for each of the respective test cases of the plurality of test cases;
receive a request requesting an evaluation of the data for generating a model to predict responses based on the plurality of factors;
obtain different group identifiers for each of different groups for distributing the plurality of test cases for the system, wherein at least one group of the different groups is a test group for testing a respective test model derived from inputs of one or more training groups of remaining groups of the different groups;
obtain a validation indication indicating one or more validations for validating a model derived from the data;
for each of the one or more validations:
generate a data set comprising a respective data element for each of the test cases of the plurality of test cases; and
control assignment of a group identifier of the different group identifiers to each of the respective data elements; and

output an indication of one or more generated data sets for the one or more validations, the one or more generated data sets for evaluating the data for generating the model to predict the responses based on the plurality of factors.

2. The computer-program product of claim 1 , wherein the instructions are operable to cause a computing device to, for each of multiple validations of the one or more validations, control assignment of the different group identifiers to each test case of the plurality of test cases such that all test cases identified by a same group identifier of a given validation of the multiple validations are distributed among as many of different groups of other validations of the multiple validations as possible given a criterion for distributing the test cases of a validation according to an orthogonal design.

3. The computer-program product of claim 1 , wherein the instructions are operable to cause a computing device to control assignment of the group identifier of the different group identifiers to each respective data element of the data set by, for one or more iterations:
determining an initial distribution of the test cases in the different groups by assigning each of the different group identifiers to each of the respective data elements;
evaluating the initial distribution by determining an initial score, wherein the initial score indicates the initial distribution compared to an orthogonal design for distributing the test cases in the different groups, wherein the initial distribution comprises an assignment of a first group identifier of the different group identifiers to a first data element for a first test case of the initial distribution;
determining an updated distribution of the test cases in the different groups by changing the assignment of the first group identifier to the first data element to an assignment of a second group identifier of the different group identifiers:
evaluating the updated distribution by:
determining an updated score, wherein the updated score indicates the updated distribution compared to the orthogonal design for distributing the test cases in the different groups;
determining a first evaluation by comparing the updated score to the initial score; and
selecting, based on the first evaluation, the updated distribution or the initial distribution for assignment of the different group identifiers to each of the respective data elements.

4. The computer-program product of claim 3 , wherein the instructions are operable to cause a computing device to:
display, on a display device, a graphical user interface for user entry indicating a number of changes of group identifiers;
receive, from a user of the graphical user interface, via one or more input devices, user input indicating the number of changes of group identifiers;
wherein determining an updated distribution comprises determining, based on the user input, a set of updated distributions comprising the updated distribution;
wherein evaluating the updated distribution comprises evaluating each of the set of updated distributions; and
selecting the updated distribution comprises selecting the updated distribution out of the set of updated distributions.

5. The computer-program product of claim 3 ,
wherein the computing device is configured to receive a selection of a criterion for evaluating the updated distribution; and
wherein the selection comprises one of an Alias-efficiency, a G-efficiency, an A-efficiency, or an I-efficiency.

6. The computer-program product of claim 3 ,
wherein X is a model matrix for modeling the data according to a given distribution of a validation of the one or more validations;
wherein the determining the initial score comprises computing |X′X| according to the initial distribution;
wherein the determining the updated score comprises computing |X′X| according to the updated distribution; and
selecting, based on the first evaluation, the updated distribution comprises selecting based on an indication of a D-efficiency for the updated distribution.

7. The computer-program product of claim 3 ,
wherein the initial distribution comprises an assignment of the second group identifier of the different group identifiers to a second data element for a second test case of the initial distribution; and
wherein the determining the updated distribution comprises changing the assignment of the first group identifier to an assignment of the second group identifier by switching the assignments of the first data element and the second data element.

8. The computer-program product of claim 1 , wherein the instructions are operable to cause a computing device to:
receive the request by receiving a user request requesting an evaluation of a model using K-fold cross-validation; and
obtain different group identifiers by generating K group identifiers for each of K-folds of the K-fold cross-validation.

9. The computer-program product of claim 1 , wherein the data comprises:
inputs for a continuous factor that represents values within a range of continuous values for the continuous factor; and
inputs for a categorical factor that represents values within a range of discrete options for the categorical factor.

10. The computer-program product of claim 1 , wherein the instructions are operable to cause a computing device to output the indication of the one or more generated data sets by:
for each of the one or more validations:
associating a first group identifier of the different group identifiers with a test group that comprises all the test cases assigned the first group identifier;
associating at least a second group identifier of the different group identifiers with a training group that comprises all the test cases assigned the second group identifier;
generating a first test model based on the training group associated with the second group identifier; and
generating a first test evaluation of the first test model at predicting the responses for each of respective test cases of the test group;

generating, based on all test evaluations generated for each of the one or more validations, a model evaluation of the data for generating the model to predict responses based on the plurality of factors; and
outputting an assessment of the model eva

Assignees

• Sas Inst Inc

Inventors

• Lekivetz Ryan Adam
• Morgan Joseph Albert
• Jones Bradley Allen
• Wolfinger Russell Dean

Classifications

• G06N3/084 Primary

  Backpropagation, e.g. using gradient descent · CPC title

• G06V10/82

  using neural networks · CPC title

• G06V10/776

  Validation; Performance evaluation · CPC title

• G06V10/764

  using classification, e.g. of video objects · CPC title

• G06F11/3684 Primary

  for test design, e.g. generating new test cases · CPC title