System for automatic error estimate correction for a machine learning model

Patent metadata

Field	Value
Publication number	US-12430401-B2
Application number	US-202117467638-A
Country	US
Kind code	B2
Filing date	Sep 7, 2021
Priority date	Sep 8, 2020
Publication date	Sep 30, 2025
Grant date	Sep 30, 2025

Abstract

A computing device computes curve descriptive values to correct an error estimate of a prediction. A predefined number of times, an input dataset is split into a training dataset and a validation dataset, a predictive model and a domain model are trained, the trained predictive model and the trained domain model are validated, a predictive error value, a residual value, and a domain error value are computed, and each value is stored in output data. A domain threshold value is computed from the stored domain error values. Each predictive error value and each residual value stored in the output data is stored in in-domain output data when a respective domain error value is less than or equal to the computed domain threshold value. Curve descriptive values are computed to describe a relationship between the residual values as a function of the prediction error values stored in the in-domain output data.

First claim

What is claimed is:

1. A non-transitory computer-readable medium having stored thereon computer-readable instructions that when executed by a computing device cause the computing device to:
(A) split an input dataset received from a distributed computer system into a training dataset and a validation dataset, wherein the input dataset includes a plurality of observation vectors;
(B) define a predictive model description and a domain model description based on the input dataset;
(C) train a predictive model associated with the predictive model description with the training dataset subsequent to defining the predictive model description;
(D) train a domain model associated with the domain model description with the training dataset subsequent to defining the domain model description;
(E) validate the trained predictive model with the validation dataset;
(F) validate the trained domain model with the validation dataset;
(G) compute a predictive error value and a residual value from the validated predictive model for each observation vector of the plurality of observation vectors included in the validation dataset;
(H) compute a domain error value from the validated domain model for each observation vector of the plurality of observation vectors included in the validation dataset;
(I) store the computed predictive error value, the computed residual value, and the computed domain error value in output data for each observation vector of the plurality of observation vectors included in the validation dataset;
(J) repeat (A) through (I) a predefined number of times;
(K) compute a domain threshold value using the stored domain error values;
(L) store each predictive error value and each residual value stored in the output data in in-domain output data when a respective stored domain error value is less than or equal to the computed domain threshold value;
(M) compute curve descriptive values based on a type of curve, wherein the curve describes a relationship between the residual values stored in the in-domain output data as a function of the prediction error values stored in the in-domain output data;
(N) output the curve descriptive values to correct an error estimate value of a predicted value of a new observation vector; and
(O) provide an indication of the error estimate value of the predicted value of the new observation vector to the distributed computing system.

2. The non-transitory computer-readable medium of claim 1 , wherein after (L) and before (M), the computer-readable instructions further cause the computing device to:
compute a standard deviation value from a total predictive error value and each predictive error values stored in the in-domain output data;
divide each predictive error value stored in the in-domain output data by the computed standard deviation value to compute a scaled, predictive error value, wherein the predictive error value in (M) is the scaled, predictive error value; and
divide each residual value stored in the in-domain output data by the computed standard deviation value to compute a scaled, residual value, wherein the residual value in (M) is the scaled, residual value.

3. The non-transitory computer-readable medium of claim 2 , wherein the computer-readable instructions further cause the computing device to:
(P) train a second predictive model using the plurality of observation vectors included in the input dataset;
(Q) train a second domain model using the plurality of observation vectors included in the input dataset;
output a first description of the trained second predictive model; and
output a second description of the trained second domain model.

4. The non-transitory computer-readable medium of claim 3 , wherein an average domain threshold value is computed from each domain threshold value computed in (J).

5. The non-transitory computer-readable medium of claim 4 , wherein after (P), the computer-readable instructions further cause the computing device to:
compute a predicted domain error value for a new observation vector using the trained second domain model, wherein the new observation vector is not included in the input dataset; and
when the computed predicted domain error value is less than the computed domain threshold value,
compute a predicted value of the new observation vector and an error estimate value associated with the computed predicted value using the trained second predictive model; and
correct the error estimate value of the computed predicted value using the computed curve descriptive values; and
output the computed predicted value and the corrected error estimate value of the new observation vector.

6. The non-transitory computer-readable medium of claim 5 , wherein, when the computed predicted domain error value is greater than the computed domain threshold value, the computer-readable instructions further cause the computing device to output an out of domain indicator for the new observation vector.

7. The non-transitory computer-readable medium of claim 1 , wherein the computer-readable instructions further cause the computing device to:
(P) train a second predictive model using the plurality of observation vectors included in the input dataset; and
(Q) output a description of the trained second predictive model.

8. The non-transitory computer-readable medium of claim 7 , wherein
after (P), the computer-readable instructions further cause the computing device to:
compute a predicted value of a new observation vector and an error estimate value associated with the computed predicted value using the trained second predictive model, wherein the new observation vector is not included in the input dataset;
correct the error estimate value of the computed predicted value using the computed curve descriptive values; and
output the computed predicted value and the corrected error estimate value of the new observation vector.

9. The non-transitory computer-readable medium of claim 1 , wherein the computer-readable instructions further cause the computing device to:
(P) train a second predictive model using the plurality of observation vectors included in the input dataset;
(Q) train a second domain model using the plurality of observation vectors included in the input dataset;
output a first description of the trained second predictive model; and
output a second description of the trained second domain model.

10. The non-transitory computer-readable medium of claim 9 , wherein an average domain threshold value is computed from each domain threshold value computed in (J).

11. The non-transitory computer-readable medium of claim 10 , wherein after (P), the computer-readable instructions further cause the computing device to:
compute a predicted domain error value for a new observation vector using the trained second domain model, wherein the new observation vector is not included in the input dataset; and
when the computed predicted domain error value is less than the computed domain threshold value,
compute a predicted value of the new observation vector and an error estimate value associated with the computed predicted value using the trained second predictive model; and
correct the error estimate value of the computed predicted value using the computed curve descriptive values; and
output the computed predicted value and the corrected error estimate value of the new observation vector.

12. The non-transitory computer-readable medium of claim 11 , wherein, when the computed predicted domain error value is greater than the computed domain threshold value, the computer-readable instructions further cause the computing device to output an out of domain indicator f

Assignees

• Wisconsin Alumni Res Found

Inventors

• Morgan Dane
• Jacobs Ryan Matthew
• Palmer Glenn

Classifications

• G06N7/01

  Probabilistic graphical models, e.g. probabilistic networks · CPC title

• G06F18/217

  Validation; Performance evaluation; Active pattern learning techniques · CPC title

• G06N3/09

  Supervised learning · CPC title

• G06N5/01

  Dynamic search techniques; Heuristics; Dynamic trees; Branch-and-bound · CPC title

• G06N20/20

  Ensemble learning · CPC title