Systems for second-order predictive data analytics, and related methods and apparatus

Patent metadata

Field	Value
Publication number	US-11922329-B2
Application number	US-201916722831-A
Country	US
Kind code	B2
Filing date	Dec 20, 2019
Priority date	May 23, 2014
Publication date	Mar 5, 2024
Grant date	Mar 5, 2024

Abstract

A predictive modeling method may include obtaining a fitted, first-order predictive model configured to predict values of output variables based on values of first input variables; and performing a second-order modeling procedure on the fitted, first-order model, which may include: generating input data including observations including observed values of second input variables and predicted values of the output variables; generating training data and testing data from the input data; generating a fitted second-order model of the fitted first-order model by fitting a second-order model to the training data; and testing the fitted, second-order model of the first-order model on the testing data. Each observation of the input data may be generated by (1) obtaining observed values of the second input variables, and (2) applying the first-order predictive model to corresponding observed values of the first input variables to generate the predicted values of the output variables.

First claim

What is claimed is:

1. A predictive modeling method comprising:
obtaining a fitted, first-order predictive model, wherein the first-order predictive model is configured to predict values of one or more output variables of a prediction problem based on values of one or more first input variables; and
performing a second-order predictive modeling procedure on the fitted, first-order model, wherein the second-order modeling procedure is associated with a second-order predictive model, and wherein performing the second-order predictive modeling procedure on the fitted, first-order model includes:
generating second-order input data including a plurality of second-order observations, wherein each second-order observation includes respective observed values of one or more second input variables and predicted values of the output variables, and wherein generating the second-order input data comprises, for each second-order observation: obtaining the respective observed values of the second input variables and corresponding observed values of the first input variables, and applying the first-order predictive model to the corresponding observed values of the first input variables to generate the respective predicted values of the output variables,
generating, from the second-order input data, second-order training data and second-order testing data,
generating a fitted second-order predictive model of the fitted first-order model by fitting the second-order predictive model to the second-order training data, and
testing the fitted, second-order predictive model of the fitted first-order model on the second-order testing data.

2. The method of claim 1 , wherein obtaining the fitted, first-order model comprises blending two fitted predictive models.

3. The method of claim 1 , wherein the second-order predictive model is a RuleFit model, a generalized additive model, or a blend thereof.

4. The method of claim 1 , wherein performing the second-order predictive modeling procedure further includes performing cross-validation or nested cross-validation of the second-order predictive model.

5. The method of claim 1 , further comprising:
determining an accuracy score of each of the fitted predictive models, wherein the accuracy score of each fitted model represents an accuracy with which the fitted model predicts outcomes of one or more prediction problems.

6. The method of claim 5 , wherein the accuracy score of the fitted second-order model exceeds the accuracy score of the fitted first-order model.

7. The method of claim 1 , further comprising: deploying the fitted second-order model, wherein deploying the fitted second-order model comprises generating a plurality of predictions by applying the fitted second-order model to other data representing instances of the prediction problem, wherein the second-order input data do not include the other data.

8. The method of claim 7 , wherein the fitted second-order model comprises a set of one or more conditional rules, and wherein the set of one or more conditional rules comprises a set of one or more machine executable if-then statements.

9. The method of claim 7 , wherein the second-order input data are first second-order input data, and wherein deploying the fitted second-order model further comprises refreshing the fitted second-order model based, at least in part, on second second-order input data.

10. The method of claim 9 , wherein the fitted second-order model is a first fitted second-order model, and wherein refreshing the fitted second-order model based, at least in part, on the second second-order input data comprises:
generating, from the second second-order input data, second second-order training data and second second-order testing data;
generating a second fitted second-order model of the fitted first-order model by fitting the second-order predictive model to the second second-order training data;
testing the second fitted second-order model of the first-order model on the second second-order testing data; and
blending the first fitted second-order model and the second fitted second-order model to generate a refreshed second-order predictive model.

11. The method of claim 9 , wherein the fitted second-order model is a first fitted second-order model, and wherein refreshing the fitted second-order model based, at least in part, on the second second-order input data comprises:
generating third second-order input data comprising at least a portion of the first second-order input data and at least a portion of the second second-order input data;
generating, from the third second-order input data, third second-order training data and third second-order testing data;
generating a second fitted second-order model of the fitted first-order model by fitting the second-order predictive model to the third second-order training data; and
testing the second fitted second-order model of the first-order model on the third second-order testing data.

12. The method of claim 1 , wherein the second-order modeling procedure is one of a plurality of second-order modeling procedures, wherein the second-order predictive model is one of a plurality of second-predictive models, and wherein the method comprises performing the plurality of second-order modeling procedures on the fitted first-order model, thereby generating a plurality of fitted second-order models of the fitted first-order model.

13. The method of claim 12 , further comprising:
determining an accuracy score of each of the fitted second-order predictive models, wherein the accuracy score of each fitted second-order model represents an accuracy with which the fitted second-order model predicts outcomes of one or more prediction problems;
determining which of the accuracy scores is highest; and
deploying the fitted second-order model with the highest accuracy score.

14. A predictive modeling apparatus comprising:
a memory configured to store a machine-executable module encoding a second-order predictive modeling procedure associated with a second-order predictive model, wherein the second-order predictive modeling procedure includes a plurality of tasks including at least one pre-processing task and at least one model-fitting task; and
at least one processor configured to execute the machine-executable module, wherein executing the machine-executable module causes the apparatus to perform the second-order predictive modeling procedure on a fitted, first-order predictive model, including:
performing the pre-processing task, including obtaining the fitted, first-order predictive model, wherein the first-order predictive model is configured to predict values of one or more output variables of a prediction problem based on values of one or more first input variables; and
performing the model-fitting task, including:
generating second-order input data including a plurality of second-order observations, wherein each second-order observation includes respective observed values of one or more second input variables and predicted values of the output variables, and wherein generating the second-order input data comprises, for each second-order observation: obtaining the respective observed values of the second input variables and corresponding observed values of the first input variables, and applying the first-order predictive model to the corresponding observed values of the first input variables to generate the respective predicted values of the output variables,
generating, from the second-order input data, second-order training data and second-order testing data,
generating a fitted second-order predictive model of the fitted first-order m

Assignees

• Datarobot Inc

Inventors

• Achin Jeremy
• Degodoy Thomas
• Owen Timothy
• Conort Xavier
• Yurgenson Sergey
• Steadman Mark L
• Koundry Glen
• Chua Hon Nian

Classifications

• G06N5/04 Primary

  Inference or reasoning models · CPC title

• G06F9/5011

  the resources being hardware resources other than CPUs, Servers and Terminals · CPC title

• G06N5/02

  Knowledge representation; Symbolic representation · CPC title

• G06N20/00 Primary

  Machine learning · CPC title

• G06Q10/04

  Forecasting or optimisation specially adapted for administrative or management purposes, e.g. linear programming or "cutting stock problem" (market predictions or forecasting for commercial activities G06Q30/0202) · CPC title