Selecting forecasting models by machine learning based on analysis of model robustness

Patent metadata

Field	Value
Publication number	US-2022012609-A1
Application number	US-202016926740-A
Country	US
Kind code	A1
Filing date	Jul 12, 2020
Priority date	Jul 12, 2020
Publication date	Jan 13, 2022
Grant date	—

Abstract

A computer-implemented method, a computer program product, and a computer system for selecting predictions by models. A computer receives a request for a forecast of a dependent variable in a time domain, where the time domain includes first time periods that have normal labels due to normal predictor variable data and second time periods that have anomalous labels due to anomalous predictor variable data. The computer retrieves accuracy scores and robustness scores of models, where the accuracy scores indicate forecasting accuracy in the first time periods and the robustness scores indicate forecasting accuracy in the second time periods. For predictions in the first time period, the computer selects dependent variable values predicted by a first model that has highest values of the accuracy scores. For predictions in the second time periods, the computer selects dependent variable values predicted by a second model that has highest values of the robustness scores.

First claim

What is claimed is:

1 . A computer-implemented method for selecting predictions made by models with normal and anomalous predictor variable data, the method comprising:
receiving a request for a forecast of a dependent variable in a time domain, the time domain including first respective time periods that have normal labels due to normal predictor variable data and second respective time periods that have anomalous labels due to anomalous predictor variable data; retrieving accuracy scores and robustness scores of models, the accuracy scores indicating forecasting accuracy in the first respective time periods and the robustness scores indicating forecasting accuracy in the second respective time periods; selecting dependent variable values predicted by a first model that has highest values of the accuracy scores, for predictions in the first respective time periods; and selecting dependent variable values predicted by a second model that has highest values of the robustness scores, for predictions in the second respective time periods.

2 . The computer-implemented method of claim 1 , further comprising:
composing the forecast in the time domain, using the dependent variable values predicted by the first model in the first respective time periods and the dependent variable values predicted by the second model in the second respective time periods.

3 . The computer-implemented method of claim 1 , further comprising:
retrieving the normal labels for the first respective time periods and the anomalous labels for the second respective time periods; and retrieving dependent variable values predicted by models.

4 . The computer-implemented method of claim 1 , determining the normal labels and the anomalous labels comprises:
retrieving and analyzing predictor variable data; determining whether the predictor variable data in a time period meets one or more expected characteristics of a predictor variable; in response to determining that the predictor variable data meets the one or more expected characteristics, attaching a normal label to the time period; in response to determining that the predictor variable data does not meet the one or more expected characteristics, attaching an anomalous label to the time period; determining the normal labels for the first respective time periods and the anomalous labels for the second respective time periods; and storing the normal labels and the anomalous labels.

5 . The computer-implemented method of claim 4 , determining the accuracy scores and the robustness scores comprises:
loading the models; injecting the predictor variable data to the models; calculating predicted values of the dependent variable in the first respective time periods and the second respective time periods, using the models; and storing the predicted values.

6 . The computer-implemented method of claim 5 , further comprising:
retrieving the predicted values of the dependent variable, the normal labels, and the anomalous labels; retrieving observed values of the dependent variable; calculating the accuracy scores of the models, using the predicted values and the observed values in the first respective time periods; calculating the robustness scores of the models, using the predicted values and the observed values in the second respective time periods; and storing the accuracy scores and the robustness scores.

7 . A computer program product for selecting predictions made by models with normal and anomalous predictor variable data, the computer program product comprising a computer readable storage medium having program instructions embodied therewith, the program instructions executable by one or more processors, the program instructions executable to:
receive a request for a forecast of a dependent variable in a time domain, the time domain including first respective time periods that have normal labels due to normal predictor variable data and second respective time periods that have anomalous labels due to anomalous predictor variable data; retrieve accuracy scores and robustness scores of models, the accuracy scores indicating forecasting accuracy in the first respective time periods and the robustness scores indicating forecasting accuracy in the second respective time periods; select dependent variable values predicted by a first model that has highest values of the accuracy scores, for predictions in the first respective time periods; and select dependent variable values predicted by a second model that has highest values of the robustness scores, for predictions in the second respective time periods.

8 . The computer program product of claim 7 , further comprising the program instructions executable to:
compose the forecast in the time domain, using the dependent variable values predicted by the first model in the first respective time periods and the dependent variable values predicted by the second model in the second respective time periods.

9 . The computer program product of claim 7 , further comprising the program instructions executable to:
retrieve the normal labels for the first respective time periods and the anomalous labels for the second respective time periods; and retrieve dependent variable values predicted by models.

10 . The computer program product of claim 7 , for determining the normal labels and the anomalous labels, further comprising the program instructions executable to:
retrieve and analyzing predictor variable data; determine whether the predictor variable data in a time period meets one or more expected characteristics of a predictor variable; in response to determining that the predictor variable data meets the one or more expected characteristics, attach a normal label to the time period; in response to determining that the predictor variable data does not meet the one or more expected characteristics, attach an anomalous label to the time period; determine the normal labels for the first respective time periods and the anomalous labels for the second respective time periods; and store the normal labels and the anomalous labels.

11 . The computer program product of claim 10 , for determining the accuracy scores and the robustness scores, further comprising the program instructions executable to:
load the models; inject the predictor variable data to the models; calculate predicted values of the dependent variable in the first respective time periods and the second respective time periods, using the models; and store the predicted values.

12 . The computer program product of claim 11 , for determining the accuracy scores and the robustness scores, further comprising the program instructions executable to:
retrieve the predicted values of the dependent variable, the normal labels, and the anomalous labels; retrieve observed values of the dependent variable; calculate the accuracy scores of the models, using the predicted values and the observed values in the first respective time periods; calculate the robustness scores of the models, using the predicted values and the observed values in the second respective time periods; and store the accuracy scores and the robustness scores.

13 . A computer system for selecting predictions made by models with normal and anomalous predictor variable data, the computer system comprising one or more processors, one or more computer readable tangible storage devices, and program instructions stored on at least one of the one or more computer readable tangible storage devices for execution by at least one of the one or more processors, the program instructions executable to:
receive a request fo

Assignees

• IBM

Inventors

• Gormally Robert
• Eck Bradley
• Fusco Francesco
• Purcell Mark
• Tirupathi Seshu

Classifications

• G06Q10/04

  Forecasting or optimisation specially adapted for administrative or management purposes, e.g. linear programming or "cutting stock problem" (market predictions or forecasting for commercial activities G06Q30/0202) · CPC title

• G06N20/00 Primary

  Machine learning · CPC title

• G06N5/04 Primary

  Inference or reasoning models · CPC title