Method and system for validating ensemble demand forecasts

Patent metadata

Field	Value
Publication number	US-11170391-B2
Application number	US-201816172603-A
Country	US
Kind code	B2
Filing date	Oct 26, 2018
Priority date	Oct 26, 2018
Publication date	Nov 9, 2021
Grant date	Nov 9, 2021

Abstract

Methods and systems for forecasting demand for a plurality of items are provided. In particular, the demand forecasting system and methods described herein are useful for predicting demand of products in a retail context. Forecast models are built and used to score incoming sales data to predict future demand for items. Forecast models are validated by evaluating actual demand against predicted demand and using that information to inform how future ensemble forecast will be generated. Forecasts may be broken down into smaller components to satisfy a variety of requests for data from client applications.

First claim

The invention claimed is:

1. A system for assessing performance of demand forecasting models, the system comprising:
an enterprise forecast engine configured to generate, using a plurality of forecasting models, a plurality of demand forecasts for each of a plurality of items sold within a retail enterprise over a period of time based on at least sales data;
a demand forecast data store configured to store the plurality of demand forecasts; and
a forecast validation engine configured to:
generate and provide for display a validation user interface;
receive, as input from the validation user interface at a command line tool, selections of at least a first validation set and a second validation set of a plurality of validation sets and configuration options for visualizing the first and second validation sets, wherein:
the first validation set includes a first data set of forecasted values associated with a first demand forecast from the plurality of demand forecasts generated using a first forecasting model from the plurality of forecasting models based on the sales data,
the second validation set includes a second data set of forecasted values associated with a second demand forecast from the plurality of demand forecasts generated using a second forecasting model from the plurality of forecasting models based on the sales data, and
the configuration options include selections of a validation metric type, a subset of metric values recorded for the first and second validation sets, and a visualization type for visualizing the first and second validation sets;

send, by the command line tool, a submission packet to a validation server, the submission packet comprising the selections of the first and second validation sets and the configuration options for visualizing the first and second validation sets;
query, by the validation server, the demand forecast data store for the first data set of forecasted values of the first validation set and the second data set of forecasted values of the second validation set;
calibrate the first forecasting model and the second forecasting model with historical training data from a test data repository;
test each of the calibrated first forecasting model and the calibrated second forecasting model at the validation server by calculating predictions for each of a plurality of sets of forecast coordinates within the respective first and second data set of forecasted values of the first and second validation sets based on at least the selections of the validation metric type and the subset of metric values;
based on the predictions, calculate first forecast validation results for the first forecasting model and second forecast validation results for the second forecasting model at the validation server; and
provide, for concurrent display on the validation user interface, a first visualization of the first forecast validation results for the first forecasting model and a second visualization of the second forecast validation results for the second forecasting model to enable direct comparison of forecast performance for the first forecasting model and the second forecasting model based on the predictions, the first visualization and the second visualization corresponding to the visualization type.

2. The system of claim 1 , wherein the visualization type comprises one or more of box-plots, Q-Q plots, and histograms.

3. The system of claim 1 , wherein the command line tool is executable on a computing system communicatively connected to the validation server.

4. The system of claim 1 , wherein the predictions are stored in the test data repository that comprises a test directory.

5. The system of claim 1 , wherein the first forecast validation results and the second forecast validation results are stored in a validation data repository that comprises a validation database.

6. The system of claim 1 , wherein each of the plurality of demand forecasts is generated as an aggregate demand forecast for sales of an individual item from the plurality of items across all locations of the retail enterprise over the period of time.

7. The system of claim 6 , wherein the aggregate demand forecast is converted into forecast distributions.

8. The system of claim 1 , wherein the enterprise forecast engine is further configured to:
receive a client request, the client request including parameters for a demand forecast, the parameters including at least one of an item from the plurality of items, a location of the retail enterprise, and a particular time period within the period of time; and
based on the parameters, build an ensemble model, the ensemble model comprising a weighted combination of two or more forecasting models from the plurality of forecasting models, wherein each of the two or more forecasting models and an associated weighting of each of the two or more forecasting models are selected based on calculated forecast validation results for the two or more forecasting models that are representative of an ability of the two or more forecasting models to predict demand for the at least one of the item, the location, and the particular time period included as the parameters of the client request.

9. The system of claim 8 , wherein the two or more forecasting models include the first forecasting model and the second forecasting model selected and weighted based on the respective first forecast validation results and the second forecast validation results.

10. A method of validating and visualizing demand forecast models, the method comprising:
generating, using a plurality of demand forecasting models, a plurality of demand forecasts for each of a plurality of items sold within a retail enterprise over a period of time based on at least sales data;
storing the plurality of demand forecasts in a demand forecast data store;
generating and providing for display a validation user interface;
receiving, as input from the validation user interface, selections of at least a first validation set and a second validation set of a plurality of validation sets and configuration options for visualizing the first and second validation sets, wherein:
the first validation set comprises a first data set of forecasted values associated with a first demand forecast from the plurality of demand forecasts generated using a first demand forecasting model of the plurality of demand forecasting models based on the sales data,
the second validation set comprises a second data set of forecasted values associated with a second demand forecast from the plurality of demand forecasts generated using a second demand forecasting model of the plurality of demand forecasting models based on the sales data, and
the configuration options include selections of a validation metric type, a subset of metric values recorded for the first and second validation sets, and a visualization type for visualizing the first and second validation sets; and

querying the demand forecast data store to retrieve the first data set of forecasted values of the first validation set and the second data set of forecasted values of the second validation set;
calibrating the first demand forecasting model and the second demand forecasting model with historical training data;
testing each of the calibrated first demand forecasting model and the calibrated second demand forecasting model by calculating predictions for each of a plurality of sets of forecast coordinates within the respective first and second data set of forecasted values of the first and second validation sets based on at least the selections of the validation metric type and the subset of metric values;
based on the predictions, calculating first forecast validation resu

Assignees

• Target Brands Inc

Inventors

• Morgan Adam
• Das Anand
• Shah Samir H
• Yelland Phillip

Classifications

• G06N3/044

  Recurrent networks, e.g. Hopfield networks · CPC title

• G06N3/09

  Supervised learning · CPC title

• G06N3/0442

  characterised by memory or gating, e.g. long short-term memory [LSTM] or gated recurrent units [GRU] · CPC title

• G06N20/20

  Ensemble learning · CPC title

• G06Q30/0202 Primary

  Market predictions or forecasting for commercial activities · CPC title