Prediction model testing framework

Patent metadata

Field	Value
Publication number	US-10372572-B1
Application number	US-201615365852-A
Country	US
Kind code	B1
Filing date	Nov 30, 2016
Priority date	Nov 30, 2016
Publication date	Aug 6, 2019
Grant date	Aug 6, 2019

Abstract

A prediction model testing system includes a test environment that is used to test a prediction model under test (PMUT). A metrics collector in a production environment collects and stores production metrics data generated from computing resources in a production environment. A production predictor in the production environment generates production predictions for the metrics, using a production prediction model. A test manager may make the production metrics data available in a test environment. Test predictions are generated in the test environment from the metrics data using the PMUT. The test manager may then calculate respective prediction errors of the production prediction model and the PMUT, and generate a report indicating the differences between the two sets of prediction errors. The report may be used by the test management system to determine whether a test of the PMUT was successful.

First claim

What is claimed is:

1. A system comprising:
a metrics collector implemented by one or more computing devices comprising one or more hardware processors and memory configured to:
capture data for one or more metrics generated from one or more computing resources in a production environment of a service provider network; and
store the captured data in the production environment;

a production predictor implemented by one or more computing devices comprising one or more hardware processors and memory configured to:
generate, in the production environment, production predictions for the one or more metrics based at least in part on a production prediction model and the captured data; and

a test manager implemented by one or more computing devices comprising one or more hardware processors and memory configured to:
provide the captured data in a test environment;
generate, in the test environment, test predictions for at least a subset of the one or more metrics based at least in part on the prediction model under test (PMUT) and the captured data; and
obtain observed data for the one or more metrics from the production environment corresponding to the test predictions;
determine test prediction errors between the test predictions and observed data for the one or more metrics from the production environment corresponding to the test predictions;
determine production prediction errors between the production predictions and the observed data; and
generate a report indicating differences between the test prediction errors and the production prediction errors.

2. The system of claim 1 , wherein:
at least one computing resource of the one or more computing resources in the service provider network is a virtual machine instance hosted by one or more virtual machine hosts, and
at least one metric of the one or more metrics is generated from operation of the virtual machines instance.

3. The system of claim 1 , wherein to generate the report, the test manager is configured to generate in the report differences in one or more respective numerical values of the test predictions and the production predictions.

4. The system of claim 1 , wherein the test manager is configured to:
transmit information from the report to a client device via a graphical user interface (GUI); and
receive input via the GUI indicating whether a test of the PMUT was successful or unsuccessful.

5. The system of claim 4 , wherein the test manager is configured to:
generate a determination of whether the test of the PMUT was successful or unsuccessful based at least in part on information from the report; and
transmit a recommendation to the client device via the GUI recommending whether or not to promote the PMUT from a testing stage managed by the test manager.

6. A method, comprising:
providing captured data in a test environment, the captured data comprising data for one or more metrics generated from operation of one or more computing resources in a production environment;
generating, in the test environment, one or more test predictions for at least a subset of the one or more metrics based at least in part on a prediction model under test (PMUT) and the captured data; and
obtaining observed data for one or more metrics from the production environment corresponding to the test predictions;
determining test prediction errors between the test predictions and the observed data;
determining production prediction errors between production predictions and the observed data, the production predictions generated in a production environment based at least in part on a production prediction model and the captured data; and
generating a report indicating differences between the test prediction errors and the production prediction errors.

7. The method of claim 6 , further comprising:
storing the captured data in a production metrics data store; and
wherein providing the captured data in the test environment comprises copying the captured data from the production metrics data store to a metrics data store (PMUT data store).

8. The method of claim 7 , wherein copying the captured data from the production metrics data store to the PMUT data store comprises reformatting the captured data.

9. The method of claim 6 , wherein generating the report comprises generating in the report differences in one or more respective numerical values of the test predictions and the production predictions.

10. The method of claim 6 , further comprising:
determining, based at least in part on the report, that the differences between the test prediction errors and the production prediction errors are within a tolerance; and
indicating to a test management system that a test of the PMUT was successful.

11. The method of claim 10 , further comprising selecting a value for the tolerance based at least in part on a type of a metric in the report.

12. The method of claim 6 , further comprising:
during a testing stage:
modifying or replacing the PMUT to create a new PMUT;
generating a new report using the new PMUT;
determining, based at least in part on the new report, whether the new PMUT is successfully tested; and

promoting the new PMUT from the testing stage based at least in part on the determination that the new PMUT is successfully tested.

13. The method of claim 6 , further comprising:
transmitting information from the report to a client device via a graphical user interface (GUI); and
receiving input via the GUI indicating whether a test of the PMUT was successful or unsuccessful.

14. The method of claim 6 , wherein generating the one or more test predictions comprises:
selecting one or more subsets of metrics from the one or more metrics; and
generating test predictions for the one or more subsets of metrics; and
wherein generating the report comprises:
generating one or more sub-reports indicating differences between test prediction errors and production prediction errors for respective ones of the one or more subsets of metrics;
determining that an aggregate set of metrics included in the one or more sub-reports for the PMUT satisfies a test coverage policy; and
aggregating the one or more sub-reports to produce the report.

15. The method of claim 14 , wherein test predictions for at least two of the one or more subsets of metrics are generated at least partly in parallel.

16. A non-transitory computer-accessible storage medium storing program instructions that when executed on one or more processors cause the one or more processors to:
providing captured data from a production data store, the captured data comprising data for one or more metrics generated from operation of one or more computing resources in a production environment;
generate, in a test environment, one or more test predictions for at least a subset of the one or more metrics based at least in part on a prediction model under test (PMUT) and the captured data; and
obtaining observed data for the one or more metrics from the production environment corresponding to the test predictions;
determining test prediction errors between the test predictions and the observed data;
determine production prediction errors between production predictions and the observed data, the production predictions generated in the production environment based at least in part on a production prediction model and the captured data; and
generating a report indicating differences between the test predictions and the production predictions.

17. The non-transitory computer-accessible storage medium of claim 16 , wherein to provide

Assignees

• Amazon Tech Inc

Inventors

• Kesarwani Nitin
• Yang Sharon Xuelun
• Huang Wei

Classifications

• G06N7/01

  Probabilistic graphical models, e.g. probabilistic networks · CPC title

• G06F11/261 Primary

  by simulating additional hardware, e.g. fault simulation · CPC title

• G06N7/005

  Physics · mapped topic