Artificial intelligence system incorporating automatic model switching based on model parameter confidence sets

Patent metadata

Field	Value
Publication number	US-11164093-B1
Application number	US-201816054817-A
Country	US
Kind code	B1
Filing date	Aug 3, 2018
Priority date	Aug 3, 2018
Publication date	Nov 2, 2021
Grant date	Nov 2, 2021

Abstract

Computer systems and associated methods are disclosed to implement a model executor that dynamically selects machine learning models for choosing sequential actions. In embodiments, the model executor executes and updates an active model to choose sequential actions. The model executor periodically initiates a recent model and updates the recent model along with the active model based on recently chosen actions and results of the active model. The model executor periodically compares respective confidence sets of the two models' parameters. If the two confidence sets are sufficiently divergent, a replacement model is selected to replace the active model. In embodiments, the replacement model may be selected from a library of past models based on their similarity with the recent model. In embodiments, past models that exceed a certain age or have not been recently used as the active model are removed from the library.

First claim

What is claimed is:

1. A system comprising:
one or more computers that implement a model executor, configured to;
execute an active model having a first vector of model parameters to, in individual time periods:
select an action for the time period according to the first vector of model parameters, wherein the first vector of model parameters is selected from a first confidence set;
receive a result of the action; and
update the first confidence set based at least in part on previously selected actions of the active model and their corresponding results;

initiate a recent model having a second vector of same model parameters as the first vector of the active model and, in individual subsequent time periods:
continue to execute the active model to select a subsequent action, receive a subsequent result, and update the first confidence set; and
update the recent model along with the active model, including a second confidence set of the second vector based at least in part on subsequent actions selected by the active model and corresponding subsequent results in the subsequent time periods; and

responsive to a determination that the first and second confidence sets are sufficiently different, replace the active model with a replacement model, wherein additional actions are selected according to the replacement model.

2. The system of claim 1 , wherein to replace the active model, the model executor is configured to replace the active model with the recent model.

3. The system of claim 1 , wherein the model executor is configured to:
determine a set of past models that are sufficiently similar to the recent model based at least in part on the second confidence set of the recent model and the past models' respective confidence sets of model parameters; and
select one model from the set as the replacement model.

4. The system of claim 1 , further comprising a model repository that stores the set of past models, and wherein the model executor is configured to:
store the active model that was replaced as a past model in the model repository, wherein the past model is not updated;
responsive to a determination that the past model is sufficiently similar to another recent model based at least in part on respective confidence sets of model parameters of the past model and the other recent model, replace a current active model with the past model; and
responsive to a determination that the past model is older than a threshold age or has not been active in a threshold period of time, delete the past model from the model repository.

5. The system of claim 1 , wherein:
to select the action, the model executor is configured to select a content to be delivered to a client; and
to receive the result of the action, the model executor is configured to receive user feedback data from the client for the content.

6. A method comprising:
executing an active model having a first vector of model parameters, wherein the execution includes performing, in individual time periods:
selecting an action for the time period according to the first vector of model parameters, wherein the first vector of model parameters is selected from a first confidence set;
receiving a result of the action; and
updating the first confidence set based at least in part on previously selected actions of the active model and their corresponding results;

initiating a recent model having a second vector of same model parameters as the first vector of active model and in individual subsequent time periods:
continuing to execute the active model to select a subsequent action, receive a subsequent result, and update the first confidence set; and
updating the recent model along with the active model, including a second confidence set of the second vector based at least in part on subsequent actions selected by the active model and corresponding subsequent results in the subsequent time periods; and

responsive to a determination that the first and second confidence sets are sufficiently different, replacing the active model with a replacement model, wherein additional actions are selected according to the replacement model.

7. The method of claim 6 , wherein replacing the active model comprises replacing the active model with the recent model.

8. The method of claim 6 , further comprising:
determining a set of past models that are sufficiently similar to the recent model based at least in part on the second confidence set of the recent model and the past models' respective confidence sets of model parameters; and
selecting one model from the set as the replacement model.

9. The method of claim 8 , wherein the set of past models are retrieved from a model repository, and further comprising:
storing the active model that was replaced as a past model in the model repository, wherein the past model is not updated; and
responsive to a determination that the past model is sufficiently similar to another recent model based at least in part on respective confidence sets of respective model parameters of the stored model and the other recent model, replacing a current active model with the past model.

10. The method of claim 9 , further comprising:
responsive to a determination that the past model is older than a threshold age or has not been active in a threshold period of time, deleting the past model from the model repository.

11. The method of claim 6 , further comprising:
determining a plurality of past models that are sufficiently similar to the recent model based on the second confidence set and the past models' respective confidence sets of model parameters; and
combining the plurality of past models using an averaging technique to generate the replacement model.

12. The method of claim 6 , wherein:
selecting the action comprises selecting a content to be delivered via a user interface; and
receiving the result of the action comprises receiving user feedback data from the user interface after the content was delivered.

13. The method of claim 6 , wherein updating the recent model comprises:
updating the recent based on a moving window of most recent actions and corresponding results in the subsequent time periods.

14. The method of claim 6 , further comprising:
providing a machine learning service that hosts a plurality of machine learning models for a plurality of respective clients;
receiving, at the machine learning service, one or more configuration parameters that controls the replacing of the active model; and
performing, via the machine learning service, the replacing of the active model according to the one or more parameters.

15. The method of claim 6 , further comprising:
displaying, via a graphical user interface, an animation of the first confidence set of the active model and the second confidence set of the recent model.

16. A non-transitory computer-accessible storage medium storing program instructions that when executed on one or more processors cause the one or more processors to:
execute an active model that has a first vector of model parameters to, in individual time periods:
select an action for the time period according to the first vector of model parameters, wherein the first vector of model parameters is selected from a first confidence set;
receive a result of the action; and
update the first confidence set based at least in part on previously selected actions of the active model and their corresponding results;

initiate a recent model having a second vector of same model parameters as the first vector of active model and in individual subsequent time periods

Assignees

• Amazon Tech Inc

Inventors

• Zappella Giovanni

Classifications

• G06N20/00 Primary

  Machine learning · CPC title

• G06F30/20

  Design optimisation, verification or simulation (optimisation, verification or simulation of circuit designs G06F30/30) · CPC title

• G06N5/045 Primary

  Explanation of inference; Explainable artificial intelligence [XAI]; Interpretable artificial intelligence · CPC title