Automated reinforcement-learning-based application manager that learns and improves a reward function

Patent metadata

Field	Value
Publication number	US-10963313-B2
Application number	US-201916518763-A
Country	US
Kind code	B2
Filing date	Jul 22, 2019
Priority date	Aug 27, 2018
Publication date	Mar 30, 2021
Grant date	Mar 30, 2021

Abstract

The current document is directed to automated reinforcement-learning-based application managers that learn and improve the reward function that steers reinforcement-learning-based systems towards optimal or near-optimal policies. Initially, when the automated reinforcement-learning-based application manager is first installed and launched, the automated reinforcement-learning-based application manager may rely on human-application-manager action inputs and resulting state/action trajectories to accumulate sufficient information to generate an initial reward function. During subsequent operation, when it is determined that the automated reinforcement-learning-based application manager is no longer following a policy consistent with the type of management desired by human application managers, the automated reinforcement-learning-based application manager may use accumulated trajectories to improve the reward function.

First claim

The invention claimed is:

1. An automated reinforcement-learning-based application manager that manages a computing environment that includes one or more applications and one or more of a distributed computing system having multiple computer systems interconnected by one or more networks, a standalone computer system, and a processor-controlled user device, the reinforcement-learning based application manager comprising:
one or more processors, one or more memories, and one or more communications subsystems;
a set of actions that can be issued to the computing environment; and
an iterative control process that repeatedly
selects and issues a next action to the computing environment according to a control policy that uses a state vector that represents a current state of the computing system,
receives, from the computing environment in response to execution of the issued next action,
a reward, generated by a reward function, which the control process uses to attempt to learn an optimal or near-optimal control policy, and
one of an observation and state vector, which the control process subsequently uses as the current state, and

when the reward function is determined to differ by more than a threshold difference from a reward function corresponding to the desired operational behavior of the computing environment, generates a new reward function from stored state/action trajectories.

2. The automated reinforcement-learning-based application manager of claim 1 wherein the reward function is a parameterized function of parameters that receives a state vector and returns a real value.

3. The automated reinforcement-learning-based application manager of claim 1 wherein the reward function is a linear combination of terms, each term comprising the product of a parameters and a corresponding function that computes a value based on one or more metrics or configuration parameters in the state vector.

4. The automated reinforcement-learning-based application manager of claim 3 wherein a new reward function is generated by one of:
an iterative process that, for each stored state/action trajectory, generates a number of synthetic trajectories, generates a set of estimated reward-function parameters that maximizes a difference between the stored state/action trajectory and synthetic trajectories, and incorporates the set of estimated reward-function parameters into the reward function;
optimization computations that optimize a set of estimated reward-function parameters with respect to the stored state/action trajectories;
a linear-programming reward-function-generation process;
a maximum-entropy reward-function-generation process; and
a neural-network-based reward-function-generation process.

5. The automated reinforcement-learning-based application manager of claim 1 wherein the iterative control process selects a next action, according to a control policy that uses a state vector that represents a current state of the computing system, by:
when an action specified by a human application manager is available, selecting the action specified by a human application manager as the next action; and
when an action specified by a human application manager is not available, selecting, as the next action, an action returned by the control policy.

6. The automated reinforcement-learning-based application manager of claim 5 wherein the iterative control process issues a next action to the computing environment by:
when the next action has been specified by a human application manager, issuing the next action to the computing environment; and
when the next action has been obtained using the control policy, proposing the next action to a human application manager;
waiting for a response;
when a rejection response is received, storing an indication of the rejection and selecting, as the next action, a no-op action;
when a counterproposal response is received, storing an indication of the counterproposal and selecting, as the next action, the counterproposal;
when a time-out occurs or when an accept response is received, storing an acceptance indication; and
issuing the next action to the computing environment.

7. The automated reinforcement-learning-based application manager of claim 6 wherein the next action is proposed to a human application manager through a management interface and wherein the rejection response, counterproposal response, and accept response are received through the management interface.

8. The automated reinforcement-learning-based application manager of claim 1 wherein, when the automated reinforcement-learning-based application manager receives, from the computing environment in response to execution of the issued next action, a reward and one of an observation and state vector, the automated reinforcement-learning-based application manager records a state/action transition in a current trajectory maintained in memory.

9. The automated reinforcement-learning-based application manager of claim 8 wherein, when automated reinforcement-learning-based application manager determines that a stop/start point has been reached, the automated reinforcement-learning-based application manager stores the current trajectory into a trajectory database and initializes a new current trajectory.

10. The automated reinforcement-learning-based application manager of claim 1 wherein the automated reinforcement-learning-based application manager determines the the reward function differs by more than a threshold difference from a reward function corresponding to the desired operational behavior of the computing environment by analyzing indications of human-manager rejection, counterproposal, and acceptance responses to proposed actions.

11. A method for learning and improving a reward function while control a computing environment by an automated reinforcement-learning-based application manager that manages the computing environment that includes one or more applications and one or more of a distributed computing system having multiple computer systems interconnected by one or more networks, a standalone computer system, and a processor-controlled user device, the automated reinforcement-learning-based application manager having one or more processors, one or more memories, one or more communications subsystems, and a set of actions that can be issued to the computing environment, the method comprising:
iteratively
selecting and issuing a next action to the computing environment according to a control policy that uses a state vector that represents a current state of the computing system,
receiving, from the computing environment in response to execution of the issued next action,
a reward, generated by a reward function, which the control process uses to attempt to learn an optimal or near-optimal control policy, and
one of an observation and state vector, which the control process subsequently uses as the current state, and

when the reward function is determined to differ by more than a threshold difference from a reward function corresponding to the desired operational behavior of the computing environment, generating a new reward function from stored state/action trajectories.

12. The method of claim 11 wherein the reward function is a parameterized function of parameters that receives a state vector and returns a real value.

13. The method of claim 12 wherein a new reward function is generated by one of:
an iterative process that, for each stored state/action trajectory, generates a number of synthetic trajectories, generates a set of estimated reward-function parameters that maximizes a difference between the stored state/action t

Assignees

• Vmware Inc

Inventors

• Nag Dev
• Yankov Yanislav
• Wang Dongni
• Burk Gregory T
• Stephen Nicholas Mark Grant

Classifications

• G06N7/01

  Probabilistic graphical models, e.g. probabilistic networks · CPC title

• G06N3/092

  Reinforcement learning · CPC title

• G06N3/02

  Neural networks · CPC title

• G06F9/5077 Primary

  Logical partitioning of resources; Management or configuration of virtualized resources (specific details on emulation or internal functioning of virtual machines G06F9/455) · CPC title

• G06N3/08 Primary

  Learning methods · CPC title