Automated reinforcement-learning-based application manager that uses local agents

Patent metadata

Field	Value
Publication number	US-10970649-B2
Application number	US-201916518785-A
Country	US
Kind code	B2
Filing date	Jul 22, 2019
Priority date	Aug 27, 2018
Publication date	Apr 6, 2021
Grant date	Apr 6, 2021

Abstract

The current document is directed to automated reinforcement-learning-based application managers that use local agents. Local agents provide finer-granularity monitoring of an application or application subcomponents and provide continued application management in the event of interruption of network traffic between an automated reinforcement-learning-based application manager and the application or application subcomponents managed by the automated reinforcement-learning-based application manager.

First claim

The invention claimed is:

1. An automated reinforcement-learning-based application manager that manages a computing environment that includes one or more applications, one or more local agents, and one or more of a distributed computing system having multiple computer systems interconnected by one or more networks, a standalone computer system, and a processor-controlled user device, the reinforcement-learning based application manager comprising:
one or more processors, one or more memories, and one or more communications subsystems;
a set of actions that can be issued to the computing environment; and
an iterative control process that repeatedly
selects and issues a next action to one or more of the one or more local agents according to a control policy that uses a state vector that represents a current state of the computing system,
receives, from one or more of the one or more local agents in response to execution of the issued next action, one of an observation and state vector, from which the control process generates a current state and a reward, which the control process uses to attempt to learn an optimal or near-optimal control policy.

2. The automated reinforcement-learning-based application manager of claim 1 wherein each of the one or more local agents serves as an intermediary between the automated reinforcement-learning-based application manager and a portion of the computing environment, receiving actions issued by the iterative control process, executing certain of the received actions, and issuing other of the received actions to the portion of the computing environment, and receiving state information from the portion of the computing environment and forwarding the state information to the iterative control process.

3. The automated reinforcement-learning-based application manager of claim 2 wherein each of the one or more local agents monitors the portion of the computing environment to detect various conditions and, when one or more of the various conditions is detected,
issues an action to the portion of the computing environment,
collects state information from the portion of the computing environment, and
forwards information about the issued action and the collected state information to the iterative control process.

4. The automated reinforcement-learning-based application manager of claim 3 wherein each local agent detects the various conditions by monitoring a set of rules, each rule based on a threshold supplied by an action issued by the iterative control process, and each rule indicating an action to issue when the rule indicates that a condition has occurred.

5. The automated reinforcement-learning-based application manager of claim 2 wherein, when a local agent determines that the automated reinforcement-learning-based application manager is currently incapable of issuing actions to the local agent, the local agent controls the portion of the computing environment by repeatedly
issuing an action selected from a static control policy,
collecting state information from the portion of the computing environment, and
queuing information about the action and the collected state information for transmission to the automated reinforcement-learning-based application manager.

6. The automated reinforcement-learning-based application manager of claim 1 wherein iterative control process additionally repeatedly
receives, from one or more of the one or more local agents, an action/state-information pair from which the iterative control process generates a current state and a reward, which the control process uses to attempt to learn an optimal or near-optimal control policy.

7. The automated reinforcement-learning-based application manager of claim 1 wherein each local agent includes an iterative agent control process that repeatedly
receives a next action from the automated reinforcement-learning-based application manager,
executes the action or issues the action of a portion of the computing environment,
collects state information from the portion of the computing environment, and
returns the collected state information to the automated reinforcement-learning-based application manager.

8. The automated reinforcement-learning-based application manager of claim 7 wherein the iterative agent control process additionally repeatedly
monitors the portion of the computing environment for certain conditions,
when one or more conditions has occurred,
issues an action to the portion of the computing environment,
collects state information from the portion of the computing environment, and
returns the collected state information to the automated reinforcement-learning-based application manager.

9. The automated reinforcement-learning-based application manager of claim 7 wherein, when local agent determines that automated-reinforcement-learning-based-application-manager control is absent, the iterative agent control process repeatedly
selects a next action using a static policy;
issues the selected action to the portion of the computing environment,
collects state information from the portion of the computing environment, and
queues information about the action and the collected state information for transmission to the automated reinforcement-learning-based application manager.

10. The automated reinforcement-learning-based application manager of claim 1 wherein the iterative control process decomposes actions selected according to the control policy into local-agent actions, each of which the iterative control process directs to a different local agent.

11. The automated reinforcement-learning-based application manager of claim 1 wherein the iterative control process composes a state vector or observation vector from state information returned by multiple local agents.

12. A method that provides fault tolerance and fine-granularity control to an automated reinforcement-learning-based application manager that manages a computing environment that includes one or more applications and one or more of a distributed computing system having multiple computer systems interconnected by one or more networks, a standalone computer system, and a processor-controlled user device, the automated reinforcement-learning-based application manager having one or more processors, one or more memories, one or more communications subsystems, and a set of actions that can be issued to the computing environment, the method comprising:
iteratively
selecting and issuing a next action to one or more of the one or more local agents according to a control policy that uses a state vector that represents a current state of the computing system,
receiving, from one or more of the one or more local agents in response to execution of the issued next action, one of an observation and state vector, from which the control process generates a current state and a reward, which the control process uses to attempt to learn an optimal or near-optimal control policy.

13. The method of claim 12 wherein each of the one or more local agents serves as an intermediary between the automated reinforcement-learning-based application manager and a portion of the computing environment, receiving actions issued by the iterative control process, executing certain of the received actions, and issuing other of the received actions to the portion of the computing environment, and receiving state information from the portion of the computing environment and forwarding the state information to the iterative control process.

14. The method of claim 13 wherein each of the one or more local agents monitors the portion of the computing environ

Assignees

• Vmware Inc

Inventors

• Nag Dev
• Yankov Yanislav
• Wang Dongni
• Burk Gregory T
• Stephen Nicholas Mark Grant

Classifications

• G06N7/01

  Probabilistic graphical models, e.g. probabilistic networks · CPC title

• G06F2009/4557

  Distribution of virtual machine instances; Migration and load balancing · CPC title

• G06F9/45558

  Hypervisor-specific management and integration aspects · CPC title

• G06N20/00 Primary

  Machine learning · CPC title

• G06N5/043

  Distributed expert systems; Blackboards · CPC title