Systems and methods for learning reusable options to transfer knowledge between tasks

Patent metadata

Field	Value
Publication number	US-11511413-B2
Application number	US-202016900291-A
Country	US
Kind code	B2
Filing date	Jun 12, 2020
Priority date	Jun 12, 2020
Publication date	Nov 29, 2022
Grant date	Nov 29, 2022

Abstract

A robot that includes an RL agent that is configured to learn a policy to maximize the cumulative reward of a task, to determine one or more features that are minimally correlated with each other. The features are then used as pseudo-rewards, called feature rewards, where each feature reward corresponds to an option policy, or skill, the RL agent learns to maximize. In an example, the RL agent is configured to select the most relevant features to learn respective option policies from. The RL agent is configured to, for each of the selected features, learn the respective option policy that maximizes the respective feature reward. Using the learned option policies, the RL agent is configured to learn a new (second) policy for a new (second) task that can choose from any of the learned option policies or actions available to the RL agent.

First claim

What is claimed is:

1. A method comprising:
learning a first policy to maximize a cumulative reward of a first task in an environment, the first policy being learned using a reinforcement learning algorithm and first transition tuples collected in an environment, wherein each first transition tuple includes state, action, reward of the first policy after taking the action;
extracting a feature network from the neural network;
computing the variance of each feature output by the feature network;
selecting at least one feature of the features based on the computed variance;
for each selected feature, learning an option policy from second transition tuples collected in the environment that maximizes a cumulative feature reward of the selected feature and storing the learned option policy for the selected feature in an augmented action space, wherein each second transition tuple includes state, action, feature reward and next state; and
learning a second policy to maximize a second cumulative reward for a second task, the second policy learned by choosing one of the learned option policies in the augmented action space and using a reinforcement learning algorithm and third transition tuples collected in an environment, wherein each third transition tuple includes state, the chosen option policy, reward of the chosen learned option policy after taking the action generated by the chosen learned option policy, and next state.

2. The method as claimed in claim 1 , wherein selecting at least one feature of the features based on the computed variance comprises selecting at least one feature that exceeds a predetermined threshold variance.

3. The method as claimed in claim 1 , wherein the selecting at least one of the features comprises ranking the features from highest variance to lowest variance, and selecting a specified number of features having the highest variance.

4. The method as claimed in claim 1 , wherein the feature network generates features that are minimally correlated with each other.

5. The method as claimed in claim 1 , wherein each learned option policy maps state to action.

6. The method as claimed in claim 1 , wherein the first policy is modeled as a neural network that maps state to action.

7. The method as claimed in claim 1 , wherein the first policy is an action that maximizes a value function in a state and wherein the value function is modelled as a neural network that maps state and action to value.

8. The method as claimed in claim 1 , further comprising storing, in the augmented action space, primitive actions, and learning the second policy by choosing a primitive action and executing the primitive action for one time step.

9. The method as claimed in claim 1 , wherein learning the second policy comprises executing the chosen learned option policy over one or more time steps until a future cumulated reward of the selected learned option policy is maximized.

10. A robot comprising:
memory;
a processing unit configured to execute instructions of an agent stored in the memory to:
learn a first policy to maximize a cumulative reward of a first task in an environment, the first policy being learned using a reinforcement learning algorithm and first transition tuples collected in an environment, wherein each first transition tuple includes state, action, reward of the first policy after taking the action;
extract a feature network from the neural network;
compute the variance of each feature output by the feature network;
select at least one feature of the features based on the computed variance;
for each selected feature, learn an option policy from second transition tuples collected in the environment that maximizes a cumulative feature reward of the selected feature and storing the learned option policy for the selected feature in an augmented action space, wherein each second transition tuple includes state, action, feature reward and next state; and
learn a second policy to maximize a second cumulative reward for a second task, the second policy learned by choosing one of the learned option policies in the augmented action space and using a reinforcement learning algorithm and third transition tuples collected in an environment, wherein each third transition tuple includes state, the chosen option policy, reward of the chosen learned option policy after taking the action generated by the chosen learned option policy, and next state.

11. The robot as claimed in claim 10 , wherein at least one feature of the features is selected based on the computed variance comprises selecting at least one feature that exceeds a predetermined threshold variance.

12. The robot as claimed in claim 10 , wherein the selection at least one of the features comprises ranking the features from highest variance to lowest variance, and selecting a specified number of features having the highest variance.

13. The robot as claimed in claim 10 , wherein the feature network generates features that are minimally correlated with each other.

14. The robot as claimed in claim 10 , wherein each learned option policy maps state to action.

15. The robot as claimed in claim 10 , wherein the first policy is modeled as a neural network that maps state to action.

16. The robot as claimed in claim 10 , wherein the first policy is an action that maximizes a value function in a state and wherein the value function is modelled as a neural network that maps state and action to value.

17. The robot as claimed in claim 1 , wherein the processing unit is further configured to execute further instructions of the agent stored in the memory to store, in the augmented action space, primitive actions, and learn the second policy by choosing a primitive action and executing the primitive action for one time step.

18. The method as claimed in claim 1 , wherein the second policy is learned by executing the chosen learned option policy over one or more time steps until a future cumulated reward of the selected learned option policy is maximized.

19. A non-transitory computer-readable medium having instructions stored thereon which when executed by an agent of a robot cause the agent to:
learn a first policy to maximize a cumulative reward of a first task in an environment, the first policy being learned using a reinforcement learning algorithm and first transition tuples collected in an environment, wherein each first transition tuple includes state, action, reward of the first policy after taking the action;
extract a feature network from the neural network;
compute the variance of each feature output by the feature network;
select at least one feature of the features based on the computed variance;
for each selected feature, learn an option policy from second transition tuples collected in the environment that maximizes a cumulative feature reward of the selected feature and storing the learned option policy for the selected feature in an augmented action space, wherein each second transition tuple includes state, action, feature reward and next state; and

learn a second policy to maximize a second cumulative reward for a second task, the second policy learned by choosing one of the learned option policies in the augmented action space and using a reinforcement learning algorithm and third transition tuples collected in an environment, wherein each third transition tuple includes state, the chosen option policy, reward of the chosen learned option policy after taking the action generated by the chosen learned option policy, and next state.

Assignees

• Mavrin Borislav
• Graves Daniel Mark
• Huawei Tech Co Ltd

Inventors

• Mavrin Borislav
• Graves Daniel Mark

Classifications

• B25J9/163 Primary

  learning, adaptive, model based, rule based expert control · CPC title

• G06N3/08

  Learning methods · CPC title

• G06N3/006

  based on simulated virtual individual or collective life forms, e.g. social simulations or particle swarm optimisation [PSO] · CPC title

• G05B2219/40499

  Reinforcement learning algorithm · CPC title

• B25J9/161 Primary

  Hardware, e.g. neural networks, fuzzy logic, interfaces, processor · CPC title