Selecting reinforcement learning actions using goals and observations

Patent metadata

Field	Value
Publication number	US-10628733-B2
Application number	US-201615091840-A
Country	US
Kind code	B2
Filing date	Apr 6, 2016
Priority date	Apr 6, 2015
Publication date	Apr 21, 2020
Grant date	Apr 21, 2020

Abstract

Methods, systems, and apparatus, including computer programs encoded on computer storage media, for reinforcement learning using goals and observations. One of the methods includes receiving an observation characterizing a current state of the environment; receiving a goal characterizing a target state from a set of target states of the environment; processing the observation using an observation neural network to generate a numeric representation of the observation; processing the goal using a goal neural network to generate a numeric representation of the goal; combining the numeric representation of the observation and the numeric representation of the goal to generate a combined representation; processing the combined representation using an action score neural network to generate a respective score for each action in the predetermined set of actions; and selecting the action to be performed using the respective scores for the actions in the predetermined set of actions.

First claim

What is claimed is:

1. A method for selecting an action to be performed by a reinforcement learning agent that interacts with an environment by receiving observations characterizing a current state of the environment and, in response, performing actions from a predetermined set of actions, wherein the method comprises:
receiving an observation characterizing a current state of the environment;
receiving, from a user or from an external system, an input identifying a target state from a set of target states of the environment that is different from the current state of the environment characterized by the observation, wherein the target state is a state to be reached by the reinforcement learning agent by interacting with the environment;
generating a goal that corresponds to the target state identified in the input received from the user or from the external system; and
processing the goal and the observation using a universal value function approximator (UVFA) that generalizes over states and goals in order to select an action to be performed by the reinforcement learning agent in order to reach the provided target state in response to the observation based on the observation and the generated goal, comprising:
processing the observation using an observation neural network to generate a numeric representation of the observation;
processing the goal using a goal neural network to generate a numeric representation of the goal;
combining the numeric representation of the observation and the numeric representation of the goal to generate a combined representation of the observation and the goal;
processing the combined representation of the observation and the goal using an action score neural network to generate a respective score for each action in the predetermined set of actions; and
selecting the action to be performed using the respective scores for the actions in the predetermined set of actions.

2. The method of claim 1 , wherein the action score neural network is configured to receive as input the combined representation and to process the combined representation to generate the respective score for each action in the predetermined set of actions.

3. The method of claim 1 , wherein the action score neural network is configured to receive as input the combined representation and an action from the predetermined set of actions and to process the combined representation and the received action to generate a score for the action, and wherein processing the combined representation using the action score neural network comprises, for each action in the predetermined set of actions:
processing the action and the combined representation using the action score neural network to generate the respective score for the action.

4. The method of claim 1 , wherein selecting the action to be performed comprises:
selecting a highest-scoring action.

5. The method of claim 1 , wherein selecting the action to be performed comprises:
selecting a random action from the set of actions with probability ε; and
selecting a highest-scoring action with probability 1−ε.

6. The method of claim 1 , wherein the observation neural network is a multi-layer perceptron (MLP) neural network.

7. The method of claim 6 , wherein the goal neural network is a multi-layer perceptron (MLP) neural network.

8. The method of claim 1 , wherein the goal neural network and the observation neural networks share one or more parameters.

9. The method of claim 1 , wherein the combined representation is a dot product of the numeric representation of the observation and the numeric representation of the goal.

10. The method of claim 1 , wherein combining the numeric representation of the observation and the numeric representation of the goal to generate the combined representation comprises:
processing the numeric representation of the observation and the numeric representation of the goal using a combining neural network to generate the combined representation.

11. The method of claim 1 , further comprising:
training the action score neural network, the observation neural network, and the goal neural network.

12. The method of claim 11 , wherein the training comprises:
training the action score neural network to determine trained goal embeddings and trained observation embeddings;
training the observation neural network on the trained observation embeddings; and
training the goal neural network on the trained goal embeddings.

13. The method of claim 12 , wherein training the action score neural network comprises determining the trained goal embeddings and trained observation embeddings using a matrix factorization technique.

14. The method of claim 12 , wherein the action score neural network is trained to generate action scores that match a set of target action scores, and wherein the target action scores for the training of the action score neural network are determined by learning a separate action score neural network for each goal in the predetermined set of goals.

15. The method of claim 12 , wherein the action score neural network is trained to generate action scores that match a set of target action scores, and wherein the target action scores for the training of the action score neural network are determined by bootstrapping from the states in visited transitions for each goal in a training set.

16. The method claim 1 , further comprising:
providing an interface for selecting target states, such that a selected goal is used in selecting actions to be performed until the selected goal is switched through the interface.

17. A system comprising one or more computers and one or more storage devices storing instructions that, when executed by the one or more computers, cause the one or more computers to perform operations for selecting an action to be performed by a reinforcement learning agent that interacts with an environment by receiving observations characterizing a current state of the environment and, in response, performing actions from a predetermined set of actions, the operations comprising:
receiving an observation characterizing a current state of the environment;
receiving, from a user or from an external system, an input identifying a target state from a set of target states of the environment that is different from the current state of the environment characterized by the observation, wherein the target state is a state to be reached by the reinforcement learning agent by interacting with the environment;
generating a goal that corresponds to the target state identified in the input received from the user or from the external system; and
processing the goal and the observation using a universal value function approximator (UVFA) that generalizes over states and goals in order to select an action to be performed by the reinforcement learning agent in order to reach the provided target state in response to the observation based on the observation and the generated goal, comprising:
processing the observation using an observation neural network to generate a numeric representation of the observation;
processing the goal using a goal neural network to generate a numeric representation of the goal;
combining the numeric representation of the observation and the numeric representation of the goal to generate a combined representation of the observation and the goal;
processing the combined representation of the observation and the goal using an action score neural network to generate a respective score for each action in the predetermined set of actions; and
selecting the action

Assignees

• Deepmind Tech Ltd

Inventors

• Schaul Tom
• Horgan Daniel George
• Gregor Karol
• Silver David

Classifications

• G06N20/00

  Machine learning · CPC title

• G06N3/08 Primary

  Learning methods · CPC title

• G06N3/04

  Architecture, e.g. interconnection topology · CPC title

• G06N3/084 Primary

  Backpropagation, e.g. using gradient descent · CPC title

• G06N3/0454

  Physics · mapped topic