Data-efficient reinforcement learning for continuous control tasks

Patent metadata

Field	Value
Publication number	US-10664725-B2
Application number	US-201916528260-A
Country	US
Kind code	B2
Filing date	Jul 31, 2019
Priority date	Jan 31, 2017
Publication date	May 26, 2020
Grant date	May 26, 2020

Abstract

Methods, systems, and apparatus, including computer programs encoded on computer storage media, for data-efficient reinforcement learning. One of the systems is a system for training an actor neural network used to select actions to be performed by an agent that interacts with an environment by receiving observations characterizing states of the environment and, in response to each observation, performing an action selected from a continuous space of possible actions, wherein the actor neural network maps observations to next actions in accordance with values of parameters of the actor neural network, and wherein the system comprises: a plurality of workers, wherein each worker is configured to operate independently of each other worker, wherein each worker is associated with a respective agent replica that interacts with a respective replica of the environment during the training of the actor neural network.

First claim

What is claimed is:

1. A system for training an actor neural network used to select actions to be performed by an agent that interacts with an environment by receiving observations characterizing states of the environment and, in response to each observation, performing an action selected from a continuous space of possible actions, wherein the actor neural network maps observations to next actions in accordance with values of parameters of the actor neural network, and wherein the system comprises:
a plurality of workers,
wherein each worker is configured to operate independently of each other worker,
wherein each worker is associated with a respective agent replica that interacts with a respective replica of the environment during the training of the actor neural network, and
wherein each worker is further configured to repeatedly perform operations comprising:
determining current values of the parameters of the actor neural network and of a critic neural network from a shared memory accessible by each of the plurality of workers;
obtaining a minibatch of experience tuples from a replay memory, each experience tuple comprising a training observation characterizing a training state of the environment, a training action from the continuous space of actions performed by one of the agent replicas in response to the training observation, a training reward received by the agent replica for performing the training action, and a next training observation characterizing a next training state of the environment;
determining updates to the current values of the parameters of the actor neural network and to the current values of the parameters of the critic neural network, comprising, for each experience tuple in the minibatch:
processing the training observation and the training action in the experience tuple using the critic neural network to determine a neural network output for the experience tuple in accordance with the current values of the parameters of the critic neural network,
determining a target neural network output for the experience tuple from the training reward in the experience tuple and the next training observation in the experience tuple,
determining an update to the current values of the parameters of the critic neural network using errors between the target neural network outputs and the neural network outputs for the experience tuples in the minibatch, and
determining an update to the current values of the parameters of the actor neural network using the critic neural network, comprising:
 processing the training observation in the experience tuple using the actor neural network in accordance with the current values of the parameters of the actor neural network to generate a next action for the training observation; and
 determining a parameter update for the actor neural network based on (i) a gradient of the critic neural network with respect to the next action taken at the training observation—next action input pair and in accordance with the current values of the parameters of the critic neural network and (ii) the gradient of the actor neural network with respect to the parameters of the actor neural network taken at the training observation and in accordance with current values of the parameters of the actor neural network; and

writing the updates to the current values of the parameters of the actor neural network and to the current values of the parameters of the critic neural network to the shared memory.

2. The system of claim 1 , the operations further comprising:
receiving a current observation characterizing a current state of the environment replica interacted with by the agent replica associated with the worker;
selecting a current action to be performed by the agent replica associated with the worker in response to the current observation using the actor neural network and in accordance with the current values of the parameters;
identifying an actual reward resulting from the agent replica performing the current action in response to the current observation;
receiving a next observation characterizing a next state of the environment replica interacted with by the agent replica, wherein the environment replica transitioned into the next state from the current state in response to the agent replica performing the current action;
generating a new experience tuple that includes the current observation, the current action, the actual reward, and the next observation; and
storing the new experience tuple in the replay memory.

3. The system of claim 2 , the operations further comprising:
performing multiple iterations of the determining, obtaining, determining, and writing of claim 1 prior to selecting a new action to be performed in response to the next observation.

4. The system of claim 3 , wherein the number of iterations performed prior to selecting the new action is a predetermined number greater than one.

5. The system of claim 3 , wherein selecting the new action to be performed in response to the next observation comprises:
determining new values of the parameters of the actor neural network from the shared memory;
processing the new observation using the actor neural network in accordance with the new values of the parameters to map the new observation to a new next action; and
selecting the new action to be performed based on the new next action.

6. The system of claim 1 , wherein determining a target neural network output for the experience tuple comprises:
processing the next training observation using a target actor neural network to determine a predicted next action for the experience tuple in accordance with current values of parameters of the target actor neural network, wherein the target actor neural network is identical to the actor neural network but the current values of the parameters of the target actor neural network are different from the current values of the parameters of the actor neural network;
processing the next training observation and the predicted next action for the experience tuple using a target critic neural network to generate a predicted next neural network output in accordance with current values of parameters of the target critic neural network, wherein the target critic neural network is identical to the critic neural network but the current values of the parameters of the target critic neural network are different from the current values of the parameters of the critic neural network; and
determining the target neural network output for the experience tuple from the training reward and the predicted next neural network output for the experience tuple.

7. The system of claim 6 , the operations further comprising:
updating the current values of the parameters of the target actor neural network using updated values of the parameters of the actor neural network; and
updating the current values of the parameters of the target critic neural network using updated values of the parameters of the critic neural network.

8. The system of claim 7 , wherein the current values of the parameters of the target actor neural network and the target critic neural network are constrained to change slowly during the training of the actor neural network.

9. The system of claim 7 , the operations further comprising:
determining that a threshold number of writes to the shared memory have occurred since a preceding update to the values of the parameters of the target actor neural network and the target critic neural network; and
in response, updating the current values of the parameters of the target actor neural network and the current values of the parameters of the target critic neural network.

10. The system of claim 1 , wherein ea

Assignees

• Deepmind Tech Ltd

Inventors

• Riedmiller Martin
• Hafner Roland
• Vecerik Mel
• Lillicrap Timothy Paul
• Lampe Thomas
• Popov Ivaylo
• Barth-Maron Gabriel
• Heess Nicolas Manfred Otto

Classifications

• G06N3/088

  Non-supervised learning, e.g. competitive learning · CPC title

• G06N3/08

  Learning methods · CPC title

• G06K9/6257

  Physics · mapped topic

• G06K9/6264 Primary

  Physics · mapped topic

• G06N3/0454

  Physics · mapped topic