Data-efficient reinforcement learning for continuous control tasks

Patent metadata

Field	Value
Publication number	US-12346786-B2
Application number	US-202318351440-A
Country	US
Kind code	B2
Filing date	Jul 12, 2023
Priority date	Jan 31, 2017
Publication date	Jul 1, 2025
Grant date	Jul 1, 2025

Abstract

Methods, systems, and apparatus, including computer programs encoded on computer storage media, for data-efficient reinforcement learning. One of the systems is a system for training an actor neural network used to select actions to be performed by an agent that interacts with an environment by receiving observations characterizing states of the environment and, in response to each observation, performing an action selected from a continuous space of possible actions, wherein the actor neural network maps observations to next actions in accordance with values of parameters of the actor neural network, and wherein the system comprises: a plurality of workers, wherein each worker is configured to operate independently of each other worker, wherein each worker is associated with a respective agent replica that interacts with a respective replica of the environment during the training of the actor neural network.

First claim

What is claimed is:

1. A system for training an actor neural network used to select actions to be performed by an agent that interacts with an environment by receiving observations characterizing states of the environment and, in response to each observation, performing an action selected from a continuous space of possible actions, wherein the actor neural network maps observations to next actions in accordance with values of parameters of the actor neural network, and wherein the system comprises:
a plurality of workers, each of the plurality of workers having access to a shared memory configured to store current parameters of the actor neural network,
wherein each worker is configured to operate independently of each other worker,
wherein each worker is configured to communicate with and provide instructions to a respective agent replica that interacts with a respective replica of the environment during the training of the actor neural network,
wherein each worker is configured to repeatedly perform (i) a set of updating operations comprising determining that a threshold number of writes to the shared memory have occurred since a preceding update to the values of the parameters of the actor neural network, and in response to the determination, updating current values of the parameters of the actor neural network in the shared memory, and (ii) a set of acting operations to control the respective agent replica using the actor neural network to perform an action in the respective replica of the environment in order to generate training data for training the actor neural network,
wherein the set of the acting operations comprises:
receiving a current observation characterizing a current state of the environment replica interacted with by the agent replica associated with the worker,
selecting a current action to be performed by the agent replica associated with the worker in response to the current observation using the actor neural network and in accordance with the current values of the parameters,
identifying an actual reward resulting from the agent replica performing the current action in response to the current observation, and
receiving a next observation characterizing a next state of the environment replica interacted with by the agent replica, wherein the environment replica transitioned into the next state from the current state in response to the agent replica performing the current action, and

wherein each worker performs a plurality of iterations of the set of updating operations after selecting the current action and prior to selecting a new action in response to the next observation.

2. The system of claim 1 , the set of updating operations comprising:
determining current values of the parameters of the actor neural network and of a critic neural network from the shared memory accessible by each of the plurality of workers;
obtaining a minibatch of experience tuples from a replay memory, each experience tuple comprising a training observation characterizing a training state of the environment, a training action from the continuous space of actions performed by one of the agent replicas in response to the training observation, a training reward received by the agent replica for performing the training action, and a next training observation characterizing a next training state of the environment;
determining updates to the current values of the parameters of the actor neural network and to the current values of the parameters of the critic neural network using the minibatch of experience tuples; and
writing the updates to the current values of the parameters of the actor neural network and to the current values of the parameters of the critic neural network to the shared memory.

3. The system of claim 2 , the acting operations further comprising:
generating a new experience tuple that includes the current observation, the current action, the actual reward, and the next observation, and
storing the new experience tuple in the replay memory.

4. The system of claim 2 , wherein selecting the new action to be performed in response to the next observation comprises:
determining new values of the parameters of the actor neural network from the shared memory;
processing the new observation using the actor neural network in accordance with the new values of the parameters to map the new observation to a new next action; and
selecting the new action to be performed based on the new next action.

5. The system of claim 1 , wherein each worker executes independently of each other worker on the same computer.

6. The system of claim 1 , wherein each worker executes independently of each other worker on different computers.

7. The system of claim 3 , the acting operations further comprising:
determining that the next state that the environment transitioned to is a state in which a subtask of a task being performed by the agent replica has been completed; and
assigning the actual reward to be a reward associated with completion of the subtask by a composite reward function.

8. The system of claim 1 , wherein the agent replica interacts with the environment replica to complete a task, and wherein the acting operations further comprise:
identifying a trajectory of actions and corresponding states that resulted in successful completion of the task by the agent replica or by another actor; and
initializing an initial state of the agent replica and the environment to a state that matches one of the states in the trajectory.

9. The system of claim 8 , wherein initializing the initial state of the agent replica and the environment to a state that matches one of the states in the trajectory comprises:
selecting one of the states in the trajectory randomly; and
initializing an initial state of the agent replica and the environment to a state that matches the selected state.

10. The system of claim 1 , wherein each worker operates asynchronously from each other worker and wherein each worker writes to and reads from a shared memory storing the current values of the parameters asynchronously from each other worker.

11. The system of claim 1 , wherein the plurality of iterations is a predetermined number of iterations greater than one.

Assignees

• Deepmind Tech Ltd

Inventors

• Riedmiller Martin
• Hafner Roland
• Vecerik Mel
• Lillicrap Timothy Paul
• Lampe Thomas
• Popov Ivaylo
• Barth-Maron Gabriel
• Heess Nicolas Manfred Otto

Classifications

• G06N3/092 Primary

  Reinforcement learning · CPC title

• G06N3/0499

  Feedforward networks · CPC title

• G06N3/098

  Distributed learning, e.g. federated learning · CPC title

• G06F18/2148

  characterised by the process organisation or structure, e.g. boosting cascade · CPC title

• G06F18/2185

  the supervisor being an automated module, e.g. intelligent oracle · CPC title