Continuous control with deep reinforcement learning

Patent metadata

Field	Value
Publication number	US-12561573-B2
Application number	US-202318497931-A
Country	US
Kind code	B2
Filing date	Oct 30, 2023
Priority date	Jul 24, 2015
Publication date	Feb 24, 2026
Grant date	Feb 24, 2026

Abstract

Methods, systems, and apparatus, including computer programs encoded on computer storage media, for training an actor neural network used to select actions to be performed by an agent interacting with an environment. One of the methods includes obtaining a minibatch of experience tuples; and updating current values of the parameters of the actor neural network, comprising: for each experience tuple in the minibatch: processing the training observation and the training action in the experience tuple using a critic neural network to determine a neural network output for the experience tuple, and determining a target neural network output for the experience tuple; updating current values of the parameters of the critic neural network using errors between the target neural network outputs and the neural network outputs; and updating the current values of the parameters of the actor neural network using the critic neural network.

First claim

What is claimed is:

1 . A method performed by one or more computers, the method comprising:
receiving a current observation characterizing a current state of an environment; processing the current observation using an actor neural network to map an input observation to a point in a continuous action space that defines an action to be performed by an agent in response to the input observation; and causing the agent to perform the action defined by the point in the continuous action space, wherein the actor neural network has been trained jointly with a critic neural network that is configured to receive a critic input specifying a training observation and a training action and to process the critic input to generate a neural network output that is an estimate of time-discounted total future rewards that will be received in response to the agent performing the training action in response to the training observation, and wherein jointly training the critic neural network and the actor neural network comprises:
obtaining a minibatch of experience tuples from a replay memory, each experience tuple comprising a training observation characterizing a training state of an environment, a training action from a continuous space of actions performed by an agent in response to receiving the training observation, a training reward received by the agent for performing the training action, and a next training observation characterizing a next training state of the environment; and
updating current values of parameters of the actor neural network using the minibatch of experience tuples, wherein the updating comprises:
for each experience tuple in the minibatch:
processing the training observation and the training action in the experience tuple using the critic neural network to determine a neural network output for the experience tuple in accordance with current values of parameters of the critic neural network;
processing the next training observation using a target actor neural network to determine a predicted next action for the experience tuple in accordance with current values of parameters of the target actor neural network, wherein the target actor neural network has a same neural network architecture as the actor neural network but the current values of the parameters of the target actor neural network are different from the current values of the parameters of the actor neural network;
processing the next training observation and the predicted next action for the experience tuple using a target critic neural network to generate a predicted next neural network output in accordance with current values of parameters of the target critic neural network, wherein the target critic neural network has a same neural network architecture as the critic neural network but the current values of the parameters of the target critic neural network are different from the current values of the parameters of the critic neural network; and
determining a target neural network output for the experience tuple from the training reward and the predicted next neural network output for the experience tuple,

updating the current values of the parameters of the critic neural network using errors between the target neural network outputs and the neural network outputs for the experience tuples in the minibatch; and
updating current values of the parameters of the actor neural network using the critic neural network.

2 . The method of claim 1 , wherein jointly training the critic neural network and the actor neural network further comprises:
updating the current values of the parameters of the target actor neural network using the updated values of the parameters of the actor neural network; and updating the current values of the parameters of the target critic neural network using the updated values of the parameters of the critic neural network.

3 . The method of claim 2 , wherein the current values of the parameters of the target actor neural network and the target critic neural network are constrained to change slowly during the training of the actor neural network.

4 . The method of claim 1 , wherein the critic neural network, the actor neural network, or both include one or more batch normalization neural network layers.

5 . The method of claim 1 , wherein the current observation comprises one or more low-dimensional feature vectors that characterize the current state of the environment.

6 . The method of claim 5 , wherein values of different dimensions of the low-dimensional feature vectors have varying ranges.

7 . The method of claim 1 , wherein the current observation comprises high-dimensional pixel inputs from one or more images that characterize the current state of the environment.

8 . The method of claim 1 , wherein jointly training the critic neural network and the actor neural network further comprises:
independently from updating the current values of the parameters of the actor neural network using the minibatch of experience tuples:
generating a new experience tuple, comprising:
receiving a new training observation,
processing the new training observation using the actor neural network to select a new training action to be performed by the agent in accordance with the current values of the parameters of the actor neural network,
receiving a new training reward in response to the agent performing the new training action,
receiving a new next training observation, and
generating a new experience tuple that includes the new training observation, the new training action, the new training reward, and the new next training observation; and

adding the new experience tuple to the replay memory.

9 . The method of claim 8 , wherein processing the new training observation using the actor neural network to select a new training action comprises:
processing the new training observation using the actor neural network to generate an initial new training action; sampling from a noise process to obtain a noise factor; and adjusting the initial new action by the noise factor to generate the new training action.

10 . The method of claim 1 , wherein updating the current values of the parameters of the actor neural network using the critic neural network comprises:
for each experience tuple in the minibatch:
processing the training observation in the experience tuple using the actor neural network in accordance with the current values of the parameters of the actor neural network to generate a next action for the training observation;
determining, by backpropagating through the critic neural network, a first gradient of the critic neural network with respect to the next action taken at the training observation-next action input pair and in accordance with the current values of the parameters of the critic neural network;
determining, by backpropagating through the actor neural network, a second gradient of the actor neural network with respect to the parameters of the actor neural network taken at the training observation and in accordance with current values of the parameters of the actor neural network; and
determining a parameter update for the actor neural network from the first gradient and the second gradient.

11 . A system comprising one or more computers and one or more storage devices storing instructions that, when executed by the one or more computers, cause the one or more computers to perform operations comprising:
receiving a current observation characterizing a current state of an environment; processing the current observation using an actor neural network to map an input observation to a point in a continuous action space that defines an action to be performe

Assignees

• Deepmind Tech Ltd

Inventors

• Lillicrap Timothy Paul
• Hunt Jonathan James
• Pritzel Alexander
• Heess Nicolas Manfred Otto
• Erez Tom
• Tassa Yuval
• Silver David
• Wierstra Daniel Pieter

Classifications

• G06N3/08 Primary

  Learning methods · CPC title

• G06N3/045 Primary

  Combinations of networks · CPC title

• G06N3/084

  Backpropagation, e.g. using gradient descent · CPC title

• G06N3/006 Primary

  based on simulated virtual individual or collective life forms, e.g. social simulations or particle swarm optimisation [PSO] · CPC title

• G06N3/049

  Temporal neural networks, e.g. delay elements, oscillating neurons or pulsed inputs · CPC title