Reinforcement learning using pseudo-counts

Patent metadata

Field	Value
Publication number	US-11727264-B2
Application number	US-201716303501-A
Country	US
Kind code	B2
Filing date	May 18, 2017
Priority date	May 20, 2016
Publication date	Aug 15, 2023
Grant date	Aug 15, 2023

Abstract

Methods, systems, and apparatus, including computer programs encoded on computer storage media, for training a neural network used to select actions to be performed by an agent interacting with an environment. One of the methods includes obtaining data identifying (i) a first observation characterizing a first state of the environment, (ii) an action performed by the agent in response to the first observation, and (iii) an actual reward received resulting from the agent performing the action in response to the first observation; determining a pseudo-count for the first observation; determining an exploration reward bonus that incentivizes the agent to explore the environment from the pseudo-count for the first observation; generating a combined reward from the actual reward and the exploration reward bonus; and adjusting current values of the parameters of the neural network using the combined reward.

First claim

The invention claimed is:

1. A computer-implemented method for training a neural network used to select actions to be performed by an agent interacting with an environment, the computer-implemented method comprising:
obtaining data identifying a first observation characterizing a first state of the environment;
selecting, using the neural network, an action to be performed by the agent in response to the first observation;
controlling the agent to perform the selected action;
receiving an actual reward resulting from the agent performing the action in response to the first observation;
determining a pseudo-count for the first observation using a sequential density model which represents a likelihood that the first observation occurs given a sequence of previous observations, wherein the pseudo-count depends upon a number of previous occurrences of the first observation during the training of the neural network;
determining an exploration reward bonus that incentivizes the agent to explore the environment from the pseudo-count for the first observation, wherein the exploration reward bonus is lower when the pseudo-count is higher and vice-versa;
generating a combined reward from the actual reward and the exploration reward bonus; and
training the neural network by adjusting current values of parameters of the neural network using the combined reward.

2. The computer-implemented method of claim 1 , wherein adjusting the current values of the parameters comprises:
using the combined reward in place of the actual reward in performing an iteration of a reinforcement learning technique.

3. The computer-implemented method of claim 1 , wherein the neural network is configured to receive the first observation and generates an output that defines a probability distribution over possible actions, with a probability for each action being a probability that the action maximizes chances of the agent completing a task performed by the agent.

4. The computer-implemented method of claim 1 , wherein the neural network is configured to receive an observation-action pair which is the first observation and an action performed by the agent in response to the first observation, and to generate a Q-value for the observation-action pair that represents an estimated return resulting from the agent performing the action in response the observation in the observation-action pair.

5. The computer-implemented method of claim 1 , wherein generating the combined reward comprises summing the actual reward and the exploration reward bonus.

6. The computer-implemented method of claim 1 , wherein the exploration reward bonus RB satisfies:

RB
=

β

(

N
^

⁡

(
x
)

+
a

)

b

,

wherein x is the first observation, {circumflex over (N)}(x) is the pseudo-count for the first observation, a and b are constants, and β is a parameter selected by a parameter sweep.

7. The computer-implemented method of claim 1 , wherein the pseudo-count {circumflex over (N)}(x) for the first observation is of the following form:

N
^

n

⁡

(
x
)

=

ρ
n

⁡

(
x
)

⁢

(

1
-

ρ
n
′

⁡

(
x
)

)

ρ
n
′

⁡

(
x
)

-

ρ
n

⁡

(
x
)

,

wherein ρ n (x) is a value of a sequential density model for the first observation and ρ n ′(x) is a recoding probability for the first observation, wherein the recoding probability is a value of the sequential density model after observing a new occurrence of the first observation.

8. The computer-implemented method of claim 1 , wherein the sequential density model is a pixel-level density model.

9. The computer-implemented method of claim 1 , wherein the agent is a mechanical agent and the environment is a real-world environment, and wherein the neural network is trained as the agent explores the environment.

10. One or more non-transitory computer storage media encoded with instructions that, when executed by one or more computers, cause the one or more computers to perform operations for training a neural network used to select actions to be performed by an agent interacting with an environment, the operations comprising:
obtaining data identifying a first observation characterizing a first state of the environment;
selecting, using the neural network, an action to be performed by the agent in response to the first observation;
controlling the agent to perform the selected action;
receiving an actual reward resulting from the agent performing the action in response to the first observation;
determining a pseudo-count for the first observation using a sequential density model which represents a likelihood that the first observation occurs given a sequence of previous observations, wherein the pseudo-count depends upon a number of previous occurrences of the first observation during the training of the neural network;
determining an exploration reward bonus that incentivizes the agent to explore t

Assignees

• Deepmind Tech Ltd

Inventors

• Gendron-Bellemare Marc
• Munos Remi
• Sriram Srinivasan

Classifications

• G06N3/092

  Reinforcement learning · CPC title

• G06N3/08 Primary

  Learning methods · CPC title

• G06F17/18

  for evaluating statistical data {, e.g. average values, frequency distributions, probability functions, regression analysis (forecasting specially adapted for a specific administrative, business or logistic context G06Q10/04)} · CPC title

• G06N3/047

  Probabilistic or stochastic networks · CPC title

• G06N3/006

  based on simulated virtual individual or collective life forms, e.g. social simulations or particle swarm optimisation [PSO] · CPC title