Environment prediction using reinforcement learning

Patent metadata

Field	Value
Publication number	US-10733501-B2
Application number	US-201916403314-A
Country	US
Kind code	B2
Filing date	May 3, 2019
Priority date	Nov 4, 2016
Publication date	Aug 4, 2020
Grant date	Aug 4, 2020

Abstract

Methods, systems, and apparatus, including computer programs encoded on a computer storage medium, for prediction of an outcome related to an environment. In one aspect, a system comprises a state representation neural network that is configured to: receive an observation characterizing a state of an environment being interacted with by an agent and process the observation to generate an internal state representation of the environment state; a prediction neural network that is configured to receive a current internal state representation of a current environment state and process the current internal state representation to generate a predicted subsequent state representation of a subsequent state of the environment and a predicted reward for the subsequent state; and a value prediction neural network that is configured to receive a current internal state representation of a current environment state and process the current internal state representation to generate a value prediction.

First claim

What is claimed is:

1. A method performed by one or more data processing apparatus, the method comprising:
receiving, by the one or more data processing apparatus, one or more observations characterizing states of an environment being interacted with by an agent;
providing, by the one or more data processing apparatus, the one or more observations as input to a state representation neural network, wherein the state representation neural network is configured to:
receive the one or more observations, and
process the one or more observations to generate an internal state representation of a current environment state;

for each of a plurality of internal time steps:
generating, by the one or more data processing apparatus, using a prediction neural network and a value prediction neural network and from an internal state representation for the internal time step: (i) an internal state representation for a next internal time step, (ii) a predicted reward for the next internal time step, and (iii) a value prediction that is an estimate of a future cumulative discounted reward from the next internal time step onwards;
wherein the prediction neural network is configured to, for each of the plurality of internal time steps:
receive the internal state representation for the internal time step; and
process the internal state representation for the internal time step to generate:
an internal state representation for a next internal time step, and a predicted reward for the next internal time step;

wherein the value prediction neural network is configured to, for each of the plurality of internal time steps:
receive the internal state representation for the internal time step, and
process the internal state representation for the internal time step to generate a value prediction that is an estimate of a future cumulative discounted reward from the next internal time step onwards; and

determining, by the one or more data processing apparatus, an aggregate reward from the predicted rewards and the value predictions for the internal time steps, wherein the aggregate reward is an estimate of an outcome associated with the states of the environment characterized by the observations.

2. The method of claim 1 , further comprising:
providing the aggregate reward as an estimate of the outcome associated with the states of the environment characterized by the observations.

3. The method of claim 1 , wherein the prediction neural network is further configured to generate a predicted discount factor for the next internal time step, and further comprising using the predicted discount factors for the internal time steps in determining the aggregate reward.

4. The method of claim 1 , wherein the state representation neural network comprises a recurrent neural network.

5. The method of claim 3 , further comprising:
for each internal time step,
processing, by the one or more data processing apparatus, an internal state representation for the internal time step using a lambda neural network to generate a lambda factor for the next internal time step;

determining, by the one or more data processing apparatus, a respective k-step return for each internal time step by combining the predicted reward and the predicted discount factor for each of a first k internal time steps and the value prediction for a k-th internal time step; and
using the lambda factors to determine weights for the k-step returns in determining the aggregate reward.

6. The method of claim 1 , wherein the state representation neural network comprises a feedforward neural network.

7. The method of claim 1 , wherein the prediction neural network comprises a recurrent neural network.

8. The method of claim 1 , wherein the prediction neural network comprises a feedforward neural network that has different parameter values at each of the plurality of internal time steps.

9. A system comprising:
one or more computers; and
one or more storage devices communicatively coupled to the one or more computers, wherein the one or more storage devices store instructions that, when executed by the one or more computers, cause the one or more computers to perform operations comprising:
receiving, by the one or more computers, one or more observations characterizing states of an environment being interacted with by an agent;
providing, by the one or more computers, the one or more observations as input to a state representation neural network, wherein the state representation neural network is configured to:
receive the one or more observations, and
process the one or more observations to generate an internal state representation of a current environment state;

for each of a plurality of internal time steps:
generating, by the one or more computers, using a prediction neural network and a value prediction neural network and from an internal state representation for the internal time step: (i) an internal state representation for a next internal time step, (ii) a predicted reward for the next internal time step, and (iii) a value prediction that is an estimate of a future cumulative discounted reward from the next internal time step onwards;
wherein the prediction neural network is configured to, for each of the plurality of internal time steps:
receive the internal state representation for the internal time step; and
process the internal state representation for the internal time step to generate:
an internal state representation for a next internal time step, and
a predicted reward for the next internal time step;

wherein the value prediction neural network is configured to, for each of the plurality of internal time steps:
receive the internal state representation for the internal time step, and
process the internal state representation for the internal time step to generate a value prediction that is an estimate of a future cumulative discounted reward from the next internal time step onwards; and

determining, by the one or more computers, an aggregate reward from the predicted rewards and the value predictions for the internal time steps, wherein the aggregate reward is an estimate of an outcome associated with the states of the environment characterized by the observations.

10. The system of claim 9 , wherein the operations further comprise:
providing the aggregate reward as an estimate of the outcome associated with the states of the environment characterized by the observations.

11. The system of claim 9 , wherein the prediction neural network is further configured to generate a predicted discount factor for the next internal time step, and wherein the operations further comprise using the predicted discount factors for the internal time steps in determining the aggregate reward.

12. The system of claim 9 , wherein the state representation neural network comprises a recurrent neural network.

13. The system of claim 11 , wherein the operations further comprise:
for each internal time step,
processing, by the one or more data processing apparatus, an internal state representation for the internal time step using a lambda neural network to generate a lambda factor for the next internal time step;

determining, by the one or more computers, a respective k-step return for each internal time step by combining the predicted reward and the predicted discount factor for each of a first k internal time steps and the value prediction for a k-th internal time step; and
using the lambda factors to determine weights for the k-step returns in determining the aggregate reward.

14. The system of claim 9 , wherein the state representation neural network comprises a

Assignees

• Deepmind Tech Ltd

Inventors

• Silver David
• Schaul Tom
• Hessel Matteo
• van Hasselt Hado Philip

Classifications

• G06N3/044

  Recurrent networks, e.g. Hopfield networks · CPC title

• G06N3/045 Primary

  Combinations of networks · CPC title

• G06N3/006 Primary

  based on simulated virtual individual or collective life forms, e.g. social simulations or particle swarm optimisation [PSO] · CPC title

• G06N3/047

  Probabilistic or stochastic networks · CPC title

• G06N3/092 Primary

  Reinforcement learning · CPC title