Training neural networks using a prioritized experience memory

Patent metadata

Field	Value
Publication number	US-11568250-B2
Application number	US-202016866365-A
Country	US
Kind code	B2
Filing date	May 4, 2020
Priority date	Nov 12, 2015
Publication date	Jan 31, 2023
Grant date	Jan 31, 2023

Abstract

Methods, systems, and apparatus, including computer programs encoded on a computer storage medium, for training a neural network used to select actions performed by a reinforcement learning agent interacting with an environment. In one aspect, a method includes maintaining a replay memory, where the replay memory stores pieces of experience data generated as a result of the reinforcement learning agent interacting with the environment. Each piece of experience data is associated with a respective expected learning progress measure that is a measure of an expected amount of progress made in the training of the neural network if the neural network is trained on the piece of experience data. The method further includes selecting a piece of experience data from the replay memory by prioritizing for selection pieces of experience data having relatively higher expected learning progress measures and training the neural network on the selected piece of experience data.

First claim

What is claimed is:

1. A method for training a neural network used to select actions performed by a reinforcement learning agent interacting with an environment by performing actions that cause the environment to transition states, the method comprising:
maintaining a replay memory, the replay memory storing a plurality of pieces of experience data that each represents information about an interaction of the agent with the environment, wherein each piece of experience data is an experience tuple that comprises a respective current observation characterizing a respective current state of the environment, and a respective current action performed by the agent in response to the respective current observation;
associating, with each piece of experience data in the replay memory, a respective expected learning progress measure, wherein for each of one or more pieces of experience data, the respective expected learning process measure is derived from a result of a preceding time that the piece of experience data was used in training the neural network and is computed based on an error measured at least with respect to a target expected return resulting from the interaction;
selecting a piece of experience data that is a selected experience tuple that comprises a selected current observation characterizing a selected current state of the environment, and a selected current action performed by the agent in response to the selected current observation from the replay memory for use in training the neural network by prioritizing for selection pieces of experience data having relatively higher expected learning progress measures;
training, using a reinforcement learning technique, the neural network on the selected piece of experience data; and
selecting actions in response to observations using the neural network.

2. The method of claim 1 , wherein:
the target expected return resulting from the interaction comprises a target expected total reward that could have been received by the agent following the interaction characterized by the selected piece of experience data; and
training the neural network on the selected piece of experience data comprises determining, with respect to the target expected total reward, an updated error for the selected piece of experience data.

3. The method of claim 2 , further comprising:
determining an updated expected learning progress measure for the selected piece of experience data based on an absolute value of the updated error; and
associating, in the replay memory, the selected piece of experience data with the updated expected learning progress measure.

4. The method of claim 1 , wherein selecting the piece of experience data from the replay memory by prioritizing for selection pieces of experience data having relatively higher expected learning progress measures comprises:
determining, based on the respective expected learning progress measures for the pieces of experience data, a respective probability for each of the pieces of experience data in the replay memory; and
sampling a piece of experience data from the replay memory in accordance with the determined probabilities.

5. The method of claim 4 , wherein determining, based on the respective expected learning progress measures for the pieces of experience data, a respective probability for each of the pieces of experience data in the replay memory comprises:
determining a respective probability for each piece of experience data such that pieces of experience data having higher expected learning progress measures have higher probabilities than pieces of experience data having relatively lower expected learning progress measures.

6. The method of claim 5 , wherein the probability P(i) for a piece of experience data i satisfies:

P
⁡

(
i
)

=

p
i
α

∑
k

⁢

⁢

p
k
α

where α is a predetermined constant, k ranges across the pieces of experience data in the replay memory, and pi is a priority for the piece of experience data i derived from the expected learning progress measure for the piece of experience data i.

7. The method of claim 6 , wherein the priority is the expected learning measure plus a constant value.

8. The method of claim 6 wherein the priority is a fraction having a predetermined positive value as a numerator and a rank of the piece of experience data i in a ranking of the pieces of experience data in the replay memory according to their expected learning progress measures as a denominator.

9. The method of claim 6 , wherein the priority is set to a maximum value for a piece of experience data that has not yet been used in training.

10. The method of claim 1 , wherein each piece of experience data further comprises a respective next state characterizing a respective next state of the environment, and a reward received in response to the agent performing the current action.

11. The method of claim 2 , wherein training the neural network on the selected piece of experience data further comprises:
using the updated error in adjusting values of the parameters of the neural network.

12. The method of claim 11 , wherein using the updated error in adjusting the values of the parameters comprises:
determining a weight for the updated error using the expected learning progress measure for the selected experience tuple;
adjusting the updated error using the weight; and
using the adjusted error as a target error for adjusting the values of the parameters of the neural network.

13. The method of claim 12 , further comprising annealing an exponent used in computing the weight during the training of the neural network.

14. The method of claim 10 , wherein the expected learning progress measure for each experience tuple in the replay memory is a derivative of an absolute value of a temporal difference learning error determined for the experience tuple the preceding time the experience tuple was used in training the neural network.

15. The method of claim 10 , wherein the expected learning progress measure for each experience tuple in the replay memory is a norm of an induced weight-change by using the experience tuple to train the neural network.

16. A system comprising one or more computers and one or more storage devices storing instructions that are operable, when executed by the one or more computers, to cause the one or more computers to perform operations for training a neural network used to select actions performed by a reinforcement learning agent interacting with an environment by performing actions that cause the environment to transition states, the operations comprising:
maintaining a replay memory, the replay memory storing a plurality of pieces of experience data that each represents information about an interaction of the agent with the environment, wherein each piece of experience data is an experience tuple that comprises a respective current observation characterizing a r

Assignees

• Deepmind Tech Ltd

Inventors

• Schaul Tom
• Quan John
• Silver David

Classifications

• Y04S10/50

  Systems or methods supporting the power network operation or management, involving a certain degree of interaction with the load-side end user applications · CPC title

• G06N3/088

  Non-supervised learning, e.g. competitive learning · CPC title

• G06N3/08 Primary

  Learning methods · CPC title

• G06N3/092 Primary

  Reinforcement learning · CPC title

• G06N3/047

  Probabilistic or stochastic networks · CPC title