Neural network for reinforcement learning

Patent metadata

Field	Value
Publication number	US-9349092-B2
Application number	US-201414293928-A
Country	US
Kind code	B2
Filing date	Jun 2, 2014
Priority date	Dec 3, 2012
Publication date	May 24, 2016
Grant date	May 24, 2016

Abstract

A neural model for reinforcement-learning and for action-selection includes a plurality of channels, a population of input neurons in each of the channels, a population of output neurons in each of the channels, each population of input neurons in each of the channels coupled to each population of output neurons in each of the channels, and a population of reward neurons in each of the channels. Each channel of a population of reward neurons receives input from an environmental input, and is coupled only to output neurons in a channel that the reward neuron is part of. If the environmental input for a channel is positive, the corresponding channel of a population of output neurons are rewarded and have their responses reinforced, otherwise the corresponding channel of a population of output neurons are punished and have their responses attenuated.

First claim

What is claimed is:

1. A neural network for reinforcement-learning and for action-selection comprising:
a plurality of channels;
a population of input neurons in each of the channels;
a population of output neurons in each of the channels, each population of input neurons in each of the channels coupled to each population of output neurons in each of the channels by first synapses; and
a population of reward neurons in each of the channels, wherein each population of reward neurons receives input from an environmental input, and wherein each channel of reward neurons is coupled only to output neurons in a channel that the reward neuron is part of by second synapses;
wherein if the environmental input for a channel is positive, the corresponding channel of a population of output neurons are rewarded and have their responses reinforced;
wherein if the environmental input for a channel is negative, the corresponding channel of a population of output neurons are punished and have their responses attenuated; and
wherein the neural network comprises memristors.

2. The neural network of claim 1 wherein the first synapses and the second synapses have a spike-timing dependent plasticity wherein
g syn =g max ·g eff ·( V−E syn )

where
gmax is a maximum conductance of the first and second synapses,
geff is a current synaptic efficacy between 0 and a maximum value of geffmax,
Esyn is a reversal potential for the first and second synapses,
V is a voltage, and
gsyn is a synapse conductance.

3. The neural network of claim 2 wherein
g eff →g eff +g effmax F (Δ t )

where

Δ
⁢

⁢
t

=

t
pre

-

t
post

F
⁡

(

Δ
⁢

⁢
t

)

=

{

A
+

⁢

ⅇ

(

Δ
⁢

⁢
t

τ
+

)

A
-

⁢

ⅇ

(

Δ
⁢

⁢
t

τ
-

)

⁢

⁢

if
⁢

(

g
eff

<
0

)

⁢

⁢
then
⁢

⁢

g
eff

→

0
⁢

⁢

if
⁢

(

geff
>
geffmax

)

⁢

⁢
then
⁢

⁢
geff

→

geffmax
.

4. The neural network of claim 1 wherein each population of input neurons, each population of output neurons, and each population of reward neurons comprise a Leaky-Integrate and Fire (LIF) device wherein

C
m

⁢

ⅆ
V

ⅆ
t

=

-

g
leak

⁡

(

V

Assignees

• Hrl Lab Llc
• Hrl Lab Llc

Inventors

• Thibeault Corey M
• Srinivasa Narayan

Classifications

• G06N3/092 Primary

  Reinforcement learning · CPC title

• G06N3/049

  Temporal neural networks, e.g. delay elements, oscillating neurons or pulsed inputs · CPC title

• G06N20/00

  Machine learning · CPC title

• G06N3/0499

  Feedforward networks · CPC title

• G06N3/08 Primary

  Learning methods · CPC title