Spiking neuron network adaptive control apparatus and methods

Patent metadata

Field	Value
Publication number	US-9367798-B2
Application number	US-201213623842-A
Country	US
Kind code	B2
Filing date	Sep 20, 2012
Priority date	Sep 20, 2012
Publication date	Jun 14, 2016
Grant date	Jun 14, 2016

Abstract

Adaptive controller apparatus of a plant may be implemented. The controller may comprise an encoder block and a control block. The encoder may utilize basis function kernel expansion technique to encode an arbitrary combination of inputs into spike output. The controller may comprise spiking neuron network operable according to reinforcement learning process. The network may receive the encoder output via a plurality of plastic connections. The process may be configured to adaptively modify connection weights in order to maximize process performance, associated with a target outcome. The relevant features of the input may be identified and used for enabling the controlled plant to achieve the target outcome.

First claim

What is claimed:

1. An adaptive controller apparatus, the apparatus comprising:
one or more computer-readable storage media configured to store an encoder block, the encoder block comprising a continuous-to-spiking expansion kernel, the expansion kernel comprising a plurality of spiking neurons, individual ones of the plurality of spiking neurons including a plurality of receptive fields associated therewith, the encoder block being configured to encode a continuous input signal into a spiking output signal using the expansion kernel; and
a spiking neuron network configured to receive the spiking output from the plurality of spiking neurons via one or more connections, the spiking neuron network being further configured to generate a control signal using a reinforcement learning process and an external signal;
wherein:
individual ones of the plurality of receptive fields are characterized by an input range associated with the continuous input signal;
individual ones of the plurality of spiking neurons are configured to generate one or more spikes based on at least a portion of the continuous input signal corresponding to respective individual ones of the plurality of receptive fields;
the spiking output signal comprises the one or more spikes; and
the spiking neuron network is characterized by a first plasticity mechanism modulated by the external signal.

2. The apparatus of claim 1 , wherein:
individual ones of the one or more connections are characterized by a connection efficacy; and
the spiking neuron network is characterized by a second plasticity mechanism configured to adjust the connection efficacy in accordance with the external signal.

3. The apparatus of claim 2 , wherein:
the adjustment of the connection efficacy comprises one or both of a connection potentiation or a connection depression;
the potentiation comprises an increase of the connection efficacy; and
the depression comprises a reduction of the connection efficacy.

4. The apparatus of claim 3 , wherein:
the increase of the connection efficacy is characterized by a first time-dependent function having a time window associated therewith;
the one or more spikes are characterized by a pre-synaptic time;
the external signal comprises a reinforcement signal, the reinforcement signal comprising one or more spikes characterized by a reinforcement spike time;
the time window is selected based on at least one of the pre-synaptic time and the reinforcement spike time; and
integration of the time-dependent function over the time window generates a positive value.

5. The apparatus of claim 2 , wherein:
the reinforcement learning process is configured to produce a target outcome; and
the external signal comprises a reinforcement indication configured based on whether a current outcome associated with the control signal is within a pre-determined range from the target outcome.

6. The apparatus of claim 2 , wherein:
the reinforcement learning process is characterized by a performance measure comprising a distance between a target outcome and current outcome, associated with a current state;
a current value of the performance measure is based on the adjustment of the connection efficacy configured to cause a transition of the process from a prior state to the current state;
the reinforcement is positive when the current outcome is closer to the target outcome as characterized by a smaller value of the distance; and
the reinforcement is negative when the current outcome is farther from the target outcome as characterized by a greater value of the distance.

7. The apparatus of claim 1 , wherein the continuous input signal includes one or more of an analog signal, a polyadic signal with arity greater than 2, an n-bit long discrete signal with n-bits greater than 2, or a real-valued signal.

8. The apparatus of claim 1 , wherein the spike output signal comprises a bit stream.

9. The apparatus of claim 1 , wherein:
the spiking network comprises one or more spiking neurons;
individual ones of the plurality of spikes are configured to cause individual ones of the one or more spiking neurons to generate one or more spiking responses; and
the control signal comprises a combination of individual ones of the one or more spiking responses.

10. The apparatus of claim 1 , wherein:
individual ones of the plurality of receptive fields are effectuated using individual ones of a plurality of band-pass filters;
the continuous input signal is characterized by an input signal extent comprising a range of values of the input signal; and
individual ones of the plurality of band-pass filters are characterized by a pass band in the input signal extent.

11. The apparatus of claim 10 , wherein individual ones of the plurality of band-pass filters comprise a gain and a temporal component, the temporal component being configured to modify one or both of (i) the gain with time or (ii) the pass band with time.

12. The apparatus of claim 11 , wherein the temporal component is obtained using a time derivative of the input signal.

13. The apparatus of claim 12 , wherein:
the external signal comprises a reinforcement spike; and
the temporal component is obtained using a time integration of the input signal over a time window, the time window preceding a time of the reinforcement spike.

14. The apparatus of claim 1 , wherein:
the input signal comprises two or more dimensions, individual ones of the two or more dimensions being configured to describe an input property;
individual ones of the plurality of receptive fields are characterized by a dimensional space configured to match the two or more dimensions; and
at least the portion of the continuous input signal comprises two or more properties, the two or more properties including the input property.

15. The apparatus of claim 1 , wherein the plurality of receptive fields comprise an overlapping set of receptive fields.

16. The apparatus of claim 1 , wherein the plurality of receptive fields comprise a non-overlapping set of receptive fields so that individual ones of the plurality of receptive fields cover non-overlapping portions of the input signal extent.

17. The apparatus of claim 1 , wherein:
the continuous input signal is characterized by an input extent comprising a range of values of the continuous input signal; and
individual ones of the plurality of receptive fields are characterized by a unimodal concave function of at least a portion of the range.

18. The apparatus of claim 1 , wherein:
the unimodal concave function comprises a combination of two or more of (i) a Gaussian distribution, (ii) a sigmoid distribution, or (iii) a linear distribution.

19. A non-transitory computer-readable storage medium having instructions embodied thereon, the instructions being executable by one or more processors to perform a method of implementing reinforcement learning, the method comprising:
receiving an non-spiking input signal characterized by an input range of values;
transforming the input signal into one or more spiking signals using one or more filters;
operating one or more stochastic spiking neurons in accordance with a stochastic reinforcement learning process; and
combining the one or more spiking signals into an output using the one or more stochastic spiking neurons and a reinforcement signal.

20. An adaptive controller apparatus, the apparatus comprising:
one or more processors configured to execute computer program modules, the computer program modules being executable t

Assignees

• Brain Corp

Inventors

• Coenen Olivier
• Sinyavskiy Oleg

Classifications

• G06N3/092 Primary

  Reinforcement learning · CPC title

• G05B13/027

  using neural networks only · CPC title

• G06N3/049

  Temporal neural networks, e.g. delay elements, oscillating neurons or pulsed inputs · CPC title

• G06N3/082

  modifying the architecture, e.g. adding, deleting or silencing nodes or connections · CPC title

• G06N3/0495

  Quantised networks; Sparse networks; Compressed networks · CPC title