Interactive reinforcement learning with dynamic reuse of prior knowledge

Patent metadata

Field	Value
Publication number	US-2019236458-A1
Application number	US-201916263930-A
Country	US
Kind code	A1
Filing date	Jan 31, 2019
Priority date	Jan 31, 2018
Publication date	Aug 1, 2019
Grant date	—

Abstract

Systems, methods, and computer readable media directed to interactive reinforcement learning with dynamic reuse of prior knowledge are described in various embodiments. The interactive reinforcement learning is adapted for providing computer implemented systems for dynamic action selection based on confidence levels associated with demonstrator data or portions thereof.

First claim

What is claimed is:

1 . A system for biasing a machine learning architecture using one or more demonstrator data sets, the machine learning architecture for controlling one or more actions conducted by an agent in an environment which transitions between one or more states, the system comprising:
a receiver configured to obtain one or more demonstrator data sets, each demonstrator data set including a data structure representing one or more state-action pairs observed in one or more interactions with the environment; a data storage configured to maintain, for each demonstrator data set or sub-portions thereof, one or more confidence data values, associated with at least one state of the one or more states; a supervised classifier for training using the one or more demonstrator data sets or sub-portions thereof; an action execution processor configured to generate control signals for executing an action associated with an action-source selected from at least one of the one or more demonstrator data sets based on the supervised classifier or an internal policy function maintained by the machine learning architecture, the selecting based at least upon the one or more confidence data values; and a state observer configured to monitor a new state resulting from the execution of the action and an associated reward outcome; and to update the internal policy function maintained by the machine learning architecture based at least on the observed reward outcome.

2 . The system of claim 1 , wherein the state observer is configured to update at least one of the confidence data values of the one or more confidence data values based on the observed reward outcome.

3 . The system of claim 1 , wherein the confidence data values are generated using a dynamic temporal difference confidence measurement based on the relation:
C(s)←(1−F(α))×C(s)+F(α)×[F(r)+γ×C(s′)]

where γ is a discount factor, r is a reward function, and α is an update parameter.

4 . The system of claim 3 , wherein the temporal difference confidence measurement includes a dynamic rate update function based on the relation:

F


(
α
)

=

α
×
max


{

1

Σ
i





exp


(

θ
i
T

·
x

)



[

exp


(

θ
1
T

·
x

)

)

exp


(

θ
2
T

·
x

)

)

…

exp


(

θ
i
T

·
x

)

)

]

}

.

5 . The system of claim 3 , wherein the temporal difference confidence measurement includes a dynamic confidence update function based on the relation:

F


(
r
)

=

r
r_max

×
max


{

1

Σ
i





exp


(

θ
i

Assignees

• Royal Bank Of Canada

Inventors

• Taylor Matthew Edmund
• Wang Zhaodong

Classifications

• G06N3/045

  Combinations of networks · CPC title

• G06N7/01

  Probabilistic graphical models, e.g. probabilistic networks · CPC title

• G06N5/022 Primary

  Knowledge engineering; Knowledge acquisition · CPC title

• G06N3/006 Primary

  based on simulated virtual individual or collective life forms, e.g. social simulations or particle swarm optimisation [PSO] · CPC title

• G06F17/17

  Function evaluation by approximation methods, e.g. inter- or extrapolation, smoothing, least mean square method ({G06F17/18 takes precedence } ; interpolation for numerical control G05B19/18) · CPC title