State-augmented reinforcement learning

Patent metadata

Field	Value
Publication number	US-11880765-B2
Application number	US-202017074054-A
Country	US
Kind code	B2
Filing date	Oct 19, 2020
Priority date	Oct 19, 2020
Publication date	Jan 23, 2024
Grant date	Jan 23, 2024

Abstract

A processor training a reinforcement learning model can include receiving a first dataset representing an observable state in reinforcement learning to train a machine to perform an action. The processor receives a second dataset. Using the second dataset, the processor trains a machine learning classifier to make a prediction about an entity related to the action. The processor extracts an embedding from the trained machine learning classifier, and augments the observable state with the embedding to create an augmented state. Based on the augmented state, the processor trains a reinforcement learning model to learn a policy for performing the action, the policy including a mapping from state space to action space.

First claim

What is claimed is:

1. A computer-implemented method comprising:
receiving a first dataset representing an observable state in reinforcement learning to train a machine to perform an action;
receiving a second dataset;
training a machine learning classifier using the second dataset to make a prediction about an entity related to the action;
extracting an embedding from the trained machine learning classifier;
augmenting the observable state with the embedding to create an augmented state; and
based on the augmented state, training a reinforcement learning model to learn a policy for performing the action, the policy including a mapping from state space to action space,
wherein the embedding includes a vector representation of information in heterogeneous form converted by the machine learning classifier into a usable form by the reinforcement learning model.

2. The method of claim 1 , wherein a plurality of second datasets are received and a corresponding plurality of different machine learning classifiers are trained to make the prediction about the entity related to the action.

3. The method of claim 1 , wherein the second dataset includes unstructured data and a natural language processing creates word embeddings for training the machine learning classifier.

4. The method of claim 1 , wherein the reinforcement learning model includes deep neural networks.

5. The method of claim 1 , wherein the extracting the embedding from the trained machine learning classifier includes extracting features in the last layer before the softmax layer in the machine learning classifier to represent the embedding.

6. The method of claim 1 , wherein the machine learning classifier includes a neural network.

7. The method of claim 1 , wherein the prediction about an entity includes asset movement and the action includes asset allocation in portfolio management.

8. A system comprising:
a hardware processor; and
a memory device coupled with the hardware processor;
the hardware processor configured to at least:
receive a first dataset representing an observable state in reinforcement learning to train a machine to perform an action;
receive a second dataset;

train a machine learning classifier using the second dataset to make a prediction about an entity related to the action;
extract an embedding from the trained machine learning classifier;
augment the observable state with the embedding to create an augmented state; and

based on the augmented state, train a reinforcement learning model to learn a policy for performing the action, the policy including a mapping from state space to action space,
wherein the embedding includes a vector representation of information in heterogeneous form converted by the machine learning classifier into a usable form by the reinforcement learning model.

9. The system of claim 8 , wherein a plurality of second datasets are received and a corresponding plurality of different machine learning classifiers are trained to make the prediction about the entity related to the action.

10. The system of claim 8 , wherein the second dataset includes unstructured data and a natural language processing creates word embeddings for training the machine learning classifier.

11. The system of claim 8 , wherein the reinforcement learning model includes deep neural networks.

12. The system of claim 8 , wherein the hardware processor is configured to extract features in the last layer before the softmax layer in the machine learning classifier to represent the embedding.

13. The system of claim 8 , wherein the machine learning classifier includes a neural network.

14. The system of claim 8 , wherein the prediction about an entity includes asset movement and the action includes asset allocation in portfolio management.

15. A computer program product comprising a computer readable storage medium having program instructions embodied therewith, the program instructions readable by a device to cause the device to:
receive a first dataset representing an observable state in reinforcement learning to train a machine to perform an action;
receive a second dataset;
train a machine learning classifier using the second dataset to make a prediction about an entity related to the action;
extract an embedding from the trained machine learning classifier;
augment the observable state with the embedding to create an augmented state; and
based on the augmented state, train a reinforcement learning model to learn a policy for performing the action, the policy including a mapping from state space to action space,
wherein the embedding includes a vector representation of information in heterogeneous form converted by the machine learning classifier into a usable form by the reinforcement learning model.

16. The computer program product of claim 15 , wherein a plurality of second datasets are received and a corresponding plurality of different machine learning classifiers are trained to make the prediction about the entity related to the action.

17. The computer program product of claim 15 , wherein the second dataset includes unstructured data and a natural language processing creates word embeddings for training the machine learning classifier.

18. The computer program product of claim 15 , wherein the reinforcement learning model includes deep neural networks.

19. The computer program product of claim 15 , wherein the device is caused to extract features in the last layer before the softmax layer in the machine learning classifier to represent the embedding.

20. The computer program product of claim 15 , wherein the prediction about an entity includes asset movement and the action includes asset allocation in portfolio management.

Assignees

• IBM
• Univ Of Illinois At Urbana Champaign

Inventors

• Chen Pin-Yu
• Zhu Yada
• Xiong Jinjun
• Bhaskaran Kumar
• Ye Yunan
• Li Bo

Classifications

• G06N3/0442

  characterised by memory or gating, e.g. long short-term memory [LSTM] or gated recurrent units [GRU] · CPC title

• G06N3/09

  Supervised learning · CPC title

• G06N3/092

  Reinforcement learning · CPC title

• G06N3/08 Primary

  Learning methods · CPC title

• G06F40/279

  Recognition of textual entities · CPC title