Learning method and program

Patent metadata

Field	Value
Publication number	US-11449801-B2
Application number	US-202016812497-A
Country	US
Kind code	B2
Filing date	Mar 9, 2020
Priority date	Jun 14, 2019
Publication date	Sep 20, 2022
Grant date	Sep 20, 2022

Abstract

According to one embodiment, a learning method, comprises receiving a first signal including a previous auxiliary variable value, previous action information regarding a previous action, or a set of previous scores, receiving current sensor data, selecting a current action of the control target based on the first signal, the current sensor data, and a parameter for obtaining a score from sensor data, causing the control target to execute the current action, receiving next sensor data and a reward, and updating the parameter based on the current sensor data, current action information regarding the current action, the next sensor data, and the reward. A degree of selecting a previous action as the current action is increased.

First claim

What is claimed is:

1. A learning method, comprising:
first receiving a first signal including a previous auxiliary variable value, previous action information regarding a previous action of a control target, or a set of previous scores;
second receiving current sensor data;
selecting a current action of the control target based on the first signal, the current sensor data, and a value of a parameter for obtaining a score from sensor data;
causing the control target to execute the current action;
third receiving next sensor data and a reward; and
updating a value of the parameter based on the current sensor data, current action information regarding the current action, the next sensor data, and the reward,
wherein the selecting comprises increasing a degree of selecting a previous action as the current action.

2. The learning method of claim 1 , wherein the first receiving, the second receiving, the selecting, the causing, the third receiving, and the updating are executed every control period of the control target.

3. The learning method of claim 1 , wherein the selecting comprises:
first calculating a set of current scores based on the current sensor data and the value of the parameter before update;
second calculating a current auxiliary variable value based on the previous auxiliary variable value; and
third selecting the current action based on the set of current scores and the current auxiliary variable, and wherein the second calculating comprises:
setting the previous auxiliary variable value as the current auxiliary variable value.

4. The learning method of claim 1 , wherein the selecting comprises:
first calculating a set of current scores based on the current sensor data and the value of the parameter before update;
second calculating a set of mixed scores based on the set of current scores and the previous action information;
third calculating a current auxiliary variable value based on the previous auxiliary variable value; and
fourth selecting a current action based on the set of mixed scores and the current auxiliary variable value, and wherein the second calculating comprises calculating the set of mixed scores by mixing the set of current scores and a set of scores in which a score for a same action as the previous action is larger than scores for actions other than the previous action.

5. The learning method of claim 1 , wherein the selecting comprises:
first calculating a set of current scores based on the current sensor data and the value of the parameter;
second calculating a set of mixed scores from the set of current scores and the set of previous scores;
third calculating a current auxiliary variable value based on the previous auxiliary variable value; and
fourth selecting a current action based on the set of mixed scores and the current auxiliary variable value, and wherein the second calculating comprises:
calculating the set of mixed scores by mixing the set of previous scores and the set of current scores.

6. The learning method of claim 1 , wherein the selecting comprising:
increasing a degree of selecting the previous action as the current action during a period required from start of execution of the current action to completion of execution of the current action by the control target.

7. The learning method of claim 1 , wherein the control target comprises an automobile, and wherein the selecting comprises:
increasing a degree of selecting the previous action as the current action during a period required from start of lane change to completion of the lane change by the automobile.

8. The learning method of claim 1 , wherein the control target comprises an automobile, and wherein the selecting comprises:
increasing a degree of selecting the previous action as the current action during a period required from start of speed change to completion of the speed change by the automobile.

9. A non-transitory computer-readable storage medium having stored thereon a computer program that is executable by a computer, the computer program comprising instructions capable of causing the computer to execute functions of:
first receiving a first signal including a previous auxiliary variable value, previous action information regarding a previous action of a control target, or a set of previous scores;
second receiving current sensor data;
selecting a current action of the control target based on the first signal, the current sensor data, and a value of a parameter for obtaining a score from sensor data;
causing the control target to execute the current action;
third receiving next sensor data and a reward; and
updating a value of the parameter based on the current sensor data, current action information regarding the current action, the next sensor data, and the reward,
wherein the selecting comprises increasing a degree of selecting a previous action as the current action.

10. The storage medium of claim 9 , wherein the first receiving, the second receiving, the selecting, the causing, the third receiving, and the updating are executed every control period of the control target.

11. The storage medium of claim 9 , wherein the selecting comprises:
first calculating a set of current scores based on the current sensor data and the value of the parameter before update;
second calculating a current auxiliary variable value based on the previous auxiliary variable value; and
third selecting the current action based on the set of current scores and the current auxiliary variable, and wherein the second calculating comprises:
setting the previous auxiliary variable value as the current auxiliary variable value.

12. The storage medium of claim 9 , wherein the selecting comprises:
first calculating a set of current scores based on the current sensor data and the value of the parameter before update;
second calculating a set of mixed scores based on the set of current scores and the previous action information;
third calculating a current auxiliary variable value based on the previous auxiliary variable value; and
fourth selecting a current action based on the set of mixed scores and the current auxiliary variable value, and wherein the second calculating comprises calculating the set of mixed scores by mixing the set of current scores and a set of scores in which a score for a same action as the previous action is larger than scores for actions other than the previous action.

13. The storage medium of claim 9 , wherein the selecting comprises:
first calculating a set of current scores based on the current sensor data and the value of the parameter;
second calculating a set of mixed scores from the set of current scores and the set of previous scores;
third calculating a current auxiliary variable value based on the previous auxiliary variable value; and
fourth selecting a current action based on the set of mixed scores and the current auxiliary variable value, and wherein the second calculating comprises:
calculating the set of mixed scores by mixing the set of previous scores and the set of current scores.

14. The storage medium of claim 9 , wherein the selecting comprising:
increasing a degree of selecting the previous action as the current action during a period required from start of execution of the current action to completion of execution of the current action by the control target.

15. The storage medium of claim 9 , wherein the control target comprises an automobile, and wherein the selecting comprises:
increasing a degree of selecting the previous action as the current action during a period required from start of lane change to

Assignees

• Toshiba Kk

Inventors

• Nonaka Ryosuke
• Kaneko Toshimitsu
• Nakamura Norihiro

Classifications

• G06N3/044

  Recurrent networks, e.g. Hopfield networks · CPC title

• G06N3/045

  Combinations of networks · CPC title

• G06N3/092

  Reinforcement learning · CPC title

• G06N3/0464

  Convolutional networks [CNN, ConvNet] · CPC title

• G06N3/006

  based on simulated virtual individual or collective life forms, e.g. social simulations or particle swarm optimisation [PSO] · CPC title