Exploration method based on reward decomposition in multi-agent reinforcement learning

Patent metadata

Field	Value
Publication number	US-2024256885-A1
Application number	US-202318517931-A
Country	US
Kind code	A1
Filing date	Nov 22, 2023
Priority date	Jan 26, 2023
Publication date	Aug 1, 2024
Grant date	—

Abstract

Provided is an exploration method based on reward decomposition in multi-agent reinforcement learning. The exploration method includes: generating a positive reward estimation model through neural network training based on training data including states of all agents, actions of all the agents, and a global reward true value; generating, for each of the agents, a first individual utility function based on the global reward true value and generating a second individual utility function using the positive reward estimation model; and determining an action of each of the agents using the first individual utility function and the second individual utility function based on the state of each of the agents.

First claim

What is claimed is:

1 . An exploration method based on reward decomposition in multi-agent reinforcement learning, the exploration method comprising:
generating a positive reward estimation model through neural network training based on training data including states of all agents, actions of all the agents, and a global reward true value; generating, for each of the agents, a first individual utility function based on the global reward true value and generating a second individual utility function using the positive reward estimation model; and determining an action of each of the agents using the first individual utility function and the second individual utility function based on the state of each of the agents.

2 . The exploration method of claim 1 , wherein the generating of the positive reward estimation model includes:
inputting the state of the agent included in the training data into an encoding neural network to generate a state encoding vector, and inputting the action of the agent included in the training data into the encoding neural network to generate an action encoding vector; inputting the state encoding vector and the action encoding vector into a global reward neural network to generate a global reward estimation value; and training a positive local reward neural network included in the global reward neural network using a loss function based on the global reward estimation value and the global reward true value, to generate the positive reward estimation value.

3 . The exploration method of claim 2 , wherein the generating of the positive reward estimation model includes:
inputting the global reward estimation value and the global reward true value into the loss function; and training the positive local reward neural network such that a function value of the loss function is minimized, to generate the positive reward estimation model.

4 . The exploration method of claim 1 , wherein the determining of the action includes:
selecting any one of the first individual utility function and the second individual utility function according to a predetermined criterion; and selecting any one of a random action and an action that maximizes a value of the selected individual utility function according to a predetermined criterion based on the state of each of the agents.

5 . The exploration method of claim 4 , wherein the selecting of any one of the first individual utility function and the second individual utility function includes selecting any one of the first individual utility function and the second individual utility function according to a preset probability,
wherein a probability of selecting the first individual utility function is set to 1-ζ, and a probability of selecting the second individual utility function is set to ζ, and the probability of selecting the second individual utility function is initially set to 1 and converges to 0 as exploration progresses.

6 . The exploration method of claim 4 , wherein the determining of the action includes selecting any one of the random action and the action that maximizes the value of the selected individual utility function according to a preset action selection probability,
wherein a probability of selecting the random action is set to ε, and a probability of selecting the action that maximizes the value of the selected individual utility function is set to 1-ε, and the probability of selecting the random action is initially set to 1 and converges to 0 as exploration progresses.

7 . A computer system comprising:
a memory in which instructions readable by a computer are stored; and at least one processor implemented to execute the instructions, wherein the at least one processor is configured to execute the instructions to: generate a positive reward estimation model through neural network training based on training data including states of all agents, actions of all the agents, and a global reward true value; generate, for each of the agents, a first individual utility function based on the global reward true value and generating a second individual utility function using the positive reward estimation model; and determine an action of each of the agents using the first individual utility function and the second individual utility function based on the state of each of the agents.

8 . The computer system of claim 7 , wherein the at least one processor is configured to:
input the state of the agent included in the training data into an encoding neural network to generate a state encoding vector, and input the action of the agent included in the training data into the encoding neural network to generate an action encoding vector; input the state encoding vector and the action encoding vector into a global reward neural network to generate a global reward estimation value; and train a positive local reward neural network included in the global reward neural network using a loss function based on the global reward estimation value and the global reward true value, to generate the positive reward estimation value.

9 . The computer system of claim 8 , wherein the at least one processor is configured to:
input the global reward estimation value and the global reward true value into the loss function; and train the positive local reward neural network such that a function value of the loss function is minimized, to generate the positive reward estimation model.

10 . The computer system of claim 7 , wherein the at least one processor is configured to:
select any one of the first individual utility function and the second individual utility function according to a predetermined criterion; and select any one of a random action and an action that maximizes a value of the selected individual utility function according to a predetermined criterion based on the state of each of the agents.

11 . The computer system of claim 10 , wherein the at least one processor is configured to select any one of the first individual utility function and the second individual utility function according to a preset probability,
wherein a probability of selecting the first individual utility function is set to 1-ζ, and a probability of selecting the second individual utility function is set to ζ, and the probability of selecting the second individual utility function is set to 1 and converges to 0 as exploration progresses.

12 . The computer system of claim 10 , wherein the at least one processor is configured to select any one of the random action and the action that maximizes the value of the selected individual utility function according to a preset action selection probability,
wherein a probability of selecting the random action is set to ε, and a probability of selecting the action that maximizes the value of the selected individual utility function is set to 1-ε, and the probability of selecting the random action is initially set to 1 and converges to 0 as exploration progresses.

Assignees

• Electronics & Telecommunications Res Inst

Inventors

• Yoo Byunghyun
• Kim Hyun Woo
• Song Hwajeon
• Yang Jeongmin
• Yi Sungwon
• Chung Euisok
• Han Ran

Classifications

• G06N7/01

  Probabilistic graphical models, e.g. probabilistic networks · CPC title

• G06N3/092 Primary

  Reinforcement learning · CPC title

• G06N3/047

  Probabilistic or stochastic networks · CPC title

• G06N3/045

  Combinations of networks · CPC title

• G06N3/08

  Learning methods · CPC title