Method of performing communication load balancing with multi-teacher reinforcement learning, and an apparatus for the same

Patent metadata

Field	Value
Publication number	US-12238190-B2
Application number	US-202318351201-A
Country	US
Kind code	B2
Filing date	Jul 12, 2023
Priority date	Oct 6, 2021
Publication date	Feb 25, 2025
Grant date	Feb 25, 2025

Abstract

A server may be provided to obtain a load balancing artificial intelligence (AI) model for a plurality of base stations in a communication system. The server may obtain teacher models based on traffic data sets collected from the base stations, respectively; perform a policy rehearsal process including obtaining student models based on knowledge distillation from the teacher models, obtaining an ensemble student model by ensembling the student models, and obtaining a policy model by interacting with the ensemble student mode; provide the policy model to each of the base stations for a policy evaluation of the policy model; and based on a training continue signal being received from at least one of the base stations as a result of the policy evaluation, update the ensemble student model and the policy model by performing the policy rehearsal process on the student models.

First claim

What is claimed is:

1. A server for obtaining a load balancing artificial intelligence (AI) model for a plurality of base stations in a communication system, the server comprising:
at least one memory storing instructions; and
at least one processor configured to execute the instructions to:
obtain a plurality of teacher models based on a plurality of traffic data sets collected from the plurality of base stations, respectively;
obtain a plurality of student models based on knowledge distillation from the plurality of teacher models, wherein the plurality of student models is trained based on a knowledge distillation loss that represents a difference between a teacher prediction of the plurality of teacher models and a student prediction of the plurality of student models, and
obtain the load balancing AI model for the plurality of base stations based on the plurality of student models.

2. The server of claim 1 , wherein the at least one processor further configured to execute the instructions to obtain the load balancing AI model for the plurality of base stations, by:
obtaining an ensemble student model by ensembling the plurality of student models; and
transmitting the ensemble student model to the plurality of base stations, respectively.

3. The server of claim 2 , the at least one processor further configured to execute the instructions to:
receive feedback information of the ensemble student model from the plurality of base stations, and
update the ensemble student model based on the received feedback information.

4. The server of claim 1 , wherein the least one processor is further configured to execute the instructions to:
obtain the plurality of teacher models by receiving model parameters of the plurality of teacher models from the plurality of base stations, and updating initialized model parameters of the plurality of teacher models based on the received model parameters.

5. The server of claim 1 , wherein the plurality of traffic data sets comprise state-action-reward trajectories that comprise states, actions, and rewards,
the states comprise at least one of an active user equipment (UE) number, a bandwidth utilization, an internet protocol (IP) throughput, a cell physical resource usage, and a speed of a download link,
the actions comprise a load balancing parameter that causes the states to be changed, and
the rewards comprise at least one of a minimum of IP throughput, a total IP throughput, and a dead cell count.

6. The server of claim 1 , wherein each of the plurality of teacher models comprises a state transition model and a reward transition model that are trained based on state-action-reward trajectories that are collected from the plurality of base stations,
wherein the state transition model is configured to output a predicted next state based on an action taken in a current state, and
wherein the reward transition model is configured to output a predicted reward based on the action taken in the current state.

7. The server of claim 1 , wherein the at least one processor further configured to execute the instructions to obtain the plurality of student models based on knowledge distillation from the plurality of teacher models, by:
computing a ground-truth loss based on a difference between a ground-truth value and a prediction of each of the plurality of student models;
computing the knowledge distillation loss based on the difference between the teacher prediction of the plurality of teacher models and the student prediction of the plurality of student models;
computing an aggregated loss that combines the ground-truth loss and the knowledge distillation loss; and
training the plurality of student models by minimizing or converging the aggregated loss.

8. The server of claim 1 , wherein the least one processor is further configured to execute the instructions to obtain a policy model by:
obtaining state-reward pairs from the plurality of student models;
computing an average of the state-reward pairs;
inputting the average of the state-reward pairs to the policy model to obtain an action as an output of the policy model;
increasing a time step by one;
based on the increased time step being less than a predetermined value, inputting the action to the plurality of student models; and
based on the increased time step being equal to the predetermined value, outputting the policy model.

9. The server of claim 2 , wherein the least one processor is further configured to execute the instructions to:
obtain a policy model by interacting with the ensemble student model;
provide the policy model to each of the plurality of base stations for a policy evaluation of the policy model; and
based on a training continue signal being received from at least one of the plurality of base stations as a result of the policy evaluation, update the ensemble student model and the policy model,
wherein the training continue signal is provided as feedback information of the ensemble student model and indicates that a reward obtained from the ensemble student model is less than a reward obtained from an existing load balancing model by a predetermined margin or more.

10. A method for obtaining a load balancing artificial intelligence (AI) model for a plurality of base stations in a communication system, the method comprising:
obtaining a plurality of teacher models based on a plurality of traffic data sets collected from the plurality of base stations, respectively;
obtaining a plurality of student models based on knowledge distillation from the plurality of teacher models, wherein the plurality of student models is trained based on a knowledge distillation loss that represents a difference between a teacher prediction of the plurality of teacher models and a student prediction of the plurality of student models, and
obtaining the load balancing AI model for the plurality of base stations based on the plurality of student models.

11. The method of claim 10 , wherein the obtaining the load balancing AI model for the plurality of base stations comprises:
obtaining an ensemble student model by ensembling the plurality of student models; and
transmitting the ensemble student model to the plurality of base stations, respectively.

12. The method of claim 11 , further comprising:
receiving feedback information of the ensemble student model from the plurality of base stations, and
updating the ensemble student model based on the received feedback information.

13. The method of claim 10 , further comprising:
obtaining the plurality of teacher models by receiving model parameters of the plurality of teacher models from the plurality of base stations, and updating initialized model parameters of the plurality of teacher models based on the received model parameters.

14. The method of claim 10 , wherein the plurality of traffic data sets comprise state-action-reward trajectories that comprise states, actions, and rewards,
the states comprise at least one of an active user equipment (UE) number, a bandwidth utilization, an internet protocol (IP) throughput, a cell physical resource usage, and a speed of a download link,
the actions comprise a load balancing parameter that causes the states to be changed, and
the rewards comprise at least one of a minimum of IP throughput, a total IP throughput, and a dead cell count.

15. The method of claim 10 , wherein each of the plurality of teacher models comprises a state transition model and a reward transition model that are trained based on state-action-reward trajectories that are collected from the plurality of base stations,
w

Assignees

• Samsung Electronics Co Ltd

Inventors

• Kang Jikun
• Chen Xi
• Hu Chengming
• Wang Ju
• Dudek Gregory Lewis
• Liu Xue

Classifications

• H04L41/16 Primary

  using machine learning or artificial intelligence · CPC title

• H04L67/1004

  Server selection for load balancing · CPC title

• H04W24/02

  Arrangements for optimising operational condition · CPC title

• G06N3/006

  based on simulated virtual individual or collective life forms, e.g. social simulations or particle swarm optimisation [PSO] · CPC title

• G06N3/092

  Reinforcement learning · CPC title