Edge network computing system with deep reinforcement learning based task scheduling

Patent metadata

Field	Value
Publication number	US-12223336-B2
Application number	US-202117490861-A
Country	US
Kind code	B2
Filing date	Sep 30, 2021
Priority date	Sep 30, 2021
Publication date	Feb 11, 2025
Grant date	Feb 11, 2025

Abstract

An edge network computing system includes: a plurality of terminal devices; a plurality of edge servers connected to the terminal device through an access network; and a plurality of cloud servers connected to the plurality of edge servers through a core network. Each edge server is configured to: receive a plurality of computing tasks originated from one of the plurality of terminal devices; use a deep Q-learning neural network (DQN) with experience replay to select one of the plurality of could servers to offload a portion of the plurality of computing tasks; and send the portion of the plurality of computing tasks to the selected cloud server and forward results of the portion of the plurality of computing tasks received from the selected cloud server to the originating terminal device.

First claim

What is claimed is:

1. An edge network computing system, comprising:
a plurality of terminal devices;
a plurality of edge servers comprising at least one processor connected to the plurality of terminal devices through an access network; and
a plurality of cloud servers connected to the plurality of edge servers through a core network,
wherein each edge server is configured to:
receive a plurality of computing tasks originated from one of the plurality of terminal devices;
use a deep Q-learning neural network (DQN) with experience replay to select one of the plurality of cloud servers to offload a portion of the plurality of computing tasks, such that each edge server selects an offloading cloud server without being correlated to selection of the offloading cloud server by other edge servers;
send the portion of the plurality of computing tasks to the selected cloud server; and
execute the portion of the plurality of computing tasks and forward results of the portion of the plurality of computing tasks received from the selected cloud server to the originating terminal device;

wherein:
a first time is a time for completing a remaining portion of the plurality of computing tasks performed by one of the plurality of edge servers;
a second time is a time for offloading and completing the portion of the plurality of computing tasks offloaded to the selected cloud server, and includes an uplink access latency, a data transmission time for transmitting the portion of the plurality of computing tasks from one of the plurality of edge servers to the selected cloud server, a computing time for completing the portion of the plurality of computing tasks by the selected cloud server, and a downlink access latency;
when the plurality of computing tasks are independent of each other, a total time for completing the plurality of computing tasks is the greater of the first time and the second time, and an optimal task partitioning is achieved when the first time is equal to the second time;
when the plurality of computing tasks are required to be performed sequentially, the total time for completing the plurality of computing tasks includes the greater of the first time and a sum of the uplink access latency and the data transmission time, the computing time, and the downlink access latency, and the optimal task partitioning is achieved when the first time is equal to the sum of the uplink access latency and the data transmission time;
each edge server includes an agent of multi-agent reinforcement learning in a Markov decision process (MDP), and the agent learns from an environment, and takes action to maximize a long-term accumulative reward;
a state space includes the uplink access latency, the downlink access latency, and a data rate of a backhaul link between the edge server and the cloud server, observed at a current time slot;
an action space includes the plurality of cloud servers as candidates for task offloading, and a task partitioning ratio having a number of discrete values; and
a reward includes minus the total time for completing the plurality of computing tasks.

2. The system according to claim 1 , wherein when using the deep Q-learning neural network (DQN) with experience replay to select the cloud server to offload the portion of the plurality of computing tasks to the cloud server, the edge server is configured to:
each time after receiving a plurality of computing tasks, use the DQN to determine a cloud server and a task partition ratio to offload some of the plurality of computing tasks to the cloud server;
store each experience including a current state, a current action, a current reward, and a next state in a replay memory;
randomly select a minibatch of past experiences from the replay memory to input to the DQN and a target network;
optimize the DQN with updated weights based on Q-values from the DQN and Q-values from the target network; and
periodically update weights of the target network with the weights of the DQN.

3. The system according to claim 2 , wherein:
the DQN includes an input layer of neurons representing the state space and the action space, and an output layer of Q-values for the state space and the action space at the input layer;
training the DQN includes mapping inputs of state-action pairs in the minibatch of past experiences to their corresponding Q-values by approximating Q-values and minimizing a loss function of mean square error between the Q-values outputted by the DON and the Q-values outputted by the target network; and
the target network includes a structure same as the DQN and weights periodically updated by the weights of the DQN obtained from training.

4. The system according to claim 2 , wherein:
each experience stored in the replay memory includes an importance weight to naturally decay obsolete data.

5. The system according to claim 1 , wherein:
the edge network computing system is a mobile edge computing system;
the plurality of terminal devices are mobile devices; and
the plurality of mobile devices are connected to the plurality of edge servers through a radio access network.

6. A task offloading method for an edge server comprising at least one processor in an edge network computing system, comprising:
receiving a plurality of computing tasks from a terminal device;
using a deep Q-learning neural network (DQN) with experience replay to select a cloud server to offload a portion of the plurality of computing tasks;
sending the portion of the plurality of computing tasks to the cloud server; and
executing the portion of the plurality of computing tasks and forwarding results of the portion of the plurality of computing tasks received from the cloud server to the terminal device;
wherein:
a first time is a time for completing a remaining portion of the plurality of computing tasks performed by one of the plurality of edge servers;
a second time is a time for offloading and completing the portion of the plurality of computing tasks offloaded to the selected cloud server, and includes an uplink access latency, a data transmission time for transmitting the portion of the plurality of computing tasks from one of the plurality of edge servers to the selected cloud server, a computing time for completing the portion of the plurality of computing tasks by the selected cloud server, and a downlink access latency;
when the plurality of computing tasks are independent of each other, a total time for completing the plurality of computing tasks is the greater of the first time and the second time, and an optimal task partitioning is achieved when the first time is equal to the second time;
when the plurality of computing tasks are required to be performed sequentially, the total time for completing the plurality of computing tasks includes the greater of the first time and a sum of the uplink access latency and the data transmission time, the computing time, and the downlink access latency, and the optimal task partitioning is achieved when the first time is equal to the sum of the uplink access latency and the data transmission time;
each edge server includes an agent of multi-agent reinforcement learning in a Markov decision process (MDP), and the agent learns from an environment, and takes action to maximize a long-term accumulative reward;
a state space includes the uplink access latency, the downlink access latency, and a data rate of a backhaul link between the edge server and the cloud server, observed at a current time slot;
an action space includes the plurality of cloud servers as candidates for task offloading, and a task partitioning ratio having a number of discrete values; and
a reward includes minus the total time for completing the plurality of computing tasks.

7. The meth

Assignees

• Intelligent Fusion Tech Inc

Inventors

• Zhao Qi
• Li Yi
• Feng Mingjie
• Li Li
• Chen Genshe

Classifications

• G06F9/4806

  Task transfer initiation or dispatching · CPC title

• G06F2209/509

  Offload · CPC title

• G06N3/045

  Combinations of networks · CPC title

• G06N7/01

  Probabilistic graphical models, e.g. probabilistic networks · CPC title

• G06N3/092

  Reinforcement learning · CPC title