Reinforcement learning-based intelligent control of packet transmissions within ad-hoc networks

Patent metadata

Field	Value
Publication number	US-11146479-B2
Application number	US-201916598841-A
Country	US
Kind code	B2
Filing date	Oct 10, 2019
Priority date	Oct 10, 2019
Publication date	Oct 12, 2021
Grant date	Oct 12, 2021

Abstract

At a source node, a plurality of packets may be determined for transmission to a destination node in a network comprising a plurality of network nodes. A transmission rate of the plurality of packets from the source node to a neighbor node in the network may be adaptively controlled, based on a determination of a current status of the network by utilizing a plurality of parameters that are estimated via a reinforcement learning routing algorithm. The plurality of parameters include an estimated cost value representing a current cost to transmit the plurality of packets to the destination node via the network. Transmissions from intermediate nodes may also be adaptively deferred based on a determination of a current status of the network by utilizing the plurality of parameters.

First claim

What is claimed is:

1. A method for an ad-hoc network having a plurality of nodes including a source node and a destination node, the method comprising, at each intermediate one of the nodes besides the destination node:
measuring a current position and a current velocity of the intermediate node;
maintaining an estimated position and an estimated velocity for each of the nodes in a neighboring subset of the nodes;
broadcasting at least one transmitted packet including the current position and the current velocity of the intermediate node;
correspondingly receiving at least one received packet from at least a proximate one of the nodes, the received packet including a current position and a current velocity of the proximate node;
updating the estimated position and the estimated velocity of the proximate node with the current position and the current velocity from the received packet;
calculating a hop duration of a respective reliable link between the intermediate node and each of at least a neighbor one of the nodes in the neighboring subset, including calculating the hop duration from the current position and the current velocity measured for the intermediate node and from the estimated position and the estimated velocity maintained for the neighbor node;
determining, at the intermediate node that is the source node, a payload packet for transmission from the source node to the destination node via the ad-hoc network; and
adaptively controlling, at the intermediate node that is the source node, a transmission rate of the payload packet from the source node to the neighbor node of the source node in the ad-hoc network, based on an estimated V-value of the neighbor node, a change in a self V-value of the intermediate node, and the hop duration of the respective reliable link between the source node and the neighbor node, wherein the self V-value and the estimated V-value are determined from a reinforcement learning routing algorithm, the self V-value converging to a current cost to transmit the payload packet from the source node to the destination node via the ad-hoc network, and the estimated V-value converging to a current cost to transmit the payload packet from the neighbor node to the destination node via the ad-hoc network,
wherein the broadcasting includes broadcasting the transmitted packet having a header including a path duration of a network path and a confidence of the network path from the intermediate node to the destination node via the ad-hoc network, wherein the path duration includes a start time and an end time inside the hop duration of the respective reliable link between the intermediate node and the neighbor node, which is a next one of the nodes along the network path, and
the correspondingly receiving includes from the proximate node that is the neighbor node correspondingly receiving the received packet having the header including a path duration of a portion the network path following the neighbor node and a confidence of the portion from the neighbor node to the destination node via the ad-hoc network, wherein the path duration received includes a start time and an end time.

2. The method of claim 1 , wherein:
the determining includes adapting a generation rate of a plurality of payload packets to match the transmission rate.

3. The method of claim 1 , wherein the ad-hoc network includes an underwater ad-hoc network.

4. The method of claim 1 , wherein:
the broadcasting includes broadcasting the transmitted packet having the header including the current position and the current velocity of the intermediate node and the self V-value of the intermediate node representing a current cost to transmit the payload packet from the intermediate node to the destination node via the ad-hoc network; and
the correspondingly receiving includes from the proximate node correspondingly receiving the received packet having the header including the current position and the current velocity of the proximate node and a self V-value of the proximate node representing a current cost to transmit the payload packet from the proximate node to the destination node via the ad-hoc network,
wherein, in accordance with the reinforcement learning routing algorithm, the method further comprises:
at the intermediate node, maintaining and updating the estimated V-value of the proximate node with the self V-value of the proximate node from the received packet; and
at the intermediate node, maintaining and updating the self V-value of the intermediate node based on the self V-value of the proximate node from the received packet, wherein the change in the self V value of the intermediate node is a difference before and after the updating,
wherein the broadcasting further includes, immediately after the updating when the change in the self V-value is not less than a predetermined threshold value, broadcasting the transmitted packet having the header including the self V-value of the intermediate node after the updating.

5. The method of claim 1 , wherein:
the determining includes generating and transmitting a plurality of payload packets from the source node while any one of the nodes currently qualifies as a beneficial neighbor node for receiving and forwarding packets to the destination node, wherein the beneficial neighbor node is in the neighboring subset and has the estimated V-value with a lower cost than the self V-value of the source node.

6. A method for an ad-hoc network having a plurality of nodes including a source node and a destination node, the method comprising, at each intermediate one of the nodes besides the destination node:
measuring a current position and a current velocity of the intermediate node;
maintaining an estimated position and an estimated velocity for each of the nodes in a neighboring subset of the nodes;
broadcasting at least one transmitted packet including the current position and the current velocity of the intermediate node;
correspondingly receiving at least one received packet from at least a proximate one of the nodes, the received packet including a current position and a current velocity of the proximate node;
updating the estimated position and the estimated velocity of the proximate node with the current position and the current velocity from the received packet;
calculating a hop duration of a respective reliable link between the intermediate node and each of at least a neighbor one of the nodes in the neighboring subset, including calculating the hop duration from the current position and the current velocity measured for the intermediate node and from the estimated position and the estimated velocity maintained for the neighbor node;
receiving a payload packet for transmission from the source node to the destination node via the ad-hoc network; and
adaptively deferring a transmission of the payload packet from the intermediate node to the neighbor node of the intermediate node in the ad-hoc network, based on an estimated V-value of the neighbor node, a change in a self V-value of the intermediate node, and the hop duration of the respective reliable link between the intermediate node and the neighbor node, wherein the estimated V-value and the self V-value are determined from a reinforcement learning routing algorithm, the self V-value converges to a current cost of transmitting the payload packet from the intermediate node to the destination node, and the estimated V-value converges to a current cost of transmitting the payload packet from the neighbor node to the destination node via the ad-hoc network,
wherein the broadcasting includes broadcasting the transmitted packet having a header including a path duration of a network path and a confidence of the network path from the intermediate node to the destination node via the ad-hoc netw

Assignees

• Us Navy

Inventors

• Plate Randall S
• Wakayama Cherry Y

Classifications

• H04L45/08 Primary

  Learning-based routing, e.g. using neural networks or artificial intelligence · CPC title

• H04W40/026

  Route selection considering the moving speed of individual devices · CPC title

• H04W40/20

  based on geographic position or location · CPC title

• H04L45/124

  using a combination of metrics · CPC title

• H04W40/26

  for hybrid routing by combining proactive and reactive routing · CPC title