Packet Routing in Dynamic Multi-Hop UAV Relay Network: A Multi-Agent Learning Approach

Abstract

The multi-hop unmanned aerial vehicle (UAV) network can serve as data relays where ground users (GUs) do not have reliable direct connections to the base station (BS). Existing works mainly focus on simple dual-hop system. In this paper, we investigate the packet routing problem in a multi-hop UAV relay network to minimize the data transmission time and enhance the network throughput. However, the dynamic network topology due to UAV mobility makes the packet routing challenging since the limited communication range of each UAV leads to volatile wireless connection. Moreover, the line-of-sight communication links may cause strong interference among UAVs. Towards this end, we propose a novel multi-agent deep reinforcement learning based algorithm, named as multi-agent QMIX (MAQMIX) to: 1) design proper UAVs’ trajectories to serve the moving GUs while maintaining the network connection; 2) allocate frequency resource properly among UAVs to alleviate the impact of interference; and 3) choose a proper next hop UAV for each data packet to reduce the transmission time and probability of network congestion. The proposed MAQMIX has two novel training mechanisms, i.e., intra-UAV and inter-UAV training mechanisms, which can tackle the large action space issue and coordinate the training among UAVs in the multi-hop UAV relay network. Simulation results demonstrate that the MAQMIX outperforms baseline schemes in terms of the network congestion avoidance, throughput, and transmission time.