Multi-Agent Deep Reinforcement Learning in Flying Ad-Hoc Networks for Delay-Constrained Applications

Abstract

As a benefit of their compelling features, unmanned aerial vehicles (UAVs) are expected to play a vital role in 6G networks. UAVs are capable of providing wireless connectivity where network infrastructure is not available, or complement the conventional base stations. UAVs organized as a Flying Ad-hoc Network (FANET) can process jobs received by ground devices through vertical offload. Moreover, horizontal offload inside the FANET can applied to balance the load among UAVs to reduce the processing delay to offloaded jobs. However, offloading decisions inside a FANET have to be dynamically adapted to the current processing load distribution based on the state of the FANET and the number of job requests coming from the served geographical area. As the number of UAVs in the FANET increases, single agent reinforcement learning approaches fall short to achieve reasonable performance due to the exponential increase in the action space. For this reason, in this paper we propose a Multi-AgeNt Intra-FANET (MANIA-F) Framework based on Multi Agent PPO (MAPPO), in which each agent chooses the best offloading probabilities to forward incoming jobs to neighboring UAVs. The horizontal offload decision problem is defined as a Markov Decision Process (MDP) that is solved via Multi-Agent Deep Reinforcement Learning (MARL).