Intelligent Joint Beamforming and Distributed Power Control for UAV-assisted Ultra-Dense Network: A Hierarchical Optimization Approach

Abstract

In this paper, the intelligent resource allocation problem has been investigated for the ultra-dense network under harsh environments, e.g. battlefield, etc. Recently, Ultra-Dense Network (UDN) and Unmanned Aerial Vehicle (UAV)-assisted dynamic network have emerged as prominent solutions to overcome the challenges of fulfilling 5G/6G extremely high capacity density requirement. To stimulate the potential of the UAV-assisted ultra-dense network, designing effective resource allocation is critical and challenging. Hence, a novel joint beamforming and distributed power control algorithm have been developed that can optimize the UAV-assisted ultra-dense network effectively and further pave the way to benefit emerging complex systems such as the Internet of Battlefield Things (IoBT). Firstly, a novel hierarchical optimization framework is designed to reduce the optimization complexity in UAV-assisted ultra-dense network resource allocation by integrating beamforming for the base station as well as UAV relays (Upper Level) and distributed power control for multi-users (Lower Level) in ultra-dense network seamlessly. Then, a deep reinforcement learning technique has been developed along with dynamic codebook development to find the optimal beamforming strategy for upper level resource allocation optimization in the UAV-assisted ultra-dense network. After that, an actor-critic-mass reinforcement learning algorithm is developed with mean-field game to obtain the optimal distributed power control for multi-users in the lower level of UAV-assisted ultra-dense network. To better coordinate upper level and lower level resource allocation optimization, base station and UAV can sense and describe the ultra-dense network communication environment as a novel multi-area signal-to-interference-noise (SINR) probability density function (PDF) with the mean value of multi-area PDF being the desired SINR targets for wireless users in the corresponding area. This two-way interaction mechanism can allow agents in the two-level UAV-assisted ultra-dense network to dynamically collaborate with each other and further achieve the overall optimal solution. The effectiveness of the proposed design has been demonstrated through real-time simulation.