Deep reinforcement learning in NOMA-assisted UAV networks for path selection and resource offloading

Abstract

This paper constructs a NOMA-based UAV-assisted Cellular Offloading (UACO) framework and designs a UAV path selection and resource offloading algorithm (UPRA) based on deep reinforcement learning. This paper focuses on the coupling relationship between path selection and resource offloading during the movement of UAVs. A joint optimization problem between UAV three-dimensional path design and resource offloading is proposed, considering the UAV’s autonomous obstacle avoidance in complex environments and the influence of obstacles in 3D space on the channel model. In particular, a constrained clustering-assignment algorithm is designed by improving the K-means algorithm and combining it with the assignment algorithm in order to achieve periodic clustering of random motion users and UAV task assignment. In addition, a semi-fixed hierarchical power allocation algorithm is embedded in the designed DQN algorithm to improve the convergence performance of the learning algorithm in this paper. Simulation results show that: the NOMA-based design is able to improve the spectrum utilization efficiency and communication throughput of the UAV network system. Compared with the artificial potential field, the proposed algorithm is able to solve the problem of falling into suboptimal solutions in path selection and improve the communication throughput. In addition, this paper explores the impact of the reward function on the training convergence and the results in deep reinforcement learning. The excellent adaptability of the designed algorithm in dynamic networks as well as in complex environments is demonstrated by random deployment of users and varying the maximum user movement speed.