Hierarchical Federated Deep Reinforcement Learning based Joint Communication and Computation for UAV Situation Awareness

Abstract

The computation-intensive situational awareness (SA) task of unmanned aerial vehicle (UAV) is greatly affected by its limited power and computing capability. To solve this challenge, we consider the joint communication and computation (JCC) design for UAV network in this paper. Firstly, a multi-objective optimization (MOO) model, which can optimize UAV computation offloading, transmit power, and local computation resources simultaneously, is built to minimize energy consumption and task execution delay. Then, we develop Thompson sampling based double-DQN (TDDQN) learning algorithm which allows the agent to explore more deeply and effectively, and propose a joint optimization algorithm that combines TDDQN and sequential least squares quadratic programming (SLSQP) to handle the MOO problem. Finally, to enhance the training speed and quality, we incorporate federated learning (FL) into the presented joint optimization algorithm and propose hierarchical federated TDDQN with SLSQP (HF TDDQN-S) to implement the JCC design. Simulation results show that the introduced HF TDDQN-S can efficiently learn the best JCC strategy and minimize the average cost contrasted with the DDQN with SLSQP (DDQN-S) and TDDQN with SLSPQ (TDDQN-S) approach, and achieve the low average delay SA with power efficient.