Tactical conflict resolution in urban airspace for unmanned aerial vehicles operations using attention-based deep reinforcement learning

Abstract

Unmanned aerial vehicles (UAVs) have gained much attention from academic and industrial areas due to the significant number of potential applications in urban airspace. A traffic management system for these UAVs is needed to manage this future traffic. Tactical conflict resolution for unmanned aerial systems (UASs) is an essential piece of the puzzle for the future UAS Traffic Management (UTM), especially in very low-level (VLL) urban airspace. Unlike conflict resolution in higher altitude airspace, the dense high-rise buildings are an essential source of potential conflict to be considered in VLL urban airspace. In this paper, we propose an attention-based deep reinforcement learning approach to solve the tactical conflict resolution problem. Specifically, we formulate this task as a sequential decision-making problem using Markov Decision Process (MDP). The double deep Q network (DDQN) framework is used as a learning framework for the host drone to learn to output conflict-free maneuvers at each time step. We use the attention mechanism to model the individual neighbor's effect on the host drone, endowing the learned conflict resolution policy to be adapted to an arbitrary number of neighboring drones. Lastly, we build a simulation environment with various scenarios covering different types of encounters to evaluate the proposed approach. The simulation results demonstrate that our proposed algorithm provides a reliable solution to minimize secondary conflict counts compared to learning and non-learning-based approaches under different traffic density scenarios.