Deep reinforcement learning based path stretch vector resolution in dense traffic with uncertainties

Abstract

With the continuous growth in the air transportation demand, air traffic controllers will have to handle increased traffic and consequently, more potential conflicts. This gives rise to the need for conflict resolution advisory tools that can perform well in high-density traffic scenarios given a noisy environment. Unlike model-based approaches, learning-based approaches can take advantage of historical traffic data and flexibly encapsulate environmental uncertainty. In this study, we propose a reinforcement learning approach that is capable of resolving conflicts, in the presence of traffic and inherent uncertainties in conflict resolution maneuvers, without the need for prior knowledge about a set of rules mapping from conflict scenarios to expected actions. The conflict resolution task is formulated as a decision-making problem in a large and complex action space. The research also includes the development of a learning environment, scenario state representation, reward function, and a reinforcement learning algorithm inspired from Q-learning and Deep Deterministic Policy Gradient algorithms. The proposed algorithm, with two stages decision-making process, is used to train an agent that can serves as an advisory tool for air traffic controllers in resolving air traffic conflicts where it can learn from historical data by evolving overtime. Our findings show that the proposed model gives the agent the capability to suggest high quality conflict resolutions under different environmental conditions. It outperforms two baseline algorithms. The trained model has high performance under low uncertainty level (success rate ≥95% ) and medium uncertainty level (success rate ≥87%) with high traffic density. The detailed analysis of different impact factors such as environment’s uncertainty and traffic density on learning performance are investigated and discussed. The environment’s uncertainty is the most important factor which affects the performance. Moreover, the combination of high-density traffic and high uncertainty will be the challenge for any learning models.