UAV maneuvering decision -making algorithm based on Twin Delayed Deep Deterministic Policy Gradient Algorithm

Jianmei Wang,Bo Li,Shiyang Liang,Evgeny Neretin,Shaomei Song,Shuangxia Bai

doi:10.37965/jait.2021.12003

Abstract

Aiming at intelligent decision-making of UAV based on situation information in air combat, a novel maneuvering decision method based on deep reinforcement learning is proposed in this paper. The autonomous maneuvering model of UAV is established by Markov Decision Process. The Twin Delayed Deep Deterministic Policy Gradient(TD3) algorithm and the Deep Deterministic Policy Gradient (DDPG) algorithm in deep reinforcement learning are used to train the model, and the experimental results of the two algorithms are analyzed and compared. The simulation experiment results show that compared with the DDPG algorithm, the TD3 algorithm has stronger decision-making performance and faster convergence speed, and is more suitable forsolving combat problems. The algorithm proposed in this paper enables UAVs to autonomously make maneuvering decisions based on situation information such as position, speed, and relative azimuth, adjust their actions to approach and successfully strike the enemy, providing a new method for UAVs to make intelligent maneuvering decisions during air combat.

Highlights

At present, unmanned aerial vehicles (UAVs) are widely used in military applications such as reconnaissance, attack, and jamming
Traditional methods such as Game Theory need to establish a clear and complete problem model. In another part of the research[6,7,8,9,10,11,12,13], the UAV maneuvering decision-making is realized by deep reinforcement learning, the autonomous maneuvering model of UAV is established by Markov Decision Process, and the decision function is fitted by neural network
After initializing the positions of the UAVs on both sides of the battle, the UAV can automatically generate maneuvering decision based on the battlefield situation information based on the deep reinforcement learning algorithm, so that it can occupy a favorable position in the air combat

Summary

Introduction

At present, unmanned aerial vehicles (UAVs) are widely used in military applications such as reconnaissance, attack, and jamming. Traditional methods such as Game Theory need to establish a clear and complete problem model. In another part of the research[6,7,8,9,10,11,12,13], the UAV maneuvering decision-making is realized by deep reinforcement learning, the autonomous maneuvering model of UAV is established by Markov Decision Process, and the decision function is fitted by neural network. UAV air combat model, and a UAV maneuvering decision algorithm based on deep reinforcement learning is proposed.

UAV air combat model

Experimental parameter settings

Conclusion