Reinforcement learning based UAV formation control in GPS-denied environment

Abstract

Highly accurate positioning is a crucial prerequisite of multi Unmanned Aerial Vehicle close-formation flight for target tracking, formation keeping, and collision avoidance. Although the position of a UAV can be obtained through the Global Positioning System (GPS), it is difficult for a UAV to obtain highly accurate positioning data in a GPS-denied environment (e.g., a GPS jamming area, suburb, urban canyon, or mountain area); this may cause it to miss a tracking target or collide with another UAV. In particular, UAV close-formation control in GPS-denied environments faces difficulties owing to the low-accuracy position, close distance between vehicles, and nonholonomic dynamics of a UAV. In this paper, on the one hand, we develop an innovative UAV formation localization method to address the formation localization issues in GPS-denied environments; on the other hand, we design a novel reinforcement learning based algorithm to achieve the high-efficiency and robust performance of the controller. First, a novel Lidar-based localization algorithm is developed to measure the localization of each aircraft in the formation flight. In our solution, each UAV is equipped with Lidar as the position measurement sensor instead of the GPS module. The k-means algorithm is implemented to calculate the center point position of UAV. A novel formation position vector matching method is proposed to match center points with UAVs in the formation and estimate their position information. Second, a reinforcement learning based UAV formation control algorithm is developed by selecting the optimal policy to control UAV swarm to start and keep flying in a close formation of a specific geometry. Third, the innovative collision risk evaluation module is proposed to address the collision-free issues in the formation group. Finally, a novel experience replay method is also provided in this paper to enhance the learning efficiency. Experimental results validate the accuracy, effectiveness, and robustness of the proposed scheme.