Heterogeneous Unmanned Swarm Formation Containment Control Based on Reinforcement Learning

Abstract

In response to the problem of time-varying spherical formation control for a heterogeneous unmanned aerial vehicle (UAV) swarm system with dynamic uncertainty in the system model, this paper proposes an optimal distributed formation containment control method based on reinforcement learning. By combining the time-varying formation containment vector design with the distributed predefined time observer and constructing the augmented system of a multi-quadrotor UAV system and an observer, the problem of time-varying formation containment control in heterogeneous swarm systems is transformed into a stabilization problem. By introducing a value function with a discount factor, the stabilization problem of a heterogeneous UAV swarm system is transformed into an optimal control problem. Using the “actor-critic” neural network, combined with an off-policy reinforcement learning algorithm and distributed control methods, the solution to the formation containment controller is achieved in a data-driven manner. The stability of distributed observer and formation containment controller as well as the convergence of reinforcement learning algorithms are demonstrated using Lyapunov and related theory. The numerical simulation results demonstrate that the observation error of the designed distributed observer converges within a predefined time of 0.1 s, while the overall formation tracking error of the heterogeneous swarm converges within 2.74 s, thereby further validating the effectiveness and superiority of the designed control scheme.