Pursuit-evasion Game Research Articles

Reinforcement learning-based formation-surrounding control for multiple quadrotor UAVs pursuit-evasion games

This paper proposes a reinforcement learning-based formation-surrounding control method for multiple quadrotor unmanned aerial vehicles (UAVs) pursuit-evasion (MPE) games system subject to external disturbances. In the framework of the MPE games, the pursuers aim to equally surround the evaders which try to avoid being surrounded when forming the desired formation. By constructing position and attitude tracking error subsystems of quadrotor UAV, this paper proposes two control strategies which combines the feedforward control technique and reinforcement learning (RL) method. First, two novel cost functions are presented for the quadrotor UAV with external disturbances. Then, two control schemes based on RL have been developed to guarantee the stability of the tracking error subsystem. Subsequently, two critic-only neural networks (NN) weight update laws that only satisfy finite excitation conditions are proposed to estimate the optimal cost function. Furthermore, Nash equilibrium for multiple quadrotor UAVs is achieved by means of RL strategy to solve the Hamilton-Jacobi-Isaacs (HJI) equations. And the property of equally surrounding is proved for the first time by utilizing Euler's formula in this paper. Finally, the numerical simulation results are given to show the effectiveness and superior performance of the proposed control method.

Optimal strategies for large-scale pursuers against one evader: A mean field game-based hierarchical control approach

This paper proposes a two-level hierarchical control approach based on pursuit-evasion game and mean field game for the problem of large-scale pursuers with multi-population against a single evader, which implements that the evader is surrounded by pursuers. At the upper layer, we model the pursuit-evasion game between the centers of the pursuer populations and the single evader, which is formulated as a linear quadratic differential game (LQDG) to obtain the optimal control of each player. Then the optimal trajectories derived from the optimal controls are input to the lower layer as the reference trajectories. At the lower layer, we formulate the tracking of reference trajectories and terminal surrounding to the evader of large-scale pursuers with multi-population as a multi-population mean-field game (MPMFG), which solves the communication and computing difficulties caused by large-scale agents. Then, we derive the variational primal–dual formulation of the proposed MPMFG model and solve it with CA-Net, a coupled alternating neural network approach. Finally, simulation experiments are performed under various pursuit-evasion scenarios, and it is verified that the proposed game-based two-level hierarchical control approach is feasible and effective.

Modeling the Solution of the Pursuit–Evasion Problem Based on the Intelligent–Geometric Control Theory

An important action-planning problem is considered for participants of the pursuit–evasion game with multiple pursuers and a high-speed evader. The objects of study are mobile robotic systems and specifically small unmanned aerial vehicles (UAVs). The problem is complicated by the presence of significant wind loads that affect the trajectory and motion strategies of the players. It is assumed that UAVs have limited computing resources, which involves the use of computationally fast and real-time heuristic approaches. A novel and rapidly developing intelligent–geometric theory is applied to address the discussed problem. To accurately calculate the points of the participant’s rapprochement, we use a geometric approach based on the construction of circles or spheres of Apollonius. Intelligent control methods are applied to synthesize complex motion strategies of participants. A method for quickly predicting the evader’s trajectory is proposed based on a two-layer neural network containing a new activation function of the “s-parabola” type. We consider a special backpropagation training scheme for the model under study. A simulation scheme has been developed and tested, which includes mathematical models of dynamic objects and wind loads. The conducted simulations on pursuit–evasion games in close to real conditions showed the prospects and expediency of the presented approach.

Deep neural network for non-cooperative space target intention recognition

Given the critical importance of intention recognition for non-cooperative space targets in enhancing spatial situational awareness and mitigating threats, this paper proposes a fast and accurate method based on a deep neural network for identifying the intentions. The proposed method takes the instantaneous single relative motion position and velocity of the non-cooperative target as input and calculates the probability of different potential intentions in real-time. To this end, the possible geometric configurations of relative motions of the targets are characterized and classified based on the Clohessy-Wiltshire (CW) equation, and intentions with respect to different types of motion are defined and grouped accordingly. To ensure the method's applicability for large range scenario, the required dataset in two-body dynamics is generated using satellite orbit extrapolation techniques. The deep neural network is trained on the dataset using back-propagation, and experiments are designed to evaluate the method's performance and the recognition accuracy reached 0.9289. The proposed method outperforms the dynamical characteristics matching-based method in terms of recognition accuracy, range of applicability, and time efficiency. The method has potential applications in spacecraft collision warning and orbital pursuit-evasion games, where rapid and accurate identification of non-cooperative target intentions is critical.

Guidance strategy of motion camouflage for spacecraft pursuit-evasion game

This work is inspired by a stealth pursuit behavior called motion camouflage whereby a pursuer approaches an evader while the pursuer camouflages itself against a predetermined background. We formulate the spacecraft pursuit-evasion problem as a stealth pursuit strategy of motion camouflage, in which the pursuer tries to minimize a motion camouflage index defined in this paper. The Euler-Hill reference frame whose origin is set on the circular reference orbit is used to describe the dynamics. Based on the rule of motion camouflage, a guidance strategy in open-loop form to achieve motion camouflage index is derived in which the pursuer lies on the camouflage constraint line connecting the central spacecraft and evader. In order to dispose of the dependence on the evader acceleration in the open-loop guidance strategy, we further consider the motion camouflage pursuit problem within an infinite-horizon nonlinear quadratic differential game. The saddle point solution to the game is derived by using the state-dependent Riccati equation method, and the resulting closed-loop guidance strategy is effective in achieving motion camouflage. Simulations are performed to demonstrate the capabilities of the proposed guidance strategies for the pursuit–evasion game scenario.

On the Lifeline Game of the Inertial Players with Integral and Geometric Constraints

In this paper, we consider a pursuit–evasion game of inertial players, where the pursuer’s control is subject to integral constraint and the evader’s control is subject to geometric constraint. In the pursuit problem, the main tool is the strategy of parallel pursuit. Sufficient conditions are obtained for the solvability of pursuit–evasion problems. Additionally, the main lemma describing the monotonicity of an attainability domain of the evader is proved, and an explicit analytical formula for this domain is given. One of the main results of the paper is the solution of the Isaacs lifeline game for a special case.

Game of Drones: Multi-UAV Pursuit-Evasion Game With Online Motion Planning by Deep Reinforcement Learning.

As one of the tiniest flying objects, unmanned aerial vehicles (UAVs) are often expanded as the ``swarm'' to execute missions. In this article, we investigate the multiquadcopter and target pursuit-evasion game in the obstacles environment. For high-quality simulation of the urban environment, we propose the pursuit-evasion scenario (PES) framework to create the environment with a physics engine, which enables quadcopter agents to take actions and interact with the environment. On this basis, we construct multiagent coronal bidirectionally coordinated with target prediction network (CBC-TP Net) with a vectorized extension of multiagent deep deterministic policy gradient (MADDPG) formulation to ensure the effectiveness of the damaged ``swarm'' system in pursuit-evasion mission. Unlike traditional reinforcement learning, we design a target prediction network (TP Net) innovatively in the common framework to imitate the way of human thinking: situation prediction is always before decision-making. The experiments of the pursuit-evasion game are conducted to verify the state-of-the-art performance of the proposed strategy, both in the normal and antidamaged situations.

Intelligent Maneuver Strategy for a Hypersonic Pursuit-Evasion Game Based on Deep Reinforcement Learning

In order to improve the problem of overly relying on situational information, high computational power requirements, and weak adaptability of traditional maneuver methods used by hypersonic vehicles (HV), an intelligent maneuver strategy combining deep reinforcement learning (DRL) and deep neural network (DNN) is proposed to solve the hypersonic pursuit–evasion (PE) game problem under tough head-on situations. The twin delayed deep deterministic (TD3) gradient strategy algorithm is utilized to explore potential maneuver instructions, the DNN is used to fit to broaden application scenarios, and the intelligent maneuver strategy is generated with the initial situation of both the pursuit and evasion sides as the input and the maneuver game overload of the HV as the output. In addition, the experience pool classification strategy is proposed to improve the training convergence and rate of the TD3 algorithm. A set of reward functions is designed to achieve adaptive adjustment of evasion miss distance and energy consumption under different initial situations. The simulation results verify the feasibility and effectiveness of the above intelligent maneuver strategy in dealing with the PE game problem of HV under difficult situations, and the proposed improvement strategies are validated as well.

Designing discrete-time control-based strategies for pursuit-evasion games on the plane

Two-player discrete-time pursuit-evasion games are proposed with control-based strategies on the plane, where each player's position is determined by the discrete-time version of the kinematic equation related to the moving structure of vehicles. In this regard, the first game is built in a one-target fashion where the high-speed pursuer tries to reach the running evader in optimal discrete time, whereas the second game is constructed from a two-target perspective where the evader becomes an attacker whose ultimate aim is to build a control-based strategy to capture the high-speed pursuer who follows the shortest tracking policy within his target region. Each player's winning and losing strategies are designed under certain playability conditions. Later, the main results are generalized for players sharing various discrete-time speeds during the game. The motivation of the study is to investigate the losing and winning strategies of the evader against the high-speed pursuer, who tries to capture his opponent in optimal discrete time on the plane. Numerical experiments are performed on four different unmanned aerial vehicles (UAVs) flying at the same altitude with various airspeeds and turning radii, where several capture scenarios and control-based strategies for each UAV are constructed according to the main results.

StarCraft adversary-agent challenge for pursuit–evasion game

A reinforcement learning environment with adversary agents is proposed in this work for pursuit–evasion game in the presence of fog of war, which is of both scientific significance and practical importance in aerospace applications. One of the most popular learning environments, StarCraft, is adopted here and the associated mini-games are analyzed to identify its potential applications and limitations for training adversary agents. The key contribution includes the analysis of the best performance that an intelligent agent could be achieved by incorporating control and differential game theory into the specific reinforcement learning environment, and the development of a StarCraft adversary-agents challenge (SAAC) environment by extending the current StarCraft mini-games. The subsequent study showcases the use of this learning environment and the effectiveness of an adversary agent for evaders. Overall, along with rapidly-emerging reinforcement learning technologies, the proposed SAAC environment should benefit pursuit–evasion studies in particular and aerospace applications in general. Last but not least, the corresponding code is available at GitHub.

Nonzero-Sum Pursuit-Evasion Game Control for Spacecraft Systems: A Q-Learning Method

In this paper, the nonzero-sum pursuit-evasion (PE) game control problem is studied for a class of linear spacecraft control systems subject to the complete information case and incomplete information case. The incomplete information includes the cost functions and the control inputs. In practical confrontation situations, due to the incomplete information constraints, it is impossible for the pursuer and the evader to build up the exact opposite cost function. Hence, a nonzero-sum game framework is utilized to describe the PE game problem of the double-spacecraft system. First of all, under the complete information case, the nonzero-sum PE game control strategy is designed by solving the coupled Riccati recursions. Then, aiming at the incomplete information case, a control gain estimator is established, which lays the foundation for the control strategy design of the pursuit spacecraft. On the basis of the estimated control gain, the pursuit control strategy is solved by using the standard discrete-time Riccati recursion. In order to further get rid of the system information, a Q-learning-based control gain is designed for the pursuit spacecraft. Finally, a numerical example on the PE spacecraft system is provided to verify the effectiveness of the proposed PE game control strategies.

Threat potential field based Pursuit–Evasion Games for underactuated Unmanned Surface Vehicles

Unmanned Surface Vehicles (USVs) have been widely applied in ocean engineering. However, there are few studies for Pursuit–Evasion Game based on USVs. Most relevant pursuit game strategies based on nonrepresentational agents are simple and do not consider the USV’s underactuation. In order to solve this problem, this paper proposes a distributed pursuit game algorithm for USVs based on Threat potential fields (TPFs). The pursuit planning can be divided into three parts based on algorithm structure: First, according to the evader’s information, the proposed algorithm can quantitatively establish a unique TPF for evaluating the danger of the evader. Second, in the TPF, energy peak and energy valley concepts are innovatively presented to provide suitable conditions for the hunters’ pursuit. Third, the pursuit game optimization model is established based on a TPF for the hunters’ game decision-making by considering the orientation, heading, and formation shape of the USVs in the pursuit process. The proposed game optimization model is then solved using the particle swarm optimization algorithm to obtain the strategies. Finally, the effectiveness of the proposed algorithm is demonstrated via numeric simulation of USVs.

Hierarchical Maneuver Decision Method Based on PG-Option for UAV Pursuit-Evasion Game

Aiming at the autonomous decision-making problem in an Unmanned aerial vehicle (UAV) pursuit-evasion game, this paper proposes a hierarchical maneuver decision method based on the PG-option. Firstly, considering various situations of the relationship of both sides comprehensively, this paper designs four maneuver decision options: advantage game, quick escape, situation change and quick pursuit, and the four options are trained by Soft Actor-Critic (SAC) to obtain the corresponding meta-policy. In addition, to avoid high dimensions in the state space in the hierarchical model, this paper combines the policy gradient (PG) algorithm with the traditional hierarchical reinforcement learning algorithm based on the option. The PG algorithm is used to train the policy selector as the top-level strategy. Finally, to solve the problem of frequent switching of meta-policies, this paper sets the delay selection of the policy selector and introduces the expert experience to design the termination function of the meta-policies, which improves the flexibility of switching policies. Simulation experiments show that the PG-option algorithm has a good effect on UAV pursuit-evasion game and adapts to various environments by switching corresponding meta-policies according to current situation.

A lionfish-inspired predation strategy in planar structured environments *

This paper investigates a pursuit-evasion game with a single pursuer and evader in a bounded environment, inspired by observations of predation attempts by lionfish (Pterois sp.). The pursuer tracks the evader with a pure pursuit strategy while using an additional bioinspired tactic to trap the evader, i.e. minimize the evader’s escape routes. Specifically, the pursuer employs symmetric appendages inspired by the large pectoral fins of lionfish, but this expansion increases its drag and therefore its work to capture the evader. The evader employs a bioinspired randomly-directed escape strategy to avoid capture and collisions with the boundary. Here we investigate the trade-off between minimizing the work to capture the evader and minimizing the evader’s escape routes. By using the pursuer’s expected work to capture as a cost function, we determine when the pursuer should expand its appendages as a function of the relative distance to the evader and the evader’s proximity to the boundary. Visualizing the pursuer’s expected work to capture everywhere in the bounded domain, yields additional insights about optimal pursuit trajectories and illustrates the role of the boundary in predator-prey interactions.

Solving Large-Scale Pursuit-Evasion Games Using Pre-trained Strategies

Pursuit-evasion games on graphs model the coordination of police forces chasing a fleeing felon in real-world urban settings, using the standard framework of imperfect-information extensive-form games (EFGs). In recent years, solving EFGs has been largely dominated by the Policy-Space Response Oracle (PSRO) methods due to their modularity, scalability, and favorable convergence properties. However, even these methods quickly reach their limits when facing large combinatorial strategy spaces of the pursuit-evasion games. To improve their efficiency, we integrate the pre-training and fine-tuning paradigm into the core module of PSRO -- the repeated computation of the best response. First, we pre-train the pursuer's policy base model against many different strategies of the evader. Then we proceed with the PSRO loop and fine-tune the pre-trained policy to attain the pursuer's best responses. The empirical evaluation shows that our approach significantly outperforms the baselines in terms of speed and scalability, and can solve even games on street maps of megalopolises with tens of thousands of crossroads -- a scale beyond the effective reach of previous methods.

Solution for Pursuit-Evasion Game of Agents by Adaptive Dynamic Programming

The paper studies a novel method for real-time solutions of the two-player pursuit-evasion game. The min-max principle is adopted to confirm the Nash equilibrium of the game. As agents in the game can form an Internet of Things (IoT) system, the real-time control law of each agent is obtained by taking a linear-quadratic cost function in adaptive dynamic programming. By introducing the Lyapunov function, we consider the scenario when capture occurs. Since most actual systems are continuous, the policy iteration algorithm is used to make the real-time policy converge to the analytical solution of the Nash equilibrium. Furthermore, we employ the value function approximation method to calculate the neural network parameters without directly solving the Hamilton–Jacobi–Isaacs equation. Simulation results depict the method’s feasibility in different scenarios of the pursuit-evasion game.

An AutoML based trajectory optimization method for long-distance spacecraft pursuit-evasion game

An AutoML based trajectory optimization method for long-distance spacecraft pursuit-evasion game

A Geometrical, Reachable Set Approach for Constrained Pursuit–Evasion Games with Multiple Pursuers and Evaders

This paper presents a solution strategy for deterministic time-optimal pursuit–evasion games with linear state constraints, convex control constraints, and linear dynamics that is consistent with linearized relative orbital motion models such as the Clohessy–Wiltshire equations and relative orbital elements. The strategy first generates polytopic inner approximations of the players’ reachable sets by solving a sequence of convex programs. A bisection method then computes the optimal termination time, which is the least time at which a set containment condition is satisfied. The pursuit–evasion games considered are games with (1) a single pursuer and single evader, (2) multiple pursuers and a single evader, and (3) a single pursuer and multiple evaders. Compared to variational methods, this reachable set strategy leads to a tractable formulation even when there are state and control constraints. The efficacy of the strategy is demonstrated in three numerical simulations for a constellation of satellites in close proximity in low earth orbit.

Multiplayer Obstacle Avoidance Pursuit-Evasion Games With Adaptive Parameter Estimation

This paper investigates a class of multi-player obstacle avoidance pursuit-evasion games with intra coalition cooperation and inter coalition confrontation (ICCICC). In order to obtain distributed control strategy, a communication topology graph is utilized to quantify the interactions of the players. The solutions of the Hamilton-Jacobi-Isaacs (HJI) equations of the pursuit-evasion games are employed to calculate Nash equilibrium (NE) of the players. An adaptive method is utilized to obtain opponent’s control parameters for updating multiple distributed control strategy. Moreover, the sufficient condition is provided to guarantee stability of multi-player pursuit-evasion (MPPE) game system with adaptive estimation. Finally, an MPPE game experiment with two evaders and two pursuers is carried out to demonstrate the effectiveness of the proposed methodology.

Optimal delta-V-based strategies in orbital pursuit-evasion games

This research investigates the orbital pursuit-evasion games with impulsive maneuvers in long-distance interception scenarios. Specifically, this paper examines the influences of escape impulses, with different directions, applied by the evader on the intercept-correction delta-v (ΔVCorr) of the pursuer that is already on an interception orbit. After obtaining the evader’s strategies based on maximizing the ΔVCorr under different circumstances, it is found that, with the knowledge of the evader’s best strategy, the pursuer could make ΔVCorr smaller by adjusting its initial interception orbit. Then, a method to modify the initial orbit with energy constraints is designed so that the pursuer can match the evader’s escape maneuver with less delta-v to prepare. Finally, this paper summarizes the best strategies of both sides after considering the initial trajectory adjustment and discusses the impact of incomplete information on tactics in orbital pursuit-evasion games.