Pursuit-evasion Problem Research Articles

Multi-UAV pursuit-evasion gaming based on PSO-M3DDPG schemes

The sample data for reinforcement learning algorithms often exhibit sparsity and instability, making the training results susceptible to falling into local optima. Mini-Max-Multi-agent Deep Deterministic Policy Gradient (M3DDPG) algorithm is a multi-agent reinforcement learning algorithm, which introduces the minimax theorem into Multi-Agent Deep Deterministic Policy Gradient (MADDPG) algorithm. It also has unstable convergence caused by sparse sample data and randomization. However, the Particle Swarm Optimisation (PSO) algorithm, unlike traditional reinforcement learning methods, involves the construction of independent populations of policy networks to generate sample data, followed by training the reinforcement learning algorithm. PSO optimizes and updates the policy population based on a fitness function, aiming to enhance the efficiency and convergence speed of the algorithm in learning from the sample data. In order to address the multi-agent pursuit-evasion problem, we propose the PSO-M3DDPG algorithm, which combines the PSO algorithm with the M3DDPG algorithm. Through experimental simulations, the improved algorithm demonstrates superior training results and faster convergence speeds, thus validating its effectiveness.

Exoatmospheric Evasion Guidance Law with Total Energy Limit via Constrained Reinforcement Learning

Due to the lack of aerodynamic forces, the available propulsion for exoatmospheric pursuit-evasion problem is strictly limited, which has not been thoroughly investigated. This paper focuses on the evasion guidance in an exoatmospheric environment with total energy limit. A Constrained Reinforcement Learning (CRL) method is proposed to solve the problem. Firstly, the acceleration commands of the evader are defined as cost and an Actor-Critic-Cost (AC2) network structure is established to predict the accumulated cost of a trajectory. The learning objective of the agent becomes to maximize cumulative rewards while satisfying the cost constraint. Secondly, a Maximum-Minimum Entropy Learning (M2EL) method is proposed to minimize the randomness of acceleration commands while preserving the agent’s exploration capability. Our approaches address two challenges in the application of reinforcement learning: constraint specification and precise control. The well-trained agent is capable of generating accurate commands while satisfying the specified constraints. The simulation results indicate that the CRL and M2EL methods can effectively control the agent’s energy consumption within the specified constraints. The robustness of the agent under information error is also validated.

A novel evasion guidance for hypersonic morphing vehicle via intelligent maneuver strategy

This paper presents a novel evasion guidance law for hypersonic morphing vehicles, focusing on determining the optimized wing's unfolded angle to promote maneuverability based on an intelligent algorithm. First, the pursuit-evasion problem is modeled as a Markov decision process. And the agent's action consists of maneuver overload and the unfolded angle of wings, which is different from the conventional evasion guidance designed for fixed-shape vehicles. The reward function is formulated to ensure that the miss distances satisfy the prescribed bounds while minimizing energy consumption. Then, to maximize the expected cumulative reward, a residual learning method is proposed based on proximal policy optimization, which integrates the optimal evasion for linear cases as the baseline and trains to optimize the performance for nonlinear engagement with multiple pursuers. Therefore, offline training guarantees improvement of the constructed evasion guidance law over conventional ones. Ultimately, the guidance law for online implementation includes only analytical calculations. It maps from the confrontation state to the expected angle of attack and the unfolded angle while retaining high computational efficiency. Simulations show that the proposed evasion guidance law can utilize the change of unfolded angle to extend the maximum overload capability. And it surpasses conventional maneuver strategies by ensuring better evasion efficacy and higher energy efficiency.

Дифференциальная игра «преследование-уклонение» на основе обучения с подкреплением

The paper discusses optimal control algorithms based on actor/critic reinforcement learning (RL) schemes. The algorithms are used to solve pursuit-evasion (PE) problems of differential games. The work focuses on the implementation of agent policy decisions in accordance with the concept of adaptive dynamic programming. The essence of solving the PE game problem is to obtain the control policy of each agent (pursuer and evader) on both sides of the game. The paper proposes an adaptive dynamic programming (ADP) method for solving Nash equilibrium policies in differential pursuit-evasion games for two players. The cost function approximation method is used to calculate the parameters of a neural network (NN) without directly solving the Hamilton-Jacoby equation.

Modeling the Solution of the Pursuit–Evasion Problem Based on the Intelligent–Geometric Control Theory

An important action-planning problem is considered for participants of the pursuit–evasion game with multiple pursuers and a high-speed evader. The objects of study are mobile robotic systems and specifically small unmanned aerial vehicles (UAVs). The problem is complicated by the presence of significant wind loads that affect the trajectory and motion strategies of the players. It is assumed that UAVs have limited computing resources, which involves the use of computationally fast and real-time heuristic approaches. A novel and rapidly developing intelligent–geometric theory is applied to address the discussed problem. To accurately calculate the points of the participant’s rapprochement, we use a geometric approach based on the construction of circles or spheres of Apollonius. Intelligent control methods are applied to synthesize complex motion strategies of participants. A method for quickly predicting the evader’s trajectory is proposed based on a two-layer neural network containing a new activation function of the “s-parabola” type. We consider a special backpropagation training scheme for the model under study. A simulation scheme has been developed and tested, which includes mathematical models of dynamic objects and wind loads. The conducted simulations on pursuit–evasion games in close to real conditions showed the prospects and expediency of the presented approach.

Guidance strategy of motion camouflage for spacecraft pursuit-evasion game

This work is inspired by a stealth pursuit behavior called motion camouflage whereby a pursuer approaches an evader while the pursuer camouflages itself against a predetermined background. We formulate the spacecraft pursuit-evasion problem as a stealth pursuit strategy of motion camouflage, in which the pursuer tries to minimize a motion camouflage index defined in this paper. The Euler-Hill reference frame whose origin is set on the circular reference orbit is used to describe the dynamics. Based on the rule of motion camouflage, a guidance strategy in open-loop form to achieve motion camouflage index is derived in which the pursuer lies on the camouflage constraint line connecting the central spacecraft and evader. In order to dispose of the dependence on the evader acceleration in the open-loop guidance strategy, we further consider the motion camouflage pursuit problem within an infinite-horizon nonlinear quadratic differential game. The saddle point solution to the game is derived by using the state-dependent Riccati equation method, and the resulting closed-loop guidance strategy is effective in achieving motion camouflage. Simulations are performed to demonstrate the capabilities of the proposed guidance strategies for the pursuit–evasion game scenario.

On the Lifeline Game of the Inertial Players with Integral and Geometric Constraints

In this paper, we consider a pursuit–evasion game of inertial players, where the pursuer’s control is subject to integral constraint and the evader’s control is subject to geometric constraint. In the pursuit problem, the main tool is the strategy of parallel pursuit. Sufficient conditions are obtained for the solvability of pursuit–evasion problems. Additionally, the main lemma describing the monotonicity of an attainability domain of the evader is proved, and an explicit analytical formula for this domain is given. One of the main results of the paper is the solution of the Isaacs lifeline game for a special case.

A survey of the pursuit–evasion problem in swarm intelligence

A survey of the pursuit–evasion problem in swarm intelligence

Cooperative strategy for pursuit-evasion problem in the presence of static and dynamic obstacles

In complex and dynamic marine environments with multiple obstacles, efficient and flexible pursuit is a challenging task. This paper proposes a self-organizing cooperative pursuit strategy. First, the Apollonius circle is introduced as the final formation of the encirclement pattern, and the trajectory of the target is predicted on the basis of the least-squares method. To improve the efficiency of collision avoidance against static irregular obstacles and obtain a more reasonable collision avoidance path, we design a discretization method to describe the boundary of obstacles and combine it with the artificial potential field (APF) method. Furthermore, dynamic obstacle avoidance is considered to extend the applicability of the algorithm to a variety of obstacle environments. Compared with the common greedy algorithm, the simulation results show that in the obstacle free and static obstacle environment, the self-organizing cooperative pursuit algorithm can shorten the pursuit time and improve the movement consistency of the USVs swarm, and the obstacle avoidance route is smoother. In complex environments containing static obstacles and dynamic ships, the self-organizing cooperative pursuit algorithm can also pursuit escape targets effectively.

A Region-Based Relay Pursuit Scheme for a Pursuit–Evasion Game With a Single Evader and Multiple Pursuers

In this article, a class of the relay pursuit–evasion problem is dealt with. Multiple pursuers and a single evader take part in the considered game. We propose a region-based relay pursuit scheme for the pursuers to capture a single evader that adopts the pure evasion strategy. First, a round pursuit strategy, which is a quickest decent control law, is proposed as an alternative strategy to the pure evasion strategy. Next, a partition of the planar space is given by using the Voronoi diagram and some circular barriers to indicate the dominance region of each pursuer and each pursuit strategy. Then, the region-based relay pursuit scheme is put forward. Finally, two simulation examples are provided to verify the effectiveness and superiority of the proposed methodology.

A hybrid optimisation method for intercepting satellite trajectory based on differential game

AbstractThis study addresses orbit design and optimisation for the situation of satellite interception in which the target spacecraft is capable of manoeuvring using continuous magnitude restricted thrust. For the purpose of designing a long-range continuous thrust interception orbit, the orbit motion equations of two satellites with J2 perturbation are constructed. This problem is assumed to be a typical pursuit-evasion problem in differential game theory; using boundary constraint conditions and a performance index function that includes time and fuel consumption, the saddle point solution corresponding to the bilateral optimal is derived, and then this pursuit-evasion problem is transformed into a two-point boundary value problem. A hybrid optimisation method using a genetic algorithm (GA) and sequential quadratic programming (SQP) is derived to obtain the optimal control strategy. The proposed model and algorithm are proved to be feasible for the given simulation cases.

Worst-Case Scenario Evasive Strategies in a Two-on-One Engagement Between Dubins’ Vehicles With Partial Information

In this letter, we consider the problem of pursuit-evasion between two pursuers against an evader with the agents modelled as Dubins’ vehicle and having partial information regarding each other’s motion. A novel, two-phase, evasive strategy based on worst-case scenario planning and proximity-based maneuvers is proposed. In the first phase, the evader assumes the pursuers to be holonomic and executes a best response strategy. An analytic and geometric approach is used to obtain the evader’s strategy. In the second phase, the evader switches to movement along high-curvature paths to side-step the pursuers when they are in proximity of the evader in an attempt to delay capture. We also propose an approximation to compute the evader strategy in the first phase, which enables it to be applicable against multiple non-holonomic pursuers. Illustrative simulation results are given.

Orbital Interception Pursuit Strategy for Random Evasion Using Deep Reinforcement Learning

Aiming at the interception problem of noncooperative evader spacecraft adopting random maneuver strategy in one-to-one orbital pursuit–evasion problem, an interception strategy with decision-making training mechanism for the pursuer based on deep reinforcement learning is proposed. Its core purpose is to improve the success rate of interception in the environment with high uncertainty. First of all, a multi-impulse orbit transfer model of pursuer and evader is established, and a modular deep reinforcement learning training method is built. Second, an effective reward mechanism is proposed to train the pursuer to choose the impulse direction and impulse interval of the orbit transfer and to learn the successful interception strategy with the optimal fuel and time. Finally, with the evader taking a random maneuver decision in each episode of training, the trained decision-making strategy is applied to the pursuer, the corresponding interception success rate of which is further analyzed. The results show that the pursuer trained can obtain universal and variable interception strategy. In each round of pursuit–evasion, with random maneuver strategy of the evader, the pursuer can adopt similar optimal decisions to deal with high-dimensional environments and thoroughly random state space, maintaining high interception success rate.

State-Following-Kernel-Based Online Reinforcement Learning Guidance Law Against Maneuvering Target

In this article, a state-following-kernel-based reinforcement learning method with an extended disturbance observer is proposed, whose application to a missile-target interception system is considered. First, the missile-target engagement is formulated as a vertical planar pursuit–evasion problem. The target maneuver is then estimated by an extended disturbance observer in real time, which leads to an infinite-horizon optimal regulation problem. Next, utilizing the local state approximation ability of state-following kernels, the critic neural network (NN) and actor NN for synchronous iteration are constructed to calculate the approximate optimal guidance policy. The states and NN weights are proven to be uniformly ultimately bounded using the Lyapunov method. Finally, numerical simulations against different types of nonstationary targets are effectively tested, and the results highlight the role of state-following kernels in the value function and policy approximation.

Optimal-Damage-Effectiveness Cooperative-Control Strategy for the Pursuit–Evasion Problem with Multiple Guided Missiles

In this paper, an optimal-damage-effectiveness cooperative-control strategy based on a damage-efficiency model and a virtual-force method is proposed to solve the pursuit-evasion problem with multiple guided missiles. Firstly, different from the overly ideal assumption in the traditional pursuit-evasion problem, an optimization problem that maximizes the damage efficiency is established and solved, making the optimal-damage-effectiveness strategy more meaningful for practical applications. Secondly, a modified virtual-force method is proposed to obtain this optimal-damage-effectiveness control strategy, which solves the numerical solution challenges brought by the high-complexity damage function. Thirdly, adaptive gain is designed in this strategy based on guidance-integrated fuze technology to achieve robust maximum damage efficiency in unpredictable interception conditions. Finally, the effectiveness and robustness of the proposed strategy are verified by numerical simulations.

Cooperative strategy for pursuit-evasion problem with collision avoidance

This paper presents a self-organizing cooperative pursuit strategy for multiple unmanned surface vessels (USVs) to pursuit an intelligent evader in the open water. Firstly, the escape strategies of intelligent evaders under different encirclement states and the distribution of pursuers formed according to Apollonius circle are described. Then, in order to achieve the conditions of successful encirclement, a flexible self-organizing cooperation strategy is proposed to form the encirclement formation. In order to better achieve the obstacles encountered in the process of pursuing in the actual environment, the obstacle avoidance methods of irregular obstacles are presented. The simulation results show that the proposed strategy can enable the pursuit USV swarm to capture the intelligent evader, and the pursuit swarm has good obstacle avoidance performance and flexibility.

Pursuit–Evasion Problem of Unmanned Surface Vehicles in a Complex Marine Environment

In order to solve the interception problem of escaping targets under environmental interference and inaccurate target perception, this paper introduces the guidance interception algorithm and introduces a Kalman filter to predict the trajectory of escaping targets. In order to make a USV pursuit under a disturbance in a time-varying real marine environment, a disturbance observer is proposed to estimate the external disturbance and model uncertainty. The simulation of pursuit–evasion results shows that the interception algorithm designed enables the pursuer to intercept the escaping target in a time-varying, disturbed marine environment. Both the simulation and experimental results are provided to validate the effectiveness of the method.

Pursuit-evasion problems under nonlinear increase of the pursuer's resource

MSC: 49N70 DOI: 10.21538/0134-4889-2022-28-3-285-295 In the paper, we investigate pursuit-evasion problems in a simple motion differential game with two players, termed a pursuer and an evader. We put different kinds of non-stationary integral constraints, which restrict the energy consumption rate of the players. On the other hand, it is assumed that at each time the players

A Game-Learning-Based Smooth Path Planning Strategy for Intelligent Air–Ground Vehicle Considering Mode Switching

Numerous missions in both civil and military fields involve the pursuit-evasion problem of vehicles. With vertical take-off and landing capability, the intelligent air–ground vehicle expands its feasible path to 3-D space, which has great advantages in the pursuit. This vehicle requires adequate path planning to obtain an optimal 3-D path and further improve the pursuit efficiency. The planning process of the air–ground vehicle currently faces the technical difficulties of acquiring the proper takeoff timing and position while optimizing the planning trajectory, especially in a complex environment with dense obstacles. To solve the above issues, a game-learning-based smooth path planning strategy for the intelligent air–ground vehicle considering mode switching is proposed in this article. First, a new reward function of the <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"> <tex-math notation="LaTeX">$Q$ </tex-math></inline-formula> -learning algorithm, considering the influence of flight obstacle crossing parameters, is presented to explore the short forward track distance. Second, in the update rule, the pursuit-evasion game acts in the mode switching decisions. During interactive learning between the vehicle and environment, this game constantly updates the Nash equilibrium solutions for mode switching and gets a series of switching decisions of the pursuer vehicle (ego vehicle). Third, a double-yaw correction for path smoothing modification is proposed to reduce turning points and avoid local path deviations. This modification provides heuristic information for the exploration of the environment, which significantly speeds up the convergence speed of the algorithm. Finally, the proposed strategy is verified on a 1000 m*1000 m map with 0–200 m obstacle height. Results show that this strategy is effective to decrease the 253-m distance compared with the traditional reinforcement learning algorithm and has a faster convergence speed. The number of trajectory direction changes is 36% less than that of the game-learning algorithm only considering mode switching. The unreasonable large angle turns are eliminated.

Numerical solution for elliptical orbit pursuit-evasion game via deep neural networks and pseudospectral method

This paper presents an efficient and stable DNNs-based Radau pseudospectral method for the free-time elliptical orbit pursuit-evasion game based on the equivalent reconstruction of the game model. Firstly, the relative dynamics equations are established by adding the nonlinear terms caused by the eccentricity to the Hill–Clohessy–Wilshire equations. Then the original pursuit-evasion problem is deduced to a 4-dimensional one-sided optimal control problem (OCP) based on the equivalent reconstruction. Secondly, in order to apply the deep neural networks (DNNs) to map the relationship between the OCP and the solution, the normalization of costates is introduced to eliminate the non-uniqueness of solutions when generating samples for training DNNs. Thirdly, the DNNs-based Radau pseudospectral method is proposed where the DNNs output the guesses of solutions to the derived OCP and the Radau pseudospectral method optimizes the histories of controls obtained by the guesses to the convergence. The simulation results demonstrate that the proposed method converges more stably and decreases the calculation time greatly compared with the traditional indirect method.