Two-player Zero-sum Differential Game Research Articles

Approximating the Value of Zero-Sum Differential Games with Linear Payoffs and Dynamics

We consider two-player zero-sum differential games of fixed duration, where the running payoff and the dynamics are both linear in the controls of the players. Such games have a value, which is determined by the unique viscosity solution of a Hamilton–Jacobi-type partial differential equation. Approximation schemes for computing the viscosity solution of Hamilton–Jacobi-type partial differential equations have been proposed that are valid in a more general setting, and such schemes can of course be applied to the problem at hand. However, such approximation schemes have a heavy computational burden. We introduce a discretized and probabilistic version of the differential game, which is straightforward to solve by backward induction, and prove that the solution of the discrete game converges to the viscosity solution of the partial differential equation, as the discretization becomes finer. The method removes part of the computational burden of existing approximation schemes.

Optimal [formula omitted] tracking control of nonlinear systems with zero-equilibrium-free via novel adaptive critic designs

In this paper, a novel adaptive critic control method is designed to solve an optimal H∞ tracking control problem for continuous nonlinear systems with nonzero equilibrium based on adaptive dynamic programming (ADP). To guarantee the finiteness of a cost function, traditional methods generally assume that the controlled system has a zero equilibrium point, which is not true in practical systems. In order to overcome such obstacle and realize H∞ optimal tracking control, this paper proposes a novel cost function design with respect to disturbance, tracking error and the derivative of tracking error. Based on the designed cost function, the H∞ control problem is formulated as two-player zero-sum differential games, and then a policy iteration (PI) algorithm is proposed to solve the corresponding Hamilton–Jacobi–Isaacs (HJI) equation. In order to obtain the online solution to the HJI equation, a single-critic neural network structure based on PI algorithm is established to learn the optimal control policy and the worst-case disturbance law. It is worth mentioning that the proposed adaptive critic control method can simplify the controller design process when the equilibrium of the systems is not zero. Finally, simulations are conducted to evaluate the tracking performance of the proposed control methods.

Synergetic learning structure-based neuro-optimal fault tolerant control for unknown nonlinear systems

In this paper, a synergetic learning structure-based neuro-optimal fault tolerant control (SLSNOFTC) method is proposed for unknown nonlinear continuous-time systems with actuator failures. Under the framework of the synergetic learning structure (SLS), the optimal control input and the actuator failure are viewed as two subsystems. Then, the fault tolerant control (FTC) problem can be regarded as a two-player zero-sum differential game according to the game theory. A radial basis function neural network-based identifier, which uses the measured input/output data, is constructed to identify the completely unknown system dynamics. To develop the SLSNOFTC method, the Hamilton–Jacobi–Isaacs equation is solved by an asymptotically stable critic neural network (ASCNN) which is composed of cooperative adaptive tuning laws. Besides, with the help of the Lyapunov stability analysis, the identification error, the weight error of ASCNN, and all signals of closed-loop system are guaranteed to be converged to zero asymptotically, rather than uniformly ultimately bounded. Numerical simulation examples further verify the effectiveness and reliability of the proposed method.

State and Control Path-Dependent Stochastic Zero-Sum Differential Games: Viscosity Solutions of Path-Dependent Hamilton–Jacobi–Isaacs Equations

In this paper, we consider the two-player state and control path-dependent stochastic zero-sum differential game. In our problem setup, the state process, which is controlled by the players, is dependent on (current and past) paths of state and control processes of the players. Furthermore, the running cost of the objective functional depends on both state and control paths of the players. We use the notion of non-anticipative strategies to define lower and upper value functionals of the game, where unlike the existing literature, these value functions are dependent on the initial states and control paths of the players. In the first main result of this paper, we prove that the (lower and upper) value functionals satisfy the dynamic programming principle (DPP), for which unlike the existing literature, the Skorohod metric is necessary to maintain the separability of càdlàg (state and control) spaces. We introduce the lower and upper Hamilton–Jacobi–Isaacs (HJI) equations from the DPP, which correspond to the state and control path-dependent nonlinear second-order partial differential equations. In the second main result of this paper, we show that by using the functional Itô calculus, the lower and upper value functionals are viscosity solutions of (lower and upper) state and control path-dependent HJI equations, where the notion of viscosity solutions is defined on a compact κ-Hölder space to use several important estimates and to guarantee the existence of minimum and maximum points between the (lower and upper) value functionals and the test functions. Based on these two main results, we also show that the Isaacs condition and the uniqueness of viscosity solutions imply the existence of the game value. Finally, we prove the uniqueness of classical solutions for the (state path-dependent) HJI equations in the state path-dependent case, where its proof requires establishing an equivalent classical solution structure as well as an appropriate contradiction argument.

A Fuel Optimization Method for the Pursuer in the Spacecraft Pursuit-Evasion Game

AbstractThe fuel optimization of the spacecraft is significant for its service life. In this paper, a fuel optimization method is proposed to solve the fuel optimization problem of the pursuer in the spacecraft pursuit-evasion game (SPE game). This method first takes both the magnitude and direction of the pursuer's thrust as the control variables to be optimized, then models the fuel optimization problem as a two-player zero-sum differential game whose payoff function is the fuel consumption of the pursuer, and finally obtains the optimal control strategy of the pursuer by solving the saddle point of the differential game. Considering that the solving of the saddle point usually leads to the solving of a high-dimensional (12-dimensional in this paper) two-point boundary value problem (TPBVP), which is very challenging, the proposed method adopts a dimension-reduction algorithm which can simplify the TPBVP to the solving of a four-dimensional nonlinear equations and a terminal constraint, and then presents an hybrid numerical algorithm combining the particle swarm optimization algorithm and the Powell algorithm to solve the nonlinear equations and terminal constraint. The simulation results show that the proposed method can achieve the fuel optimization of the pursuer in the SPE game and is robust to the orbital altitude, the initial relative state between the pursuer and evader and the magnitude of the evader's thrust.

Lipschitz estimates for partial trace operators with extremal Hessian eigenvalues

Abstract We consider the Dirichlet problem for partial trace operators which include the smallest and the largest eigenvalue of the Hessian matrix. It is related to two-player zero-sum differential games. No Lipschitz regularity result is known for the solutions, to our knowledge. If some eigenvalue is missing, such operators are nonlinear, degenerate, non-uniformly elliptic, neither convex nor concave. Here we prove an interior Lipschitz estimate under a non-standard assumption: that the solution exists in a larger, unbounded domain, and vanishes at infinity. In other words, we need a condition coming from far away. We also provide existence results showing that this condition is satisfied for a large class of solutions. On the occasion, we also extend a few qualitative properties of solutions, known for uniformly elliptic operators, to partial trace operators.

Adaptive Learning Based Output-Feedback Optimal Control of CT Two-Player Zero-Sum Games

Although optimal control with full state-feedback has been well studied, online solving output-feedback optimal control problem is difficult, in particular for learning online Nash equilibrium solution of the continuous-time (CT) two-player zero-sum differential games. For this purpose, we propose an adaptive learning algorithm to address this trick problem. A modified game algebraic Riccati equation (MGARE) is derived by tailoring its state-feedback control counterpart. An adaptive online learning method is proposed to approximate the solution to the MGARE through online data, where two operations (i.e., vectorization and Kronecker’s product) can be adopted to reconstruct the MGARE. Only system output information is needed to implement developed learning algorithm. Simulation results are carried out to exemplify the proposed control and learning method.

Zero-sum game-based neuro-optimal control of modular robot manipulators with uncertain disturbance using critic only policy iteration

In this paper, a zero-sum differential game strategy-based neuro-optimal control method is presented via critic only policy iteration-adaptive dynamic programming (COPI-ADP) approach to address optimal trajectory tracking control problem of modular robot manipulators (MRMs) with uncertain disturbance. The dynamic model of modular robot manipulator systems is formulated as an integration of joint subsystems and unknown robotic model uncertainties are identified by the developed linear extension state observer. Then, the optimal control issue of the modular robot manipulator systems with uncertain disturbance is transformed into a two-player zero-sum differential game one. Based on adaptive dynamic programming and policy iteration algorithms, the Hamilton-Jacobi-Issacs (HJI) equation is approximately solved using only critic neural network and thus facilitating the feasible derivation of the approximated optimal control policy. The trajectory of tracking errors of modular robot manipulator system is guaranteed to be uniform ultimate bounded by using the Lyapunov theory. Finally, experiments are provided to demonstrate the advantage and effectiveness of the developed control method.

Robust multi-agent differential games with application to cooperative guidance

Robust multi-agent differential games and its application to cooperative guidance are investigated in this paper. The multi-agent system is modeled by general linear dynamics with norm-bound parameter uncertainties. The players in the game are divided into two teams. One team consists of a leader, and the other team consists of a fixed number of followers agents that aim to achieve leader-following consensus. The two teams constitute the adversaries of a two-player zero-sum differential game. Based on quadratic stabilization techniques, a set of saddle point strategies of the game are designed to stabilize the closed-loop multi-agent system. By modifying the optimal solution of the multi-agent differential game of the nominal model, the sufficient conditions are presented such that the stabilization of the uncertain system is guaranteed. It is proved that the modified solution achieves quasi-optimality. It is also shown that L2-bounded consensus error can be achieved, if the control of the leader deviates from its quasi-optimal strategy. The model of cooperative guidance problem is established, based on which the robust multi-agent differential games approach is applied to design the cooperative guidance laws. Numerical examples are given to verify the effectiveness of the theoretical results and the performance of the proposed cooperative guidance law.

Event-triggered distributed self-learning robust tracking control for uncertain nonlinear interconnected systems

In this paper, a novel event-triggered self-learning robust tracking control scheme for a class of nonlinear interconnected systems with uncertain interconnections is developed based on adaptive dynamic programming (ADP). Initially, the robust tracking control problem of interconnected systems is transformed to the robust stabilization one of augmented interconnected systems. To address the uncertain interconnections as well as optimize the sample intervals, a group of auxiliary subsystems, each with a novel discounted cost function, are introduced, which enables the robust stabilization problem to be further converted to a group of two-player zero-sum differential games. Next, an event-triggered ADP algorithm is proposed to obtain the saddle point solutions, based on which the distributed event-triggered robust tracking control policies for the overall system are established. Specifically, the proposed event-triggered mechanism is asynchronous and distributed where the sample intervals are optimized to further reduce the computational burden and save the communication resources. Moreover, the tracking errors and the approximation errors of critic network weights are demonstrated to be uniformly ultimately bounded by using the Lyapunov approach. Finally, two simulation examples are provided to verify the effectiveness of the proposed method.

Robust optimal control for finite-horizon zero-sum differential games via a plug-n-play event-triggered scheme

This paper proposes a robust optimal control strategy for finite-horizon two-player Zero-Sum(ZS) differential games with partially unknown dynamics by incorporating an event-triggered scheme and the critic-only adaptive dynamic programming(ADP) method. Firstly, an online identifier is designed to reconstruct unknown system dynamics based on the data-driven technique. The identifier is running in the solving process rather than as a priori part of the solution, which simplifies the system structure and decreases the computational cost. To deal with the finite-horizon constraints, a time-varying value function and a additional term are considered to such that the terminal constraint error is minimised. A critic neural network(CNN) is used to solve the event-triggered Hamilton–Jacob—Isaacs (HJI) equation under a plug-n-play structure, which reduces the redundant information transmission as well as receives all measurement information immediately. According to the Lyapunov theory, the uniformly ultimately bounded (UUB) for the event-triggered closed-loop system and the CNN weight error are demonstrated, in the meantime the asymptotic stability of the identifier weight error is proved. Finally, the application in the missile-target interception system validates the feasibility and efficacy of the proposed method.

Event‐triggered distributed H ∞ control of physically interconnected mobile Euler–Lagrange systems with slipping, skidding and dead zone

This study addresses an event-triggered distributed ℋ ∞ control method by extending traditional zero-sum differential games for physically interconnected non-holonomic mobile mechanical multi-agent systems with external disturbance and slipping, skidding and dead-zone disturbances. Initially, a problem of physically interconnected kinematic and dynamic control is transformed into an equivalent problem of event-triggered distributed ℋ ∞ control. Subsequently, the traditional two-player zerosum differential game is extended to a three-player zero-sum differential game, where a new player is included to approximate the worst dead-zone disturbance. To find player policies, an event-triggering condition and an event-triggered control law are proposed via neural networks (NNs). Although an NN weight-tuning law is designed on the basis of adaptive dynamic programming techniques, it can relax identification procedures for unknown drift dynamics and persistent excitation conditions. It also guarantees that the closed system is stable and the cost function converges to the bounded ℒ 2 -gain optimal value, while the Zeno behaviour is excluded. Finally, the effectiveness of the proposed method is verified by an application to a dead-zone torque multi-mobile robot system through numerical simulations.

Adaptive periodic event-triggered control for missile-target interception system with finite-horizon convergence

In this paper, the optimal control problem for finite-time missile-target interception systems is posed in a finite-horizon two-player zero-sum (ZS) differential game framework using a periodic event-triggered (PET) scheme. To solve the optimal control problem, a time-varying Hamilton-Jacobi-Issac (HJI) equation and a time-dependent cost function are constructed to deal with finite-horizon constraints, and an event-based periodic adaptive dynamic programming (ADP) algorithm is employed to find the Nash equilibrium solution for the designed HJI equation. Comparing with the traditional continuous event-triggered (ET) scheme, the proposed PET scheme only verifies the event-triggered conditions at periodic sampling instants, which reduces resource consumption in monitoring and excludes the Zeno behavior. A single critic neural network (CNN) is used to implement the proposed event-based optimal control algorithm, which reduces approximate errors bust also simplifies structures. Further, an additional error term is added in the designed weight updating law to such that the terminal constraint is also minimized over time. By resorting to Lyapunov function approach, some sufficient conditions are derived to achieve the uniformly ultimately bounded (UUB) of the ET closed-loop system and the estimation weight error of CNN. Finally, a missile-target interception system is introduced to illustrate the efficiency of the presented methods.

Event-Driven-Modular Adaptive Backstepping Optimal Control for Strict-Feedback Systems Through Zero-Sum Differential Games

This paper addresses the event-driven-modular optimal tracking control problem for nonlinear strict-feedback systems with external disturbances. Through the backstepping feedforward control, the optimal tracking problem is transformed into an equivalent optimal regulation problem of affine tracking error system. Subsequently, adaptive dynamic programming technique is introduced to generate the optimal feedback controller, and solve the optimization problem of two-player zero-sum differential game. A single critic neural network is constructed to evaluate the associated cost function online, where the novel weight updating law is derived based on the gradient-descent technique. The resulting event-triggered closed-loop system, modeled as an impulsive system, is proved to be asymptotically stable by Lyapunov theory. Finally, the reliability and effectiveness of the theoretical results is validated by numerical simulation examples.

Abnormal and Singular Solutions in the Target Guarding Problem with Dynamics

The topic of this paper is a two-player zero-sum differential game known as the target guarding problem. After a brief review of Isaacs’ original problem and solution, a problem with second-order dynamics and acceleration control is considered. It is shown that there are four solution classes satisfying the necessary conditions. The four classes are (i) abnormal and non-singular, (ii) normal and non-singular, (iii) normal and pursuer singular, (iv) normal and evader singular. The normal and totally singular case is ruled out. Closed-form solutions are provided for cases ii–iv. The order of singularity in all cases is infinite. Thus, the problem exhibits many interesting properties: normality, abnormality, non-singularity, infinite-order singularity, and non-uniqueness. A practical example of each class is provided.

Approximate-optimal control algorithm for constrained zero-sum differential games through event-triggering mechanism

This paper investigates the optimization problem of two-player zero-sum differential game with control constraints in the framework of event triggering. Relying on reinforcement learning, an adaptive dynamic programming algorithm is developed to approximate the optimal solution of zero-sum game, i.e., the saddle-point equilibrium. A single-network structure is adopted, wherein a critic neural network (NN) evaluates the action. First, the constrained Hamilton–Jacobi–Isaacs equation is mathematically derived in the presence of control constraints; the event-triggering mechanism is then incorporated to reduce calculations and actions. Then, based on the gradient-descent technique, a novel weight updating law is designed for the critic NN, which ensures the solution can converge to the optimal value online. Moreover, the stability of closed-loop system is guaranteed and the unfavorable Zeno behavior is excluded by calculating the theoretical minimum triggering interval. Finally, two numerical examples are provided to verify the reliability and effectiveness of proposed algorithm.

Nonlinear Control for Spacecraft Pursuit-Evasion Game Using the State-Dependent Riccati Equation Method

Spacecraft pursuit-evasion game is formulated as a two-player zero-sum differential game. The Hill reference frame is used to describe the dynamics of the game. The goal is to derive feedback control law which forms a saddle point solution to the game. Both spacecraft use continuous-thrust engines. The linear quadratic differential game theory is applied to derive a linear control law. This theory is extended to derive a nonlinear control law using the state-dependent Riccati equation method. A state-dependent coefficient matrix is developed for this purpose. Various scenarios are considered for comparing the performance of the linear and nonlinear laws. In all the scenarios, the efficacy of the nonlinear control law is found to be superior to that of the linear control law with computation cost similar to the linear law.

Differential games, continuous Lyapunov functions, and stabilisation of non‐linear dynamical systems

In this study, the authors address the two-player zero-sum differential game problem for non-linear dynamical systems with non-linear-non-quadratic cost functions over the infinite time horizon. The pursuer's goal is to minimise the cost function and guarantee asymptotic stability of the closed-loop system, whereas the evader's goal is to maximise the cost function. Closed-loop asymptotic stability is certified by continuous Lyapunov functions that are viscosity solutions of the steady-state Hamilton-Jacobi-Isaacs equation for the controlled system. Since it is difficult to find viscosity solutions of partial differential equations for numerous problems of practical interest, they extend an inverse optimality framework to provide explicit closed-form solutions of differential game problems, which involve affine in the controls dynamical systems with quadratic cost functions and linear dynamical systems with Lagrangians in polynomial form. The authors' framework allows also to solve optimal robust control problems involving non-linear dynamical systems with non-linear-non-quadratic cost functionals and provides a generalisation of the mixed-norm ℋ 2 /ℋ ∞ optimal robust control framework. Two numerical examples illustrate the applicability of theoretical results provided.

Robust adaptive critic control design with network-based event-triggered formulation

In this paper, by incorporating the network-based event-triggered formulation, the robust adaptive critic control design for a class of nonlinear continuous-time systems is investigated to fulfill disturbance rejection. First, the designed problem with output information is formulated as a two-player zero-sum differential game and the adaptive critic mechanism is employed toward the event-based minimax optimization involving a suitable triggering condition. Then, the event-based optimal control law and the time-based worst-case disturbance law are learned by training the critic neural network. Besides, the closed-loop system is constructed with stability proof of the critic error dynamics and the sampled-data plant. The theoretical analysis has demonstrated that the infamous Zeno behavior of the proposed event-based adaptive critic design has been avoided. Finally, the developed method is applied toward the robot arm plant, as a mechanical component of the complex robot system, so as to substantiate the performance of disturbance rejection.

H ∞ suboptimal tracking controller design for a class of nonlinear systems

In this paper, a new technique is proposed to solve the H∞ tracking problem for a broad class of nonlinear systems. Towards this end, based on a discounted cost function, a nonlinear two-player zero-sum differential (NTPZSD) game is defined. Then, the problem is converted to another NTPZSD game without any discount factor in its corresponding cost function. A state-dependent Riccati equation (SDRE) technique is applied to the latter NTPZSD game in order to find its approximate solution which leads to obtain a feedback-feedforward control law for the original game. It is proved that the tracking error between the system state and its desired trajectory converges asymptotically to zero under mild conditions on the discount factor. The proposed H∞ tracking controller is applied to two nonlinear systems (the Vander Pol’s oscillator and the insulin-glucose regulatory system of type I diabetic patients). Simulation results demonstrate that the proposed H∞ tracking controller is so effective to solve the problem of tracking time-varying desired trajectories in nonlinear dynamical systems.