Two-player Zero-sum Differential Game Research Articles

Event-Triggered $H_\infty $ Control for Continuous-Time Nonlinear System via Concurrent Learning

In this paper, the ${H}_{\infty }$ optimal control problem for a class of continuous-time nonlinear systems is investigated using event-triggered method. First, the ${H}_{\infty }$ optimal control problem is formulated as a two-player zero-sum (ZS) differential game. Then, an adaptive triggering condition is derived for the ZS game with an event-triggered control policy and a time-triggered disturbance policy. The event-triggered controller is updated only when the triggering condition is not satisfied. Therefore, the communication between the plant and the controller is reduced. Furthermore, a positive lower bound on the minimal intersample time is provided to avoid Zeno behavior. For implementation purpose, the event-triggered concurrent learning algorithm is proposed, where only one critic neural network (NN) is used to approximate the value function, the control policy and the disturbance policy. During the learning process, the traditional persistence of excitation condition is relaxed using the recorded data and instantaneous data together. Meanwhile, the stability of closed-loop system and the uniform ultimate boundedness (UUB) of the critic NN’s parameters are proved by using Lyapunov technique. Finally, simulation results verify the feasibility to the ZS game and the corresponding ${H}_{\infty }$ control problem.

Intelligent Critic Control With Disturbance Attenuation for Affine Dynamics Including an Application to a Microgrid System

In this paper, a computationally efficient framework for intelligent critic control design and application of continuous-time input-affine systems is established with the purpose of disturbance attenuation. The described problem is formulated as a two-player zero-sum differential game and the adaptive critic mechanism with intelligent component is employed to solve the minimax optimization problem. First, a neural identifier is developed to reconstruct the unknown dynamical information incorporating stability analysis. Next, the optimal control law and the worst-case disturbance law are designed by introducing and tuning a critic neural network. Moreover, the closed-loop system is proved to possess the uniform ultimate boundedness. At last, the present method is applied to a smart microgrid and then is further adopted to control a general nonlinear system via simulation, thereby substantiating the performance of disturbance attenuation.

On Mixed Data and Event Driven Design for Adaptive-Critic-Based Nonlinear $H_{\infty}$ Control.

In this paper, based on the adaptive critic learning technique, the control for a class of unknown nonlinear dynamic systems is investigated by adopting a mixed data and event driven design approach. The nonlinear control problem is formulated as a two-player zero-sum differential game and the adaptive critic method is employed to cope with the data-based optimization. The novelty lies in that the data driven learning identifier is combined with the event driven design formulation, in order to develop the adaptive critic controller, thereby accomplishing the nonlinear control. The event driven optimal control law and the time driven worst case disturbance law are approximated by constructing and tuning a critic neural network. Applying the event driven feedback control, the closed-loop system is built with stability analysis. Simulation studies are conducted to verify the theoretical results and illustrate the control performance. It is significant to observe that the present research provides a new avenue of integrating data-based control and event-triggering mechanism into establishing advanced adaptive critic systems.

Differential games, partial-state stabilization, and model reference adaptive control

In this paper, we develop a unified framework to find state-feedback control laws that solve two-player zero-sum differential games over the infinite time horizon and guarantee partial-state asymptotic stability of the closed-loop system. Partial-state asymptotic stability is guaranteed by means of a Lyapunov function that is positive definite and decrescent with respect to part of the system state. The existence of a saddle point for the system׳s performance measure is guaranteed by the fact that this Lyapunov function satisfies a partial differential equation that corresponds to a steady-state form of the Hamilton–Jacobi–Isaacs equation. In the second part of this paper, we show how our differential game framework can be applied not only to solve pursuit-evasion and robust optimal control problems, but also to assess the effectiveness of a model reference adaptive control law. Specifically, the model reference adaptive control architecture is designed to guarantee satisfactory trajectory tracking for uncertain nonlinear dynamical systems, whose matched nonlinearities are captured by the regressor vector. By modeling matched and unmatched nonlinearities, which are not captured by the regressor vector, as the pursuer׳s and evader׳s control inputs in a differential game, we provide an explicit characterization of the system׳s uncertainties that do not disrupt the trajectory tracking capabilities of the adaptive controller. Two numerical examples illustrate the applicability of our results.

Robust differential game guidance laws design for uncertain interceptor-target engagement via adaptive dynamic programming

ABSTRACTIn this paper, the problem of intercepting a maneuvering target is formulated as a two-player zero-sum differential game framework affected by matched uncertainties. By introducing an appropriate cost function that reflects the uncertainties, the robust control is transformed into a two-player zero-sum differential game control problem and therefore ensures the compensation of the matched uncertainties. Additionally, the corresponding Hamilton--Jacobi--Isaacs (HJI) equation is solved by constructing a critic neural network (NN). The closed-loop system and the critic NN weight estimation error are proved to be uniform ultimate boundedness (UUB) by utilising Lyapunov approach. Finally, the effectiveness of the proposed robust guidance law is demonstrated by using a nonlinear two-dimensional kinematics, assuming first-order dynamics for the interceptor and the target.

Neural network-based online H∞ control for discrete-time affine nonlinear system using adaptive dynamic programming

In this paper, the problem of H∞ control design for affine nonlinear discrete-time systems is addressed by using adaptive dynamic programming (ADP). First, the nonlinear H∞ control problem is transformed into solving the two-player zero-sum differential game problem of the nonlinear system. Then, the critic, action and disturbance networks are designed by using neural networks to solve online the Hamilton–Jacobi–Isaacs (HJI) equation associating with the two-player zero-sum differential game. When novel weight update laws for the critic, action and disturbance networks are tuned online by using data generated in real-time along the system trajectories, it is shown that the system states, all neural networks weight estimation errors are uniformly ultimately bounded by using Lyapunov techniques. Further, it is shown that the output of the action network approaches the optimal control input with small bounded error and the output of the disturbance network approaches the worst disturbance with small bounded error. At last, simulation results are presented to demonstrate the effectiveness of the new ADP-based method.

Differential games with asymmetric information and without Isaacs’ condition

We investigate a two-player zero-sum differential game with asymmetric information on the payoff and without Isaacs condition. The dynamics is an ordinary differential equation parametrised by two controls chosen by the players. Each player has a private information on the payoff of the game, while his opponent knows only the probability distribution on the information of the other player. We show that a suitable definition of random strategies allows to prove the existence of a value in mixed strategies. Moreover, the value function can be characterised in term of the unique viscosity solution in some dual sense of a Hamilton-Jacobi-Isaacs equation. Here we do not suppose the Isaacs condition which is usually assumed in differential games.

Differential Games with Incomplete Information on a Continuum of Initial Positions and without Isaacs Condition

This article deals with a two-player zero-sum differential game with infinitely many initial positions and without Isaacs condition. The structure of information is asymmetric: The first player has a private information on the initial position, while the second player knows only a probability distribution on the initial position. In the present model, we face two difficulties: First, the incomplete information structure does not reduce to a finite set (as in the famous Aumann–Maschler model for repeated games). Second, the game does not satisfy the Isaacs condition (crucially used in classical approaches to differential games). Therefore, we use tools from optimal transportation theory and stochastic control. Our main result shows that with a suitable concept of mixed strategies, there exists a value of the game with such random strategies. As a byproduct of our approach, we obtain the Lipschitz continuity of the random value with respect to the Wasserstein distance and we show the existence of a value in pure strategies in the specific case of an initial distribution without atoms. We also discuss an extension of our model when the asymmetric information concerns continuous scenarios.

Integral Reinforcement Learning for Linear Continuous-Time Zero-Sum Games With Completely Unknown Dynamics

In this paper, we develop an integral reinforcement learning algorithm based on policy iteration to learn online the Nash equilibrium solution for a two-player zero-sum differential game with completely unknown linear continuous-time dynamics. This algorithm is a fully model-free method solving the game algebraic Riccati equation forward in time. The developed algorithm updates value function, control and disturbance policies simultaneously. The convergence of the algorithm is demonstrated to be equivalent to Newton's method. To implement this algorithm, one critic network and two action networks are used to approximate the game value function, control and disturbance policies, respectively, and the least squares method is used to estimate the unknown parameters. The effectiveness of the developed scheme is demonstrated in the simulation by designing an H <sub xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">∞</sub> state feedback controller for a power system.

Stochastic Perron's Method and Elementary Strategies for Zero-Sum Differential Games

We develop here the stochastic Perron method in the framework of two-player zero-sum differential games. We consider the formulation of the game where both players play, symmetrically, feedback strategies as opposed to the Elliott--Kalton formulation prevalent in the literature. The class of feedback strategies we use is carefully chosen so that the state equation admits strong solutions and the technicalities involved in the stochastic Perron method carry through in a rather simple way. More precisely, we define the game over elementary strategies, which are well motivated by intuition. Within this framework, the stochastic Perron method produces a viscosity subsolution of the upper Isaacs equation dominating the upper value of the game, and a viscosity supersolution of the upper Isaacs equation lying below the upper value of the game. Using a viscosity comparison result we obtain that the upper value is the unique and continuous viscosity solution of the upper Isaacs equation. An identical statement holds true for the lower value and the lower Isaacs equation. A version of the dynamic programming principle is obtained as a byproduct. If the Isaacs condition is satisfied, the game has a value over elementary (pure) strategies.

Online Partially Model-Free Solution of Two-Player Zero Sum Differential Games

An online adaptive dynamic programming based iterative algorithm is proposed for a two-player zero sum linear differential game problem arising in the control of process systems affected by disturbances. The objective in such a scenario is to obtain an optimal control policy that minimizes the specified performance index or cost function in presence of worst case disturbance. Conventional algorithms for the solution of such problems require full knowledge of system dynamics. The algorithm proposed in this paper is partially model-free and solves the two-player zero sum linear differential game problem without knowledge of state and control input matrices.

Game theory approach to state and input simultaneous estimation for discrete-time LTV systems

This article aims to provide a simple approach realising state observation and input estimation simultaneously for discrete-time LTV systems in H∞ setting. Through solving a two-player zero sum differential game, appealing results are obtained in two folds. First, necessary and sufficient solvability conditions for state and input simultaneous estimation problem are given in terms of solution to a set of difference Riccati recursion. Second, one estimator is presented with special innovation structure, where innovation information is used to update state observation tuned by gain matrix and to provide input estimation through a projector matrix, where gain matrix and projector matrix are constructed from solution to difference Riccati recursion. At last, simulation results are provided to justify proposed approach.

Adaptive dynamic programming for online solution of a zero-sum differential game

This paper will present an approximate/adaptive dynamic programming (ADP) algorithm, that uses the idea of integral reinforcement learning (IRL), to determine online the Nash equilibrium solution for the two-player zerosum differential game with linear dynamics and infinite horizon quadratic cost. The algorithm is built around an iterative method that has been developed in the control engineering community for solving the continuous-time game algebraic Riccati equation (CT-GARE), which underlies the game problem. We here show how the ADP techniques will enhance the capabilities of the offline method allowing an online solution without the requirement of complete knowledge of the system dynamics. The feasibility of the ADP scheme is demonstrated in simulation for a power system control application. The adaptation goal is the best control policy that will face in an optimal manner the highest load disturbance.

Numerical Solution of Orbital Combat Games Involving Missiles and Spacecraft

This research addresses the problem of the optimal interception of an optimally evasive orbital target by a pursuing spacecraft or missile. The time for interception is to be minimized by the pursuing space vehicle and maximized by the evading target. This problem is modeled as a two-sided optimization problem, i.e. as a two-player zero-sum differential game. The work incorporates a recently developed method, termed “semi-direct collocation with nonlinear programming”, for the numerical solution of dynamic games. The method is based on the formal conversion of the two-sided optimization problem into a single-objective one, by employing the analytical necessary conditions for optimality related to one of the two players. An approximate, first attempt solution for the method is provided through the use of a genetic algorithm in a “preprocessing” phase. Three qualitatively different cases are considered. In the first example the pursuer and the evader are represented by two spacecraft orbiting the Earth in two distinct orbits. The second and the third case involve two missiles, and a missile that pursues an orbiting spacecraft, respectively. The numerical results achieved in this work testify to the robustness and effectiveness of the method also in solving large, complex, three-dimensional problems.

Hybrid Estimation of State and Input for Linear Discrete Time-varying Systems: A Game Theory Approach

The H ∞ hybrid estimation problem for linear discrete time-varying systems is investigated in this paper, where estimated signals are linear combination of state and input. Design objective requires the worst-case energy gain from disturbance to estimation error to be less than a prescribed level. Optimal solution of the hybrid estimation problem is the saddle point of a two-player zero sum differential game. On the basis of the differential game approach, necessary and sufficient solvable conditions for the hybrid estimation problem are provided in terms of solutions to a Riccati differential equation. Moreover, one possible estimator is proposed if the solvable conditions are satisfied. The estimator is characterized by a gain matrix and an output mapping matrix, where the latter reflects the internal relations between unknown input and output estimation error. At last, a numerical example is provided to illustrate the proposed approach.

Representations Formulas for Some Differential Games with Asymmetric Information

We compute the value of several two-player zero-sum differential games in which the players have an asymmetric information on the random terminal payoff.

Differential Games with Asymmetric Information

We investigate a two-player zero-sum differential game in which the players have asymmetric information on the random terminal payoff. We prove that the game has a value and characterize this value in terms of dual solutions of some Hamilton–Jacobi equation. We also explain how to adapt the results to differential games where the initial position is random.

Existence of Value and Saddle Point in Infinite-Dimensional Differential Games

We study two-player zero-sum differential games of finite duration in a Hilbert space. Following the Berkovitz notion of strategies, we prove the existence of value and saddle-point equilibrium. We characterize the value as the unique viscosity solution of the associated Hamilton–Jacobi–Isaacs equation using dynamic programming inequalities.

Convexity in Zero-Sum Differential Games

A new approach to two-player zero-sum differential games with convex-concave cost function is presented. It employs the tools of convex and variational analysis. A necessary and sufficient condition on controls to be an open-loop saddle point of the game is given. Explicit formulas for saddle controls are derived in terms of the subdifferential of the function conjugate to the cost. Existence of saddle controls is concluded under very general assumptions, not requiring the compactness of control sets. A Hamiltonian inclusion, new to the field of differential games, is shown to describe equilibrium trajectories of the game.

Parameter Optimization for Two-Player Zero-Sum Differential Games

In addition to the classical formulation of a general two-player zero-sum differential game it is assumed that the minimizing player chooses the values of certain parameters to further decrease the pay-off functional at the expense of the maximizing player. To solve this problem in the best possible way for the minimizing player, a set of necessary optimality conditions is derived, which not only enable the determination of the saddle-point strategies for both participating players, but also the optimal parameters. Based on these conditions an iterative numerical algorithm of gradient type is suggested. As an illustration, several examples are solved applying the described algorithm.