Zero-sum Differential Game Research Articles

ADP-Based Decentralized Load Frequency Control Schemes to Multiarea Asynchronous Markov Jumping Power Systems With Experience Replay.

This article investigates the problem of robust decentralized load frequency control (LFC) in multiarea interconnection power systems, in the presence of the external disturbances and stochastic abrupt variations, such as component failures and different load demands. To capture the different operating conditions of the load in the multiarea power systems, a Markov superposition technique is skillfully employed to model the system's component matrices. To solve the Nash equilibrium solution of the zero-sum differential game problem, an improved online adaptive dynamic programming (ADP) algorithm based on the experience replay technique (ERT) is developed, which addresses the nonlinear coupling difficulties encountered in solving the game algebraic Riccati equations (Game AREs). The proposed algorithm weakens the condition that traditional policy iteration online solutions of zero-sum games require an initial stabilizing control policy pair, and considers the fact that only the transition probability matrix is known, while the prior knowledge of the dynamics is completely unknown. The algorithm is used to obtain both a minimum LFC and maximum disturbance policy for a given disturbance rejection level. Additionally, a safer LFC controller of the power systems with the Markov jumping parameters is designed. The stability and convergence of the proposed algorithm are demonstrated. Finally, the effectiveness and good performance of the proposed design method are validated using a two-area four-mode simulation example.

A novel hybrid LFC scheme for multi-area interconnected power systems considering coupling attenuation

In this paper, a hybrid load frequency control (LFC) scheme is proposed for multi-area interconnected power systems to decouple the intricate double control objectives, by dividing all subareas into the responsible areas and the free areas. The LFC in the responsible area has the function of regulating both the local frequency and the tie-line power, while the control objective of the LFC in the free area is thus simplified to regulate the local frequency only. Then, addressing the complex network coupling and uncertain dynamics, an integrated LFC controller is proposed for the free areas, which consists of two parts, namely, the coupling attenuation baseline controller and the disturbance compensation controller. The coupling attenuation baseline controller satisfying the predefined bounded L2-Gain condition is derived based on the solution to a multi-player zero-sum differential game. Additionally, a novel generalized integral observer is designed to estimate the system’s integrated disturbance, and the corresponding disturbance compensation controller is derived. After that, the ultimately uniformly bounded (UUB) stability of the integrated LFC controller combining baseline controller and disturbance compensation controller is proven rigorously. Finally, the performance superiority of the proposed hybrid LFC scheme is validated by the simulations in challenging operating modes.

Dynamic event-driven optimal consensus control for state-constrained multiagent zero-sum differential graphical games

In this paper, a dynamic event-driven optimal control scheme is proposed for the zero-sum differential graphical games in nonlinear multiagent systems with full-state constraints. Initially, to address the dual demands of optimality and state constraints, a set of system transformation functions are introduced to satisfy the state constraints of the agents. Then, by applying the principle of differential game theory, the distributed optimal control problem affected by external disturbances is formulated as a zero-sum differential graphical game, and the performance index function related to neighbor informations and disturbances is designed for each follower. Afterwards, to enhance the utilization of communication resource, a novel dynamic event-triggered mechanism characterized by a dynamic threshold parameter and an auxiliary dynamic variable is developed, which not only exhibits greater flexibility but also diminishes the frequency of triggers. Furthermore, the approximate optimal control strategies are obtained by employing an event-driven adaptive dynamic programming algorithm. Ultimately, a simulation example is presented to verify the applicability of the proposed control approach.

Decentralized Control for Large-Scale Systems With Actuator Faults and External Disturbances: A Data-Driven Method.

This article investigates optimal control for a class of large-scale systems using a data-driven method. The existing control methods for large-scale systems in this context separately consider disturbances, actuator faults, and uncertainties. In this article, we build on such methods by proposing an architecture that accommodates simultaneous consideration of all of these effects, and an optimization index is designed for the control problem. This diversifies the class of large-scale systems amenable to optimal control. We first establish a min-max optimization index based on the zero-sum differential game theory. Then, by integrating all the Nash equilibrium solutions of the isolated subsystems, the decentralized zero-sum differential game strategy is obtained to stabilize the large-scale system. Meanwhile, by designing adaptive parameters, the impact of actuator failure on the system performance is eliminated. Afterward, an adaptive dynamic programming (ADP) method is utilized to learn the solution of the Hamilton-Jacobi-Isaac (HJI) equation, which does not need the prior knowledge of system dynamics. A rigorous stability analysis shows that the proposed controller asymptotically stabilizes the large-scale system. Finally, a multipower system example is adopted to illustrate the effectiveness of the proposed protocols.

Bipartite containment control of multi-agent systems subject to adversarial inputs based on zero-sum game

In this paper, we investigate bipartite containment control problem of multi-agent systems (MASs) with signed directed graph under adversarial inputs. Firstly, we define the bipartite containment error and establish the equivalence between the bipartite containment error converging to zero and the achievement of bipartite containment control. Subsequently, we prove that the bounded L2-gain bipartite containment problem under adversarial inputs can be reformulated as a multi-player zero-sum differential graphical game problem and can be solved via the solution to the coupled Hamilton-Jacobi-Isaacs (HJI) equation. To address this, we propose a policy iteration (PI) algorithm and prove its convergence under different updating cases. The proposed algorithm is implemented by neural networks (NNs) and a numerical simulation example is provided to show its effectiveness.

Deep Reinforcement Learning-Based Differential Game Guidance Law against Maneuvering Evaders

To achieve the intelligent interception of different types of maneuvering evaders, based on deep reinforcement learning, a novel intelligent differential game guidance law is proposed in the continuous action domain. Different from traditional guidance laws, the proposed guidance law can avoid tedious manual settings and save cost efforts. First, the interception problem is transformed into the pursuit–evasion game problem, which is solved by zero-sum differential game theory. Next, the Nash equilibrium strategy is obtained through the Markov game process. To implement the proposed intelligent differential game guidance law, an actor–critic neural network based on deep deterministic policy gradient is constructed to calculate the saddle point of the differential game guidance problem. Then, a reward function is designed, which includes the tradeoffs among guidance accuracy, energy consumption, and interception time. Finally, compared with traditional methods, the interception accuracy of the proposed intelligent differential game guidance law is 99.2%, energy consumption is reduced by 47%, and simulation time is shortened by 1.58 s. All results reveal that the proposed intelligent differential game guidance law has better intelligent decision-making ability.

Time-optimal surveillance between two differential drive robots with a limited field of view

Keeping an intelligent agent under surveillance with an autonomous robot as it travels in the environment is one of the essential tasks in robotics. This work addresses a pursuit-evasion surveillance problem between two Differential Drive Robots moving on a plane without obstacles. One of them, the pursuer, is equipped with a limited field-of-view sensor and aims to maintain the other robot (evader) inside its detection region for as much time as possible. On the contrary, the evader wants to escape as soon as possible. Unlike prior studies that address the same problem, based on heuristic approaches, we frame it as a zero-sum differential game, and we use differential game theory to find a solution with theoretical guarantees. In particular, we compute the players' time-optimal motion strategies to achieve their goals near the end of the game and provide analytical expressions describing them. These strategies are in Nash equilibrium, meaning that any unilateral deviation by a player from these strategies does not provide such player benefit toward its goal. Our analysis exhibits the existence of a Transition Surface in which the evader switches its controls. We also determine the game's winner based on the players' initial configurations and velocities.

Critic learning control via zero-sum differential game for affine formation maneuver of multi-agent systems with cyber-attacks

The affine formation maneuver (AFM) control problem using the zero-sum differential game is studied for a second-order multi-agent system (MAS) against cyber-injection attacks on the actuators. First and foremost, the distributed control signals and the cyber-injected attack signals are viewed as two competing teams. Then, the leader-follower tracking error system is established where the followers aim to track the AFMs of the leaders. The leader-follower distributed optimal control policies and the cyber-attack attenuation policies are derived from the Nash equilibrium solution of the Hamilton-Jacobi-Isaac (HJI) equation. The solution of the HJI equation is estimated with the aid of the critic neural network. The weights of the critic system are learned online based on the adaptive dynamic programming (ADP) scheme. The uniformly ultimately bounded stability of the closed-loop system under the formulated zero-sum differential game control strategy has been proven by the Lyapunov stability theorem. A simulation example illustrates that the presented control method can attenuate cyber-injection attacks on the actuators and maintain the AFMs of the agents.

Safety-Aware Pursuit-Evasion Games in Unknown Environments Using Gaussian Processes and Finite-Time Convergent Reinforcement Learning.

This article develops a safe pursuit-evasion game for enabling finite-time capture, optimal performance as well as adaptation to an unknown cluttered environment. The pursuit-evasion game is formulated as a zero-sum differential game wherein the pursuer seeks to minimize its relative distance to the target while the evader attempts to maximize it. A critic-only reinforcement learning (RL)-based algorithm is then proposed for learning online and in finite time the pursuit-evasion policies and thus enabling finite-time capture of the evader. Safety is ensured by means of barrier functions associated with the obstacles, which are integrated into the running cost. Using Gaussian processes (GPs), a learning-based mechanism is devised for safely learning the unknown environment. Simulation results illustrate the efficacy of the proposed approach.

Fault-tolerant relative position control for spacecraft via zero-sum differential game

Fault-tolerant relative position control for spacecraft via zero-sum differential game

Min–max adaptive dynamic programming for zero-sum differential games

In this paper, a min–max adaptive dynamic programming approach is studied for a zero-sum differential game problem with unknown nonlinear dynamics. The unknown nonlinear dynamics is learned through the online Recurrent Neural Network (RNN). The error between the learned RNN model and the dynamics of the system is proved to converge to zero. The policy iteration algorithm is utilised to solve the Hamilton–Jacobi–Isaacs (HJI) equation associated with the differential game. Furthermore, the value function in the HJI equation is approximated through a critical Neural Network with its weights and activation functions updated online iteratively. The Uniform Ultimate Bounded stability of the closed-loop system is proved based on the Lyapunov theory. Finally, the effectiveness of the proposed solution method is demonstrated by applying it to three examples.

Linear Quadratic Zero-Sum Differential Games With Intermittent and Costly Sensing

Linear Quadratic Zero-Sum Differential Games With Intermittent and Costly Sensing

Robust control design for zero-sum differential games problem based on off-policy reinforcement learning technique

Robust control design for zero-sum differential games problem based on off-policy reinforcement learning technique

Robust reinsurance contract and investment with delay under mean-variance framework

We consider a (non) zero-sum differential game between two insurers and determine a robust reinsurance contract from joint interests of the insurers and the reinsurer under the framework of Stackelberg differential game, that is, a hybrid stochastic differential game framework is introduced in this article. We assume that each party can invest in a financial market consisting of a risk-free asset and a risky asset whose price evolution follows a stochastic volatility model, and the reinsurer charges the premium according to both the expected value premium principle and the variance premium principle. Since the performance-related capital inflow or outflow feature is introduced, the insurers and reinsurer all aim to maximize their own time-inconsistent mean-variance criterion with delay. Using the dynamic programming approach, we derive the equilibrium investment strategy, the robust equilibrium reinsurance contract, and the corresponding equilibrium value functions under two premium principles with certain conditions, respectively. Some interesting properties of the equilibrium investment strategy are proved explicitly, and several numerical examples and sensitivity analysis are presented to demonstrate the effects of model parameters on the equilibrium reinsurance contract.

Fault-tolerant tracking control for nonlinear systems with multiplicative actuator faults in view of zero-sum differential games

In view of zero-sum differential games, this paper addresses the fault-tolerant tracking control (FTTC) problem for nonlinear systems with multiplicative actuator faults. By augmenting the nominal system, the trajectory tracking problem is transformed into an optimal regulation problem. A fault observer is designed to estimate the multiplicative actuator fault factor. In order to better reflect system performance, a comprehensive performance index function including the trajectory tracking error, the control input, and the multiplicative actuator fault is constructed. By considering the control input and the multiplicative actuator fault as two players in the zero-sum differential game (ZSDG), the FTTC problem is thus transformed into a ZSDG problem. Then, a fault-tolerant control based on ZSDG is designed by solving the Hamilton–Jacobi-Isaacs (HJI) equation. Since the HJI equation is difficult to solve, a critic neural network is constructed to obtain its approximation, which constitutes the Nash equilibrium of the ZSDG. The closed-loop faulty system is guaranteed to be uniformly ultimately bounded via the Lyapunov’s direct method. The effectiveness of the developed FTTC method is demonstrated by two simulation examples.

Partial Differential Equations in Zero-Sum Differential Game and Applications on Coronavirus

In this paper, we studied a zero-sum game described by the partial differential equations as an application on Coronavirus. The game contains two players, player 1 is Coronavirus and player 2 is the population. We used ∞ -Laplacian which is denoted by ∆ ∞ . We added the time variable to the partial differential equation to see the behaviour of the spreading of Coronavirus. We used analytical methods, the Homotopy Perturbation Method and New Iterative Method, for solving the partial differential equation. A comparison between the two methods to the residual error is made. We showed in the graph the decreasing of spreading for Coronavirus with increasing the area with the time.

Radial Basis Functions for the Zero Sum Differential Game with Fractional Derivatives

Radial Basis Functions for the Zero Sum Differential Game with Fractional Derivatives

Dynamic Event-Triggered-Based Integral Reinforcement Learning Algorithm for Frequency Control of Microgrid With Stochastic Uncertainty

The intermittency of renewable energy and the uncertainty of load put forward higher robustness requirements for the frequency recovery of microgrid (MG). The energy storage equipment provides an idea for frequency control because of its fast response and strong controllability. In this paper, an on-line adaptive frequency control method is proposed to control the governor and energy storage to realize the frequency recovery of MG under stochastic uncertainty. First, the MG system with external disturbances is constructed as a zero-sum differential game model to obtain a robust optimal control scheme. Then, considering the system parameter uncertainty, an improved integral reinforcement learning (IRL) algorithm is designed, in which the reinforcement signal contains a non-quadratic function to solve the energy storage control constraints. Furthermore, a novel dynamic event-triggered control (DETC) is developed to reduce control update times of energy storage. The dynamic variable in DETC coupled with static trigger includes not only the past triggering information but also the disturbances. DETC has a larger trigger threshold than static event-triggered control (SETC). Meanwhile, the proposed algorithm is implemented by an action-critic network structure, in which the action network of energy storage is updated aperiodically. Finally, simulation results show that the proposed control algorithm can realize the frequency recovery of MG, and it has good robustness compared with other algorithms.

Zero-sum differential game guidance law for missile interception engagement via neuro-dynamic programming

In this paper, for solving the nonlinear control problem of the missile intercepting the maneuvering target, a novel nonlinear zero-sum differential game guidance law is proposed via the neuro-dynamic programming approach. First, the continuous-time nonlinear differential game problem is transformed into solving the nonlinear Hamilton–Jacobi–Isaacs (HJI) equation. Then, a critic neural network is designed to solve the corresponding nonlinear HJI equation. An adaptive weight tuning law for the critic weights is proposed, where an additional term is added to ensure the stability of the closed-loop nonlinear system. Furthermore, the uniform ultimate boundedness of the closed-loop system and the critic NN weights estimation error are proved with the Lyapunov approach. Finally, some simulation results are presented to demonstrate the effectiveness of the proposed differential game guidance law for nonlinear interception.

Approximating the Value of Zero-Sum Differential Games with Linear Payoffs and Dynamics

We consider two-player zero-sum differential games of fixed duration, where the running payoff and the dynamics are both linear in the controls of the players. Such games have a value, which is determined by the unique viscosity solution of a Hamilton–Jacobi-type partial differential equation. Approximation schemes for computing the viscosity solution of Hamilton–Jacobi-type partial differential equations have been proposed that are valid in a more general setting, and such schemes can of course be applied to the problem at hand. However, such approximation schemes have a heavy computational burden. We introduce a discretized and probabilistic version of the differential game, which is straightforward to solve by backward induction, and prove that the solution of the discrete game converges to the viscosity solution of the partial differential equation, as the discretization becomes finer. The method removes part of the computational burden of existing approximation schemes.