Nonzero-sum Differential Games Research Articles

Barrier-Critic Adaptive Robust Control of Nonzero-Sum Differential Games for Uncertain Nonlinear Systems With State Constraints

Barrier-Critic Adaptive Robust Control of Nonzero-Sum Differential Games for Uncertain Nonlinear Systems With State Constraints

Distributed Adaptive Inverse Differential Game Approach to Leader's Behavior Learning for Multiple Autonomous Followers

In this study, the leader's behavior learning problem is investigated for a class of leader-follower systems, where the leader's behavior is assumed not a priori known to all followers that are autonomous. A multi-player nonzero-sum differential game is introduced to indicate the control interaction between agents, where the leader's behavior is modelled by an unknown cost function. The autonomous followers aim to retrieve the weighting matrix in the leader's cost function collaboratively. A distributed online adaptive inverse differential game (IDG) approach to the leader's behavior learning is proposed for the autonomous followers. Specifically, a concurrent learning (CL) based adaptive law and an interactive game controller are first developed for each autonomous follower to learn the leader's feedback gain matrix online, while the feedback Nash equilibrium of the game can be achieved. Then, a linear matrix inequality (LMI) optimization problem is formulated to determine the weighting matrix of the leader's cost function for each autonomous follower. The proposed method simply requires that all followers share their interactive feedback gain matrices, not their private intents, and use only the system state data, not both the system state data and the leader's control input data. The main advantages of the proposed method are that it can be implemented online without requiring the persistent excitation condition and needs less computational power for all followers. Finally, numerical simulations are provided to demonstrate the effectiveness and feasibility of the developed method.

Nonzero-Sum Game-Based Voltage Recovery Consensus Optimal Control for Nonlinear Microgrids System.

Since most of the existing models based on the microgrids (MGs) are nonlinear, which could cause the controller oscillate, resulting in the excessive line loss, and the nonlinear could also lead to the controller design difficulty of MGs system. Therefore, this article researches the distributed voltage recovery consensus optimal control problem for the nonlinear MGs system with N -distributed generations (DGs), in the case of providing stringent real power sharing. First, based on the distributed cooperative control concept of multiagent systems and the critic neural networks (NNs), a novel distributed secondary voltage recovery consensus optimal control protocol is constructed via applying the backstepping technique and nonzero-sum (NZS) differential game strategy to realize the voltage recovery of island MGs. Meanwhile, the model identifier is established to reconstruct the unknown NZS games systems based on a three-layer NN. Then, a critic NN weight adaptive adjustment tuning law is proposed to ensure the convergence of the cost functions and the stability of the closed-loop system. Furthermore, according to Lyapunov stability theory, it is proven that all signals are uniform ultimate boundedness in the closed loop system and the voltage recovery synchronization error converges to an arbitrarily small neighborhood of the origin near. Finally, some simulation results in MATLAB illustrate the validity of the proposed control strategy.

Online solving Nash equilibrium solution of N-player nonzero-sum differential games via recursive least squares

Online solving Nash equilibrium solution of N-player nonzero-sum differential games via recursive least squares

Feedback Nash Equilibria in Differential Games With Impulse Control

We study a class of deterministic finite-horizon two-player nonzero-sum differential games where players are endowed with different kinds of controls. We assume that Player 1 uses piecewise-continuous controls, while Player 2 uses impulse controls. For this class of games, we seek to derive conditions for the existence of feedback Nash equilibrium strategies for the players. More specifically, we provide a verification theorem for identifying such equilibrium strategies, using the Hamilton-Jacobi-Bellman (HJB) equations for Player 1 and the quasi-variational inequalities (QVIs) for Player 2. Further, we show that the equilibrium number of interventions by Player 2 is upper bounded. Furthermore, we specialize the obtained results to a scalar two-player linear-quadratic differential game. In this game, Player 1's objective is to drive the state variable towards a specific target value, and Player 2 has a similar objective with a different target value. We provide, for the first time, an analytical characterization of the feedback Nash equilibrium in a linear-quadratic differential game with impulse control. We illustrate our results using numerical experiments.

Learning-Based Control With Decentralized Dynamic Event-Triggering for Vehicle Systems

The optimal control of multi-input system can be described by a multiplayer nonzero-sum differential game. This article theoretically presents an event-based adaptive learning scheme to approximate the Nash equilibrium, and practically addresses the cruise control problem for Caltech vehicle systems. This design is deployed in two aspects. On one hand, the reinforcement learning is implemented through critic neural network architecture and recalling stored experience data. On the other hand, in view of that each player’s preference is different, the decentralized triggering manner is considered to reduce communication. Based on the continuous state, the local sampled state is defined for each player, and a static triggering mechanism is formulated first. The decentralized dynamic triggering is then promoted by designing an auxiliary variable whose dynamics are constructed using static triggering information. Next, the proposed learning scheme is examined on a four-player numerical system. Finally, the learning-based controller is tested on a single-vehicle system under different tracking commands, and then, it is extended to multivehicle systems to realize cooperative optimization by introducing a novel game-in-game structure.

Excitation for Adaptive Optimal Control of Nonlinear Systems in Differential Games

This article focuses on the fulfillment of the persistent excitation (PE) condition for signals which result from transformations by means of polynomials. This is essential, e.g., for the convergence of adaptive dynamic programming algorithms due to commonly used polynomial function approximators. As theoretical statements are scarce regarding the nonlinear transformation of PE signals, we propose conditions on the system state such that its transformation by polynomials is PE. To validate our theoretical statements, we develop an exemplary excitation procedure based on our conditions using a feed-forward control approach and demonstrate the effectiveness of our method in a nonzero-sum differential game. In this setting, our approach outperforms commonly used probing noise in terms of convergence time and the degree of PE, shown by a numerical example.

Indirect Shared Control Through Non-Zero Sum Differential Game for Cooperative Automated Driving

Cooperative driving of human driver and automated system can effectively reduce the necessity of extremely accurate environment perception of highly automated vehicles, and enhance the robustness of decision-making and motion control. However, due to the two players’ different intentions, severe conflicts may exist during the cooperation, which often result in negative consequences on driving safety and maneuverability. This paper presents an indirect shared control method to model the situation and improve the driving performance, which focus on the affine input nonlinear vehicle dynamic system for shared controller design under the framework of non-zero sum differential game. The Nash equilibria strategy indicates the best response for the automated system, which can guide the automated controller to act more safely and comfortably. Aimed to obtain fast solution for practical application, approximate dynamic programming is utilized to find the Nash equilibria, which is represented by deep neural networks and solved iteratively. Driver-in-the-loop tests on a driving simulator were conducted to verify the performance of the proposed method under highway driving scenarios. The results show that the designed controller is able to reduce the driving workload and ensure the driving safety.

Nonzero-sum differential games of continuous-Time nonlinear systems with uniformly ultimately [formula omitted]-bounded by adaptive dynamic programming

In this paper, a single network adaptive dynamic programming (ADP) control method is presented to obtain the nearly optimal control policies for the non-zero sum (NZS) differential game problem of the autonomous nonlinear system. The Osgood condition, instead of the traditional Lipschitz condition, is firstly introduced to policy iteration to guarantee the existence and uniqueness of the solution of the dynamic nonlinear systems and to weaken the limited conditions of nonlinear dynamic functions f(x), g(x) and k(x). Moreover, this adaptive control pattern finds in real-time approximations of the optimal value and the non-zero sum Nash-equilibrium, while also ensuring the uniform ultimate ε-boundedness of the closed-loop system. Further, as the number of hidden-layer neurons tends to infinite, the approximation errors converge to zero. As a result, the closed-loop system is asymptotically stable. Finally, the effectiveness of the proposed near-optimal control pattern is verified by a simulation example.

A numerical iterative method for solving two-point BVPs in infinite-horizon nonzero-sum differential games: Economic applications

In this work, the Nash equilibrium solution of nonlinear infinite-horizon nonzero-sum differential games with open-loop information is investigated numerically. For this class of games, some difficulties are involved in finding an open-loop Nash equilibrium. For instance, we must solve a nonlinear differential equations system with split boundary conditions, namely the two-point boundary value problem (TPBVP), in which some of the boundary conditions are specified at the initial time and some at the infinite final time. In the current study, we provide a combined numerical algorithm based on a new set of basis functions on the half-line, called the exponential Chelyshkov functions (ECFs), and quasilinearization (QL) method to solve TPBVPs. In the first step, we reduce the nonlinear TPBVP to a sequence of linear differential equations by using the QL method. Although we have now a linearized system, another difficulty is finding the approximate solution of this linear system such that the transversality conditions at the infinite final time are satisfied. So, in the second step, we apply a collocation method based on the ECFs to solve the obtained linear system in the semi-infinite domain. The convergence of the proposed method is discussed in detail. To confirm the validity and efficiency of the proposed scheme, we compute the approximate solution of TPBVP as well as the open-loop Nash equilibrium for four applications of differential games in economics and management science.

Asynchronous learning for actor–critic neural networks and synchronous triggering for multiplayer system

In this paper, based on actor–critic neural network structure and reinforcement learning scheme, a novel asynchronous learning algorithm with event communication is developed, so as to solve Nash equilibrium of multiplayer nonzero-sum differential game in an adaptive fashion. From the point of optimal control view, each player or local controller wants to minimize the individual infinite-time cost function by finding an optimal policy. In this novel learning framework, each player consists of one critic and one actor, and implements distributed asynchronous policy iteration to optimize decision-making process. In addition, communication burden between the system and players is effectively reduced by setting up a central event generator. Critic network executes fast updates by gradient-descent adaption while actor network gives event-induced updates using the gradient projection. The closed-loop asymptotic stability is ensured along with uniform ultimate convergence. Then, the effectiveness of the proposed algorithm is substantiated on a four-player nonlinear system, revealing that it can significantly reduce sampling numbers without impairing learning accuracy. Finally, by leveraging nonzero-sum game idea, the proposed learning scheme is also applied to solve the lateral-directional stability of a linear aircraft system, and is further extended to a nonlinear vehicle system for achieving adaptive cruise control.

Non-zero sum differential game for stochastic Markovian jump systems with partially unknown transition probabilities

This paper focuses on the non-zero sum differential game problem for Markovian jump systems with partially unknown transition probabilities. Firstly, a suboptimal control problem is studied by the free-connection weighting matrix method, and then the non-zero sum differential game problem is investigated on this basis. Several sufficient conditions for the existence of ε-suboptimal control strategy and ε-suboptimal Nash equilibrium strategies are provided, and their explicit expressions are designed. Moreover, the precise form for the upper bound of the cost function is also given. To facilitate the calculation, all conditions are converted into the corresponding equivalent linear matrix inequalities or bilinear matrix inequalities form. Finally, two numerical examples are utilized to demonstrate the effectiveness of the main results.

Linear-quadratic non-zero sum differential game for mean-field stochastic systems with asymmetric information

In this study, we consider a class of asymmetric information linear-quadratic non-zero sum stochastic differential game problems. The system dynamics are governed by a forward linear mean-field stochastic differential equation and the cost functional is quadratic. In order to facilitate some practical applications, the mean-field terms for the state process and control process are considered in both the system dynamics and cost functional. Using the classical calculus of variation and dual methods, the open-loop Nash equilibrium point can be expressed by introducing an auxiliary mean-field forward backward stochastic differential equation, which comprises one forward and two backward components. Due to some Riccati equations and ordinary differential equations that possess unique solutions, the Nash equilibrium point can also be represented in feedback form for several special cases under asymmetric information. The corresponding filtering equations are derived, and the existence and uniqueness of the solutions are proved. An investment problem in finance is discussed to demonstrate the good performance of the theoretical results.

Nash equilibria in nonzero-sum differential games with impulse control

In this paper, we introduce a class of deterministic finite-horizon two-player nonzero-sum differential games where one player uses ordinary controls while the other player uses impulse controls. We use the word ‘ordinary’ to mean that Player 1 uses control strategies that are piecewise continuous functions of time. We formulate the necessary and sufficient conditions for the existence of an open-loop Nash equilibrium for this class of differential games. We specialize these results to linear-quadratic games, and show that the open-loop Nash equilibrium strategies can be computed by solving a constrained non-linear optimization problem. In particular, for the impulse player, the equilibrium timing and level of impulses can be obtained. Furthermore, for the special case of linear-state differential games, we obtain analytical characterization of equilibrium number, timing, and the level of impulse in terms of the problem data. We illustrate our results using numerical experiments.

An Impulse-Regime Switching Game Model of Vertical Competition

We study a new kind of nonzero-sum stochastic differential game with mixed impulse/switching controls, motivated by strategic competition in commodity markets. A representative upstream firm produces a commodity that is used by a representative downstream firm to produce a final consumption good. Both firms can influence the price of the commodity. By shutting down or increasing generation capacities, the upstream firm influences the price with impulses. By switching (or not) to a substitute, the downstream firm influences the drift of the commodity price process. We study the resulting impulse-regime switching game between the two firms, focusing on explicit threshold-type equilibria. Remarkably, this class of games naturally gives rise to multiple potential Nash equilibria, which we obtain thanks to a verification-based approach. We exhibit three candidate types of equilibria depending on the ultimate number of switches by the downstream firm (zero, one or an infinite number of switches). We illustrate the diversification effect provided by vertical integration in the specific case of the crude oil market. Our analysis shows that the diversification gains strongly depend on the pass-through from the crude price to the gasoline price.

Finite‐horizon optimal tracking control for constrained‐input nonlinear interconnected system using aperiodic distributed nonzero‐sum games

This paper proposes a distributed adaptive dynamic programming scheme to investigate the optimal tracking control problem for finite-horizon non-linear interconnected systems with constraint inputs under aperiodic sampling. A N-player nonzero-sum differential game system is constructed with the presented non-linear interconnected system and the tracking error system by introducing the augment vectors. To address the problems of constrained-input and finite-horizon control, a non-quadratic utility function and a finite-horizon cost function are utilized which will arise in the time-varying Hamilton–Jacobi (HJ) equation. Then, a periodic event-triggered scheme is designed to realize aperiodic sampling, where the consumption of communication resources is reduced and the Zeno behavior is avoided. Under the designed periodic event-triggered scheme, the time-varying HJ equation is almost impossible to get an analytical solution due to its hybrid properties and non-linearity. Therefore, the critic neural networks are used to estimate the optimal solution of the HJ equation, and the weight update law is constructed to guarantee the uniformly ultimate bounded of approximated errors. Further, the hybrid nonzero-sum differential game is confirmed to be uniformly ultimate bounded by using the Lyapunov theory. Finally, the obtained distributed PET control strategy is successfully applied to dispose the missile-target intercepter problem.

Adaptive Learning and Sampled-Control for Nonlinear Game Systems Using Dynamic Event-Triggering Strategy.

Static event-triggering-based control problems have been investigated when implementing adaptive dynamic programming algorithms. The related triggering rules are only current state-dependent without considering previous values. This motivates our improvements. This article aims to provide an explicit formulation for dynamic event-triggering that guarantees asymptotic stability of the event-sampled nonzero-sum differential game system and desirable approximation of critic neural networks. This article first deduces the static triggering rule by processing the coupling terms of Hamilton-Jacobi equations, and then, Zeno-free behavior is realized by devising an exponential term. Subsequently, a novel dynamic-triggering rule is devised into the adaptive learning stage by defining a dynamic variable, which is mathematically characterized by a first-order filter. Moreover, mathematical proofs illustrate the system stability and the weight convergence. Theoretical analysis reveals the characteristics of dynamic rule and its relations with the static rules. Finally, a numerical example is presented to substantiate the established claims. The comparative simulation results confirm that both static and dynamic strategies can reduce the communication that arises in the control loops, while the latter undertakes less communication burden due to fewer triggered events.

Analytical and experimental nonzero-sum differential game-based control of a 7-DOF robotic manipulator

We formulate a Nash-based feedback control law for an Euler–Lagrange system to yield a solution to noncooperative differential game. The robot manipulators are broadly used in industrial units on the account of their reliable, fast, and precise motions, while consuming a significant lumped amount of energy. Therefore, an optimal control strategy needs to be implemented in addressing efficiency issues, while delivering accuracy obligation. As a case study, we here focus on a 7-DOF robot manipulator through formulating a two-player feedback nonzero-sum differential game. First, coupled Euler–Lagrangian dynamic equations of the manipulator are briefly presented. Then, we formulate the feedback Nash equilibrium solution to achieve perfect trajectory tracking. Finally, the performance of the Nash-based feedback controller is analytically and experimentally examined. Simulation and experimental results reveal that the control law yields almost perfect tracking and achieves closed-loop stability.

Mean-field type forward-backward doubly stochastic differential equations and related stochastic differential games

We study a kind of partial information non-zero sum differential games of mean-field backward doubly stochastic differential equations, in which the coefficient contains not only the state process but also its marginal distribution, and the cost functional is also of mean-field type. It is required that the control is adapted to a sub-filtration of the filtration generated by the underlying Brownian motions. We establish a necessary condition in the form of maximum principle and a verification theorem, which is a sufficient condition for Nash equilibrium point. We use the theoretical results to deal with a partial information linear-quadratic (LQ) game, and obtain the unique Nash equilibrium point for our LQ game problem by virtue of the unique solvability of mean-field forward-backward doubly stochastic differential equation.

Nonzero-Sum Stochastic Differential Reinsurance Games with Jump–Diffusion Processes

In this paper, a nonzero-sum stochastic differential reinsurance game is studied. A model including controls for the market share (advertising), investment, and reinsurance policies is considered. A jump–diffusion process is used to represent insurance claims. Necessary conditions that would lead to the Nash equilibrium can be found in the duopoly game we consider. Cases with and without controls on the reinsurance are discussed separately. Closed-form solutions for optimal strategies are derived by applying the Hamilton–Jacobi–Bellman equation. Numerical examples are given to validate the correctness of our results.