Saddle-point Equilibrium Research Articles

Linear Exponential Quadratic Control for Mean Field Stochastic Systems

In this technical note, we consider linear exponential quadratic (LEQ) control for mean field stochastic differential equations (MFSDEs). The MFSDE includes the expectation value of state and control, and the objective functional is exponential of a quadratic functional in state, control, and their expectations. We obtain the explicit optimal solution as well as the optimal cost. The corresponding optimal solution is linear in state and its expectation, which is characterized by the Riccati differential equations (RDEs). The results are obtained by showing that after applying the completion of squares method, the remaining exponentiated stochastic integral and additional RDE terms can be eliminated together by taking expectation since they constitute the associated Radon-Nikodym derivative. As an extension of the problem, the LEQ zero-sum differential game is considered, for which we obtain the explicit optimal solution (saddle-point equilibrium) as well as the optimal cost.

Robust Policies for a Multiple-Pursuer Single-Evader Differential Game

Analysis of the pursuit–evasion differential game consisting of multiple pursuers and single evader with simple motion is difficult due to the well-known curse of dimensionality. Policies have been proposed for this scenario, and we show that these policies are global Stackelberg equilibrium strategies. However, we also show that they are not saddle-point equilibria in the feedback sense. The argument is twofold: cases where the saddle-point condition is violated and cases where the strategy profiles are not time consistent (subgame perfect). The issue of capturability is explored, and sufficient conditions for guaranteed capture are provided. A new pursuit policy is proposed which guarantees capture while also providing an upper bound for capture time. The evader policy corresponding to the global Stackelberg equilibrium is shown to provide a lower bound for capture time. Thus, these policies are robust from the pursuer and evader perspectives, respectively, should they implement them. Several other interesting pursuit and evasion policies are explored and compared with the robust policies in a series of experiments.

A second-order cone programming formulation for two player zero-sum games with chance constraints

We consider a two player finite strategic zero-sum game where each player has stochastic linear constraints. We formulate the stochastic constraints of each player as chance constraints. We show the existence of a saddle point equilibrium in mixed strategies if the row vectors of the random matrices defining the stochastic constraints are elliptically symmetric distributed random vectors. We further show that a saddle point equilibrium can be obtained from the optimal solutions of a primal-dual pair of second-order cone programs.

The Multi-pursuer Single-Evader Game

We consider a general pursuit-evasion differential game with three or more pursuers and a single evader, all with simple motion (fixed-speed, infinite turn rate). It is shown that traditional means of differential game analysis is difficult for this scenario. But simple motion and min-max time to capture plus the two-person extension to Pontryagin’s maximum principle imply straight-line motion at maximum speed which forms the basis of the solution using a geometric approach. Safe evader paths and policies are defined which guarantee the evader can reach its destination without getting captured by any of the pursuers, provided its destination satisfies some constraints. A linear program is used to characterize the solution and subsequently the saddle-point is computed numerically. We replace the numerical procedure with a more analytical geometric approach based on Voronoi diagrams after observing a pattern in the numerical results. The solutions derived are open-loop optimal, meaning the strategies are a saddle-point equilibrium in the open-loop sense.

Approximate-optimal control algorithm for constrained zero-sum differential games through event-triggering mechanism

This paper investigates the optimization problem of two-player zero-sum differential game with control constraints in the framework of event triggering. Relying on reinforcement learning, an adaptive dynamic programming algorithm is developed to approximate the optimal solution of zero-sum game, i.e., the saddle-point equilibrium. A single-network structure is adopted, wherein a critic neural network (NN) evaluates the action. First, the constrained Hamilton–Jacobi–Isaacs equation is mathematically derived in the presence of control constraints; the event-triggering mechanism is then incorporated to reduce calculations and actions. Then, based on the gradient-descent technique, a novel weight updating law is designed for the critic NN, which ensures the solution can converge to the optimal value online. Moreover, the stability of closed-loop system is guaranteed and the unfavorable Zeno behavior is excluded by calculating the theoretical minimum triggering interval. Finally, two numerical examples are provided to verify the reliability and effectiveness of proposed algorithm.

Single‐network ADP for near optimal control of continuous‐time zero‐sum games without using initial stabilising control laws

This study establishes an approximate optimal critic learning algorithm based on single-network adaptive dynamic programming aiming at solutions to continuous-time two-player zero-sum games in the absence of initial stabilising control policies. Single-network means one critic neural network, which is utilised to derive the saddle-point equilibrium of a zero-sum differential game by approximately learning the value function. First, the authors elaborate mathematically two-player zero-sum game problems and analyse the similarity of the zero-sum game problem between linear and non-linear systems. Then, this adaptive learning scheme is implemented as a critic structure that derives control and disturbance policies by learning the optimal value, and a novel weight tuning law involving a stable operator is proposed to ensure convergence and stability. Moreover, the uniform ultimate bounded stability of the whole system is rigorously proved by Lyapunov theory. Finally, reasonable simulation results are provided to confirm the effectiveness of the improved approximate optimal control technique in solving equations for a complex linear system and a non-linear system.

A Min–Max Approach to Event- and Self-Triggered Sampling and Regulation of Linear Systems

This paper presents both an event- and a self-triggered sampling and regulation scheme for continuous time linear dynamic systems by using zero-sum game formulation. A novel performance index is defined wherein the control policy is treated as the first player and the threshold for control input error due to aperiodic dynamic feedback is treated as the second player. The optimal control policy and sampling intervals are generated using the saddle point or Nash equilibrium solution, which is obtained from the corresponding game algebraic Riccati equation. To determine the optimal event-based sampling scheme, an event-triggering condition is derived by utilizing the worst case control input error as the threshold. To avoid the additional hardware for the event-triggering mechanism, a near optimal self-triggering condition is derived to determine the future sampling instants given the current state vector. To guarantee Zeno-free behavior in both the event- and self-triggered closed-loop systems, the minimum intersample times are shown to be lower bounded by a nonzero positive number. Asymptotic stability of the closed-loop system is ensured using Lyapunov stability analysis. Finally, simulation examples are provided to substantiate the analytical claims.

A distributed continuous-time method for non-convex QCQPs

This paper studies non-convex Quadratically Constrained Quadratic Programmings (QCQPs) via the continuous-time optimization dynamics. We first develop an easily checkable necessary and sufficient condition that characterizes whether a KKT point will also be a saddle-point (the pair of primal and dual optima) for a non-convex QCQP. Then we analyze the semistability of the saddle-point equilibrium set with respect to the proposed optimization dynamics. We also point out that, for certain networked QCQPs, the proposed approach exhibits an intrinsic distributed computational structure.

Saddle-point equilibrium sequence in one class of singular infinite horizon zero-sum linear-quadratic differential games with state delays

ABSTRACTWe consider an infinite horizon zero-sum linear-quadratic differential game with state delays in the dynamics. The cost functional of this game does not contain a control cost of the minimizing player (the minimizer), meaning that the considered game is singular. For this game, definitions of the saddle-point equilibrium and the game value are proposed. These saddle-point equilibrium and game value are obtained by a regularization of the singular game. Namely, we associate this game with a new differential game for the same equation of dynamics. The cost functional in the new game is the sum of the original cost functional and an infinite horizon integral of the square of the minimizer's control with a small positive weight coefficient. This new game is regular, and it is a cheap control game. An asymptotic analysis of this cheap control game is carried out. Using this asymptotic analysis, the existence of the saddle-point equilibrium and the value of the original game is established, and their expressions are derived. Illustrative example is presented.

Vehicle control synthesis using phase portraits of planar dynamics

ABSTRACTPhase portraits provide control system designers strong graphical insight into nonlinear system dynamics. These plots readily display vehicle stability properties and map equilibrium point locations and movement to changing parameters and system inputs. This paper extends the usage of phase portraits in vehicle dynamics to control synthesis by illustrating the relationship between the boundaries of stable vehicle operation and the state derivative isoclines in the yaw rate–sideslip phase plane. Closed-loop phase portraits demonstrate the potential for augmenting a vehicle's open-loop dynamics through steering and braking. The paper concludes by applying phase portrait analysis to an envelope control algorithm for yaw stability and a sliding surface controller for stabilising a saddle point equilibrium in drifting.

Mutually Quadratically Invariant Information Structures in Two-Team Stochastic Dynamic Games

We formulate a two-team linear quadratic stochastic dynamic game featuring two opposing teams each with decentralized information structures. We introduce the concept of mutual quadratic invariance (MQI), which, analogously to quadratic invariance in (single team) decentralized control, defines a class of interacting information structures for the two teams under which optimal feedback control strategies are linear and easy to compute. We show that for zero-sum two-team dynamic games with MQI information structure, structured state feedback saddle-point equilibrium strategies can be computed from equivalent structured disturbance feedforward saddle point equilibrium strategies. We also show that there is a saddle-point equilibrium in linear strategies even when the teams are allowed to use nonlinear strategies. However, for nonzero-sum games we show via a counterexample that a similar equivalence fails to hold. The results are illustrated with a simple yet rich numerical example that illustrates the importance of the information structure for dynamic games.

A hybrid stochastic game for secure control of cyber-physical systems

In this paper, we establish a zero-sum, hybrid state stochastic game model for designing defense policies for cyber-physical systems against different types of attacks. With the increasingly integrated properties of cyber-physical systems (CPS) today, security is a challenge for critical infrastructures. Though resilient control and detecting techniques for a specific model of attack have been proposed, to analyze and design detection and defense mechanisms against multiple types of attacks for CPSs requires new system frameworks. Besides security, other requirements such as optimal control cost also need to be considered. The hybrid game model we propose contains physical states that are described by the system dynamics, and a cyber state that represents the detection mode of the system composed by a set of subsystems. A strategy means selecting a subsystem by combining one controller, one estimator and one detector among a finite set of candidate components at each state. Based on the game model, we propose a suboptimal value iteration algorithm for a finite horizon game, and prove that the algorithm results an upper bound for the value of the finite horizon game. A moving-horizon approach is also developed in order to provide a scalable and real-time computation of the switching strategies. Both algorithms aim at obtaining a saddle-point equilibrium policy for balancing the system’s security overhead and control cost. The paper illustrates these concepts using numerical examples, and we compare the results with previously system designs that only equipped with one type of controller.

A Dynamic Analysis of Special Interest Politics and Electoral Competition

This paper characterizes the solution to differential games in the context of electoral competition between two political parties/politicians, in the presence of voters and a special interest group. The basic structure of the analytical model is similar to Lambertini ( https://amsacta.unibo.it/4884/1/415.pdf , 2001, Dynamic games in economics. Springer, Berlin, pp 187–204, 2014), which is extended to model the involvement of a special interest group. Furthermore, voters not only vote but also care for the level of public good provision, while the interest group cares for the regulatory benefit in exchange for financial contribution for campaign expenditure. With a quadratic cost structure, we find that a closed-loop solution collapses to an open-loop equilibrium. Moreover, at the private optimum, the expenditure offered for public good provision, regulatory benefit rendered, voting support from voters, and financial contributions from special interest group received by any political party are always higher than at the social optimum. That is, political parties have the tendency to make excessive offers of expenditure on public good to grab a larger vote share to win the election. Consequently, voters vote retrospectively to the party that offers to overspend more. A higher private optimal regulatory benefit helps the political parties to receive higher financial contributions, which could be potentially used for election campaigns and indirectly contributes to enhance their vote share. The solutions to the control and state variables constitute steady-state saddle point equilibria at both private and social optima.

Uncertain saddle point equilibrium differential games with non-anticipating strategies

In this paper, we investigate uncertain saddle point equilibrium differential games under uncertain environment. We propose an optimistic value game model and define the value function of the game by introducing the concept of non-anticipating strategy. We prove the continuity and dynamic programming property of the value function. Then we derive the uncertain Hamilton–Jacobi–Isaacs equation by the viscosity solution approach.

Saddle point equilibrium under uncertain environment

The saddle point equilibrium problem is of great importance in differential game theory. In this paper, an optimistic value model for the saddle point equilibrium problem under uncertain environment is investigated. The equilibrium equation for the proposed model is presented. Then a linear quadratic model is discussed. Finally, a counter terror problem is analyzed by the results obtained in the paper.

Controller–jammer game models of Denial of Service in control systems operating over packet-dropping links

The paper introduces a class of zero-sum games between the adversary and controller as a scenario for a ‘denial of service’ in a networked control system. The communication link is modelled as a set of transmission regimes controlled by a strategic jammer whose intention is to wage an attack on the plant by choosing a most damaging regime-switching strategy. We demonstrate that even in the one-step case, the introduced games admit a saddle-point equilibrium, at which the jammer’s optimal policy is to randomize in a region of the plant’s state space, thus requiring the controller to undertake a nontrivial response which is different from what one would expect in a standard stochastic control problem over a packet dropping link. The paper derives conditions for the introduced games to have such a saddle-point equilibrium. Furthermore, we show that in more general multi-stage games, these conditions provide ‘greedy’ jamming strategies for the adversary.

A Centrality-Based Security Game for Multihop Networks

We formulate a network security problem as a zero-sum game between an attacker who tries to disrupt a network by disabling one or more nodes, and the nodes of the network who must allocate limited resources to defend the network. The utility of the zero-sum game can be one of several network performance metrics that correspond to node centrality measures. We first present a fast centralized algorithm that uses a monotone property of the utility function to compute saddle-point equilibrium strategies for the case of single-node attacks and single- or multiple-node defense. We then extend the approach to the distributed setting by computing the necessary quantities using a finite-time distributed averaging algorithm. For simultaneous attacks to multiple nodes, the computational complexity grows quickly, so we propose a method to approximate the saddle-point equilibrium strategies based on sequential simplification, which performs well in simulations.

Dynamic Games With Asymmetric Information and Resource Constrained Players With Applications to Security of Cyberphysical Systems

We model attacks on a cyberphysical system as a game between two players—the attacker and the system. The players may not acquire the complete information about each other, and that leads to an asymmetric information game. Furthermore, the players may have a certain fixed amount of resources, which constrains their strategies across time. Accordingly, we consider a dynamic multiplayer nonzero sum game with asymmetric information in which controllers have total resource constraints. Under certain assumptions on the information structure of the game, we devise an algorithm that computes a subclass of Nash equilibria of the game. We also study a denial-of-service attack on a cyberphysical system, model it as two-player zero-sum games, and apply our algorithm to compute the saddle-point equilibrium strategies of the attacker and the controller.

Adaptive Dynamic Programming for Discrete-Time Zero-Sum Games.

In this paper, a novel adaptive dynamic programming (ADP) algorithm, called "iterative zero-sum ADP algorithm," is developed to solve infinite-horizon discrete-time two-player zero-sum games of nonlinear systems. The present iterative zero-sum ADP algorithm permits arbitrary positive semidefinite functions to initialize the upper and lower iterations. A novel convergence analysis is developed to guarantee the upper and lower iterative value functions to converge to the upper and lower optimums, respectively. When the saddle-point equilibrium exists, it is emphasized that both the upper and lower iterative value functions are proved to converge to the optimal solution of the zero-sum game, where the existence criteria of the saddle-point equilibrium are not required. If the saddle-point equilibrium does not exist, the upper and lower optimal performance index functions are obtained, respectively, where the upper and lower performance index functions are proved to be not equivalent. Finally, simulation results and comparisons are shown to illustrate the performance of the present method.

Singular infinite horizon zero-sum linear-quadratic differential game: Saddle-point equilibrium sequence

We consider an infinite horizon zero-sum linear-quadratic differential game in the case where the cost functional does not contain a control cost of the minimizing player (the minimizer). This feature means that the game under consideration is singular. For this game, novel definitions of the saddle-point equilibrium and game value are proposed. To obtain these saddle-point equilibrium and game value, we associate the singular game with a new differential game for the same equation of dynamics. The cost functional in the new game is the sum of the original cost functional and an infinite horizon integral of the square of the minimizer's control with a small positive weight coefficient. This new game is regular, and it is a cheap control game. Using the solvability conditions, the solution of the cheap control game is reduced to solution of a Riccati matrix algebraic equation with an indefinite quadratic term. This equation is perturbed by a small parameter. Subject to a proper assumption, an asymptotic expansion of a stabilizing solution to this equation is constructed and justified. Using this asymptotic expansion, the existence of the saddle-point equilibrium and the value of the original game is established, and their expressions are derived. Illustrative example is presented.