Two-player Zero-sum Research Articles

Information spillover in multiple zero-sum games

This paper considers an infinitely repeated three-player zero-sum game with two-sided incomplete information, in which an informed player plays two zero-sum games simultaneously at each stage against two uninformed players. This is a generalization of the model in Aumann et al. (Repeated games with incomplete information. MIT Press, New York, 1995) of two-player zero-sum one-sided incomplete information games. Under a correlated prior, the informed player faces the problem of how to optimally disclose information among two uninformed players in order to maximize his long-term average payoffs (i.e., undiscounted payoffs). Our objective is to understand the adverse effects of “information spillover” from one game to the other in the equilibrium payoff set of the informed player. We provide conditions under which the informed player can fully overcome such adverse effects and characterize equilibrium payoffs. In a second result, we show how the effects of information spillover on the equilibrium payoff set of the informed player might be severe. Finally, we compare our findings on the equilibrium-payoff set of the informed player with those of Bayesian Persuasion models with multiple receivers.

Optimal [formula omitted] tracking control of nonlinear systems with zero-equilibrium-free via novel adaptive critic designs

In this paper, a novel adaptive critic control method is designed to solve an optimal H∞ tracking control problem for continuous nonlinear systems with nonzero equilibrium based on adaptive dynamic programming (ADP). To guarantee the finiteness of a cost function, traditional methods generally assume that the controlled system has a zero equilibrium point, which is not true in practical systems. In order to overcome such obstacle and realize H∞ optimal tracking control, this paper proposes a novel cost function design with respect to disturbance, tracking error and the derivative of tracking error. Based on the designed cost function, the H∞ control problem is formulated as two-player zero-sum differential games, and then a policy iteration (PI) algorithm is proposed to solve the corresponding Hamilton–Jacobi–Isaacs (HJI) equation. In order to obtain the online solution to the HJI equation, a single-critic neural network structure based on PI algorithm is established to learn the optimal control policy and the worst-case disturbance law. It is worth mentioning that the proposed adaptive critic control method can simplify the controller design process when the equilibrium of the systems is not zero. Finally, simulations are conducted to evaluate the tracking performance of the proposed control methods.

A Game-Theoretic Model for One-on-One Air Combat

The primary goal of air combat maneuvering is to neutralize the enemy under any circumstances. A critical step in achieving this goal is to gain and maintain aerial advantage by choosing and executing appropriate flighter maneuvers. In this paper, one-on-one air combat is modeled as a two-player zero-sum game to obtain optimal strategies for the players to select the flighter maneuvers. A new approach for constructing the game payoff function that accounts for the loss/gain of aerial advantage during the execution of a maneuver is proposed. A case study to evaluate the proposed approach, by comparing it with an existing approach, is performed through computer simulations.

Event-Triggered Single-Network ADP for Zero-Sum Game of Unknown Nonlinear Systems with Constrained Input

In this paper, an event-triggered adaptive dynamic programming (ADP) method is proposed to deal with the H∞ problem with unknown dynamic and constrained input. Firstly, the H∞-constrained problem is regarded as the two-player zero-sum game with the nonquadratic value function. Secondly, we develop the event-triggered Hamilton–Jacobi–Isaacs(HJI) equation, and an event-triggered ADP method is proposed to solve the HJI equation, which is equivalent to solving the Nash saddle point of the zero-sum game. An event-based single-critic neural network (NN) is applied to obtain the optimal value function, which reduces the communication resource and computational cost of algorithm implementation. For the event-triggered control, a triggering condition with the level of disturbance attenuation is developed to limit the number of sampling states, and the condition avoids Zeno behavior by proving the existence of events with minimum triggering interval. It is proved theoretically that the closed-loop system is asymptotically stable, and the critic NN weight error is uniformly ultimately boundedness (UUB). The learning performance of the proposed algorithm is verified by two examples.

GPI-Based design for partially unknown nonlinear two-player zero-sum games

This article focuses on obtaining the solution of two-player zero-sum games (ZSGs) where the saddle point may not exist. From the standpoints of the upper performance index function (PIF) and lower PIF separately, we seek the solution of two-player differential ZSGs without assuming the strict existence of saddle point. To achieve the objective, we apply the generalized policy iteration (GPI) technique to design the controller. With neural network approximators, the system trajectory data are used to update the approximation of the PIFs. It is established that the upper (lower) PIF sequence is convergent along the iteration axis under the GPI scheme. When the two function sequences converge to equal values, the saddle point is attained, and vice versa. Numerical simulation results conformed to the suitability of the designed algorithm.

Existence of pure equilibria in symmetric two-player zero-sum games

Existence of pure equilibria in symmetric two-player zero-sum games

Covert Outage Minimization in the Presence of Multiple Wardens

The idea of covert communication is to conceal the presence of a transmission from an illegitimate receiver, known as a warden. This paper tackles the problem of finite blocklength covert communication in the presence of multiple colluding wardens. The system consists of Alice, who aims to covertly transmit to Bob with the help of a cooperative jammer (henceforth known as Jammer), and a Fusion Center (FC) in charge of combining the wardens' information and deciding on the presence of Alice's transmission accordingly. In our proposed approach, we utilize a two-player zero-sum game to model the interaction between Alice and Jammer jointly as one player and FC as the second player. In this game, Alice and Jammer cooperatively randomize over a range of transmitting and jamming powers to confuse FC. In contrast, FC randomly changes the detection threshold to confuse Alice. The main focus of the paper is to study the impact of employing multiple wardens on the trade-off between the probability of error at FC and the outage probability at Bob. We derive a pay-off function that can be efficiently computed using linear programming to find the optimal distributions of transmitting and jamming powers as well as thresholds used by FC. The benefit of using a cooperative jammer in neutralizing the advantage of employing multiple wardens is shown by analytical results and numerical simulations.

A game theoretical approximation for a parabolic/elliptic system with different operators

<p style='text-indent:20px;'>In this paper we find viscosity solutions to a coupled system composed by two equations, the first one is parabolic and driven by the infinity Laplacian while the second one is elliptic and involves the usual Laplacian. We prove that there is a two-player zero-sum game played in two different boards with different rules in each board (in the first one we play a Tug-of-War game taking the number of plays into consideration and in the second board we move at random) whose value functions converge uniformly to a viscosity solution to the PDE system.</p>

Extensive game decision based on the PPO-CFR algorithm under incomplete information

Human-computer gaming under incomplete information is usually described by a two-player zero-sum game model. Counterfactual regret minimization (CFR) is a popular algorithm for two-player zero-sum games with incomplete information. However, the existing CFR and its variant algorithms use fixed regret calculation and strategy update type in the iteration process, which have their advantages and disadvantages in the incomplete information extensive game, and their generalization performance is weak. To solve this problem, this paper combines the proximal policy optimization (PPO) algorithm in reinforcement learning with the CFR algorithm to train rational agents to adaptively select appropriate regret calculation and strategy update types in the CFR iteration process to improve the generalization performance of the current CFR algorithms and realize the policy optimization of the incomplete information extensive game. In this paper, general poker game experiments are used to verify the proposed algorithm, and a stepwise reward function is formulated to train the action policy of the agent. Experimental results show that compared with existing state-of-the-art methods, the PPO-CFR algorithm has better generalization performance and lower exploitability, and the iteration policy is closer to the Nash equilibrium policy.

Secure State Estimation of Cyber-Physical System under Cyber Attacks: Q-Learning vs. SARSA

This paper proposes a reinforcement learning (RL) algorithm for the security problem of state estimation of cyber-physical system (CPS) under denial-of-service (DoS) attacks. The security of CPS will inevitably decline when faced with malicious cyber attacks. In order to analyze the impact of cyber attacks on CPS performance, a Kalman filter, as an adaptive state estimation technology, is combined with an RL method to evaluate the issue of system security, where estimation performance is adopted as an evaluation criterion. Then, the transition of estimation error covariance under a DoS attack is described as a Markov decision process, and the RL algorithm could be applied to resolve the optimal countermeasures. Meanwhile, the interactive combat between defender and attacker could be regarded as a two-player zero-sum game, where the Nash equilibrium policy exists but needs to be solved. Considering the energy constraints, the action selection of both sides will be restricted by setting certain cost functions. The proposed RL approach is designed from three different perspectives, including the defender, the attacker and the interactive game of two opposite sides. In addition, the framework of Q-learning and state–action–reward–state–action (SARSA) methods are investigated separately in this paper to analyze the influence of different RL algorithms. The results show that both algorithms obtain the corresponding optimal policy and the Nash equilibrium policy of the zero-sum interactive game. Through comparative analysis of two algorithms, it is verified that the differences between Q-Learning and SARSA could be applied effectively into the secure state estimation in CPS.

Tactile Routing for Location Privacy Preservation in Wireless Sensor Networks: A Game Theoretic Approach.

Location Privacy Preservation (LPP) in Wireless Sensor Networks (WSNs) during the era of the Internet of things and smart systems is a critical element in the success of WSNs. LPP in WSN can be stated as: given a WSN with an adversary aiming to unravel the location of critical nodes of a WSN, the goal of the WSN manager is to enshroud the location of the critical nodes via routing and/or encryption mechanisms. Typical research in the LPP of WSN routing involves developing and/or estimating the performance of a fixed routing protocol under a given attack mechanism. Motivated by advancements in network softwarization, in this work, we propose an approach where the WSN manager as well as the WSN adversary can deploy multiple routing and attack mechanisms, respectively. Initially, the proposed approach is formulated as a repeated two-player zero-sum game. The formulation is further extended to handle multiple objectives and incomplete information in the game matrix. In this work, the multiple objectives are handled via the epsilon constraint method. The presence of incomplete information in the formulation is modeled as interval based uncertainty. To sum, the proposed formulation ultimately boils down to linear programming problems, which can be efficiently solved. Numerical case studies to showcase the applicability of the proposed approach are illustrated in this work. Finally, discussion on obtaining the required data from any given WSN, discussion and interpretation of the formulation’s results, and future research direction of the current work is presented.

Complete Representations and Neat Embeddings

Let \(2<n<\omega\). Then \({\sf CA}_n\) denotes the class of cylindric algebras of dimension \(n\), \({\sf RCA}_n\) denotes the class of representable \(\sf CA_n\)s, \({\sf CRCA}_n\) denotes the class of completely representable \({\sf CA}_n\)s, and \({\sf Nr}_n{\sf CA}_{\omega}(\subseteq {\sf CA}_n\)) denotes the class of \(n\)-neat reducts of \({\sf CA}_{\omega}\)s. The elementary closure of the class \({\sf CRCA}_n\)s (\(\mathbf{K_n}\)) and the non-elementary class \({\sf At}({\sf Nr}_n{\sf CA}_{\omega})\) are characterized using two-player zero-sum games, where \({\sf At}\) is the operator of forming atom structures. It is shown that \(\mathbf{K_n}\) is not finitely axiomatizable and that it coincides with the class of atomic algebras in the elementary closure of \(\mathbf{S_c}{\sf Nr}_n{\sf CA}_{\omega}\) where \(\mathbf{S_c}\) is the operation of forming complete subalgebras. For any class \(\mathbf{L}\) such that \({\sf At}{\sf Nr}_n{\sf CA}_{\omega}\subseteq \mathbf{L}\subseteq {\sf At}\mathbf{K_n}\), it is proved that \({\bf SP}\mathfrak{Cm}\mathbf{L}={\sf RCA}_n\), where \({\sf Cm}\) is the dual operator to \(\sf At\); that of forming complex algebras. It is also shown that any class \(\mathbf{K}\) between \({\sf CRCA}_n\cap \mathbf{S_d}{\sf Nr}_n{\sf CA}_{\omega}\) and \(\mathbf{S_c}{\sf Nr}_n{\sf CA}_{n+3}\) is not first order definable, where \(\mathbf{S_d}\) is the operation of forming dense subalgebras, and that for any \(2<n<m\), any \(l\geq n+3\) any any class \(\mathbf{K}\) (such that \({\sf At}({\sf Nr}_n{\sf CA}_{m})\cap {\sf CRCA}_n\subseteq \mathbf{K}\subseteq {\sf At}\mathbf{S_c}{\sf Nr}_n{\sf CA}_{l}\), \(\mathbf{K}\) is not not first order definable either.

A Generalized Minimax Q-Learning Algorithm for Two-Player Zero-Sum Stochastic Games

We consider the problem of two-player zero-sum games. This problem is formulated as a min-max Markov game in the literature. The solution of this game, which is the min-max payoff, starting from a given state is called the min-max value of the state. In this work, we compute the solution of the two-player zero-sum game utilizing the technique of successive relaxation that has been successfully applied in the literature to compute a faster value iteration algorithm in the context of Markov Decision Processes. We extend the concept of successive relaxation to the setting of two-player zero-sum games. We show that, under a special structure on the game, this technique facilitates faster computation of the min-max value of the states. We then derive a generalized minimax Q-learning algorithm that computes the optimal policy when the model information is not known. Finally, we prove the convergence of the proposed generalized minimax Q-learning algorithm utilizing stochastic approximation techniques, under an assumption on the boundedness of iterates. Through experiments, we demonstrate the effectiveness of our proposed algorithm.

Model-free adaptive optimal control policy for Markov jump systems: A value iterations algorithm

This article develops a model-free adaptive optimal control policy for discrete-time Markov jump systems. First, a two-player zero-sum game is formulated to obtain an optimal control policy that minimizes a cost function against the worst-case disturbance. Second, an action and mode-dependent value function is set up for zero-sum game to search such a policy with convergence guarantee rather than solving an optimization problem satisfying coupled algebraic Riccati equations. To be specific, motivated by the Bellman optimal principle, we develop an online value iterations algorithm to solve the zero-sum game, which is learning while controlling without any initial stabilizing policy. By this algorithm, we can achieve disturbance attenuation for Markov jump systems without knowledge of the system matrices. The adaptivity to slowly changing uncertainties can also be achieved due to the model-free feature and policy convergence. Finally, the effectiveness and practical potential of the algorithm are demonstrated by considering two numerical examples and a solar boiler system.

Model-free finite-horizon optimal control of discrete-time two-player zero-sum games

Conventionally, as the system dynamics is known, the finite-horizon optimal control of zero-sum games relies on solving the time-varying Riccati equations. In this paper, with unknown system dynamics being considered, a Q-function-based finite-horizon control method is introduced to approximate the solutions of the time-varying Riccati equations. First, a time-varying Q-function explicitly dependent on the time-varying control and disturbance is defined. Then the defined time-varying Q-function is utilised to represent the time-varying control and disturbance which are equivalent to the solutions of the time-varying Riccati equations by relaxing the system dynamics. Finally, a model-free method is introduced to approximate the defined time-varying Q-function. Simulation studies are conducted to demonstrate the validity of the developed method.

Synergetic learning structure-based neuro-optimal fault tolerant control for unknown nonlinear systems

In this paper, a synergetic learning structure-based neuro-optimal fault tolerant control (SLSNOFTC) method is proposed for unknown nonlinear continuous-time systems with actuator failures. Under the framework of the synergetic learning structure (SLS), the optimal control input and the actuator failure are viewed as two subsystems. Then, the fault tolerant control (FTC) problem can be regarded as a two-player zero-sum differential game according to the game theory. A radial basis function neural network-based identifier, which uses the measured input/output data, is constructed to identify the completely unknown system dynamics. To develop the SLSNOFTC method, the Hamilton–Jacobi–Isaacs equation is solved by an asymptotically stable critic neural network (ASCNN) which is composed of cooperative adaptive tuning laws. Besides, with the help of the Lyapunov stability analysis, the identification error, the weight error of ASCNN, and all signals of closed-loop system are guaranteed to be converged to zero asymptotically, rather than uniformly ultimately bounded. Numerical simulation examples further verify the effectiveness and reliability of the proposed method.

A representation theorem approach to probabilistic interpretation for viscosity solutions of Isaacs equations

The main objective of this paper is to show a certain representation theorem for generators of generalized backward stochastic differential equations (GBSDEs) and its application to probabilistic interpretation for viscosity solutions of Hamilton-Jacobi-Bellman-Isaacs equations with nonlinear Neumann boundary value problems. In the representation theorem a time change method is adopted to treat the difficulty brought from the random measure contained in GBSDEs. By means of the representation theorem the viscosity solution of Isaacs equations with nonlinear Neumann problems is proved to be the value function of a two-player zero-sum stochastic differential game problem with state constraints and recursive cost functionals.

One-Clock Priced Timed Games with Negative Weights

Priced timed games are two-player zero-sum games played on priced timed automata (whose locations and transitions are labeled by weights modelling the cost of spending time in a state and executing an action, respectively). The goals of the players are to minimise and maximise the cost to reach a target location, respectively. We consider priced timed games with one clock and arbitrary integer weights and show that, for an important subclass of them (the so-called simple priced timed games), one can compute, in pseudo-polynomial time, the optimal values that the players can achieve, with their associated optimal strategies. As side results, we also show that one-clock priced timed games are determined and that we can use our result on simple priced timed games to solve the more general class of so-called negative-reset-acyclic priced timed games (with arbitrary integer weights and one clock). The decidability status of the full class of priced timed games with one-clock and arbitrary integer weights still remains open.

Security State Estimation for Cyber-Physical Systems against DoS Attacks via Reinforcement Learning and Game Theory

This paper addressed the optimal policy selection problem of attacker and sensor in cyber-physical systems (CPSs) under denial of service (DoS) attacks. Since the sensor and the attacker have opposite goals, a two-player zero-sum game is introduced to describe the game between the sensor and the attacker, and the Nash equilibrium strategies are studied to obtain the optimal actions. In order to effectively evaluate and quantify the gains, a reinforcement learning algorithm is proposed to dynamically adjust the corresponding strategies. Furthermore, security state estimation is introduced to evaluate the impact of offensive and defensive strategies on CPSs. In the algorithm, the ε-greedy policy is improved to make optimal choices based on sufficient learning, achieving a balance of exploration and exploitation. It is worth noting that the channel reliability factor is considered in order to study CPSs with multiple reasons for packet loss. The reinforcement learning algorithm is designed in two scenarios: reliable channel (that is, the reason for packet loss is only DoS attacks) and unreliable channel (the reason for packet loss is not entirely from DoS attacks). The simulation results of the two scenarios show that the proposed reinforcement learning algorithm can quickly converge to the Nash equilibrium policies of both sides, proving the availability and effectiveness of the algorithm.

Equilibria in Pure Strategies for a Two-Player Zero-Sum Average Stochastic Positional Game

The problem of the existence and determining equilibria in pure stationary strategies for a two-player zero-sum average stochastic positional game is considered. We show that for such a game there exists the value and players may achieve the value by applying pure stationary strategies of choosing the actions in their positions. Based on a constructive proof of these results we propose an algorithmic approach for determining the optimal pure stationary strategies of the players.