Two-player Zero-sum Stochastic Game Research Articles

Absorbing games with irrational values

Many classes of two-player zero-sum stochastic games have the orderfield property; that is, if all payoffs and transitions belong to some field, so does the limit value. Is it also the case for absorbing games? No: In this note, we exhibit m×m absorbing games with rational data whose limit values are algebraic of degree m, for each m∈N⁎. Furthermore, we provide maximal conditions for the orderfield property to hold, namely if transitions are deterministic and one player has at most two actions. Last, we prove that any algebraic number of degree 2 is the limit value of a 2×2 absorbing game, which leads to the conjecture that any algebraic number of degree m is the limit value of an m×m absorbing game.

Subgame Maxmin Strategies in Zero-Sum Stochastic Games with Tolerance Levels

We study subgame phi -maxmin strategies in two-player zero-sum stochastic games with a countable state space, finite action spaces, and a bounded and universally measurable payoff function. Here, phi denotes the tolerance function that assigns a nonnegative tolerated error level to every subgame. Subgame phi -maxmin strategies are strategies of the maximizing player that guarantee the lower value in every subgame within the subgame-dependent tolerance level as given by phi . First, we provide necessary and sufficient conditions for a strategy to be a subgame phi -maxmin strategy. As a special case, we obtain a characterization for subgame maxmin strategies, i.e., strategies that exactly guarantee the lower value at every subgame. Secondly, we present sufficient conditions for the existence of a subgame phi -maxmin strategy. Finally, we show the possibly surprising result that each game admits a strictly positive tolerance function phi ^* with the following property: if a player has a subgame phi ^*-maxmin strategy, then he has a subgame maxmin strategy too. As a consequence, the existence of a subgame phi -maxmin strategy for every positive tolerance function phi is equivalent to the existence of a subgame maxmin strategy.

A numerical optimization pesudo-algorithm for two-player zero-sum stochastic games

ABSTRACT A two-player, zero-sum stochastic game with two variables can be transformed into the coupled control PDEs by a transition probability matrix of the Markov chain. There exist two control variables and two tensity rates () after the transformation that should be solved for. Yuan and Li (Computational Economics, 2018) give a numerical algorithm by the first-order necessary condition, which overcomes several drawbacks of the normal algorithm and describe several numerical experiments demonstrating the performance of their method. However, their method exhibits at least one shortcoming, which is that the final value of the control variables () may exceed the definition domain. This situation means that the () is infeasible and unacceptable. This paper presents a new technique to avoid that situation, and an optimization pseudo-algorithm is designed using the following steps: (i) starting from the given initial points an active-set algorithm is proposed; (ii) the limited memory update technique is used in the algorithm to obtain fast convergence and low storage; (iii) global convergence is established under suitable conditions; and (iv) numerical results are reported to demonstrate that the new algorithm is competitive with the normal algorithm.

Stochastic Game in Linear Quadratic Gaussian Control for Wireless Networked Control Systems Under DoS Attacks

Recently, the security problem of wireless networked control systems (WNCSs) has become critical. In this article, we focus on strategies designing problem for sensor and denial-of-service (DoS) attacker in WNCS based on linear quadratic Gaussian (LQG) cost. In this scenario, a sensor measures the output of the system and transmits its local state estimate to the remote estimator via an unreliable channel which may suffer interference from an intelligent DoS attacker. In each step, the sensor demands to determine the power to transmit its packet data, at the same time, the attacker needs to determine the interference power to degrade the performance of the WNCS. To analyze this interactive decision-making process, we construct a two-player zero-sum stochastic game and propose an improved LQG cost considering energy-efficient factors. Then, for continuous power-level setting, we discuss the existence and the structure of equilibrium strategies of the sensor–attacker game. For a discrete power set, an algorithm is developed to solve equilibrium strategies for both players. Finally, we present an example to illustrate our results.

Heuristic Search Value Iteration for Zero-Sum Stochastic Games

In sequential decision making, heuristic search algorithms allow exploiting both the initial situation and an admissible heuristic to efficiently search for an optimal solution, often for planning purposes. Such algorithms exist for problems with uncertain dynamics, partial observability, multiple criteria, or multiple collaborating agents. In this article, we look at two-player zero-sum stochastic games (zsSGs) with a discounted criterion, in a view to propose a solution tailored to the fully observable case, while solutions have been proposed for particular, though still more general, partially observable cases. This setting induces reasoning on both a lower and an upper bound of the value function, which leads us to proposing zsSG-HSVI, an algorithm based on heuristic search value iteration (HSVI), and which thus relies on generating trajectories. We demonstrate that, each player acting optimistically, and employing simple heuristic initializations, HSVI's convergence in finite time to an ∈-optimal solution is preserved. An empirical study of the resulting approach is conducted on benchmark problems of various sizes.

Two-player zero-sum stochastic differential games with random horizon

AbstractWe consider a two-player zero-sum stochastic differential game with a random planning horizon and diffusive state variable dynamics. The random planning horizon is a function of a non-negative continuous random variable, which is assumed to be independent of the Brownian motion driving the state variable dynamics. We study this game using a combination of dynamic programming and viscosity solution techniques. Under some mild assumptions, we prove that the value of the game exists and is the unique viscosity solution of a certain nonlinear partial differential equation of Hamilton–Jacobi–Bellman–Isaacs type.

Dynamic competition over social networks

We propose an analytical approach to the problem of influence maximization in a social network where two players compete by means of dynamic targeting strategies. We formulate the problem as a two-player zero-sum stochastic game. We prove the existence of the uniform value: if the players are sufficiently patient, both can guarantee the same mean-average opinion without knowing the exact length of the game. Furthermore, we put forward some elements for the characterization of equilibrium strategies. In general, players must implement a trade-off between a forward-looking perspective, according to which they aim to maximize the future spread of their opinion in the network, and a backward-looking perspective, according to which they aim to counteract their opponent’s previous actions. When the influence potential of players is small, we describe an equilibrium through a one-shot game based on eigenvector centrality.

Simplifying optimal strategies in limsup and liminf stochastic games

We consider two-player zero-sum stochastic games with the limsup and with the liminf payoffs. For the limsup payoff, we prove that the existence of an optimal strategy implies the existence of a stationary optimal strategy. Our construction does not require the knowledge of an optimal strategy, only its existence. The main technique of the proof is to analyze the game with specific restricted action spaces. For the liminf payoff, we prove that the existence of a subgame-optimal strategy (i.e. a strategy that is optimal in every subgame) implies the existence of a subgame-optimal strategy under which the prescribed mixed actions only depend on the current state and on the state and the actions chosen at the previous period. In particular, such a strategy requires only finite memory. The proof relies on techniques that originate in gambling theory.

Markov decision processes and stochastic games with total effective payoff

We consider finite Markov decision processes with undiscounted total effective payoff. We show that there exist uniformly optimal pure and stationary strategies that can be computed by solving a polynomial number of linear programs. This implies that in a two-player zero-sum stochastic game with perfect information and with total effective payoff there exists a stationary best response to any stationary strategy of the opponent. From this, we derive the existence of a uniformly optimal pure and stationary saddle point. Finally we show that mean payoff can be viewed as a special case of total payoff.

Subgame Maxmin Strategies in Zero-Sum Stochastic Games with Tolerance Levels

We study subgame φ-maxmin strategies in two-player zero-sum stochastic games with finite action spaces and a countable state space. Here φ denotes the tolerance function, a function which assigns a non-negative tolerated error level to every subgame. Subgame φ-maxmin strategies are strategies of the maximizing player that guarantee the lower value in every subgame within the subgame-dependent tolerance level as given by φ. First, we provide necessary and sufficient conditions for a strategy to be a subgame φ-maxmin strategy. As a special case we obtain a characterization for subgame maxmin strategies, i.e. strategies that exactly guarantee the lower value at every subgame. Secondly, we present sufficient conditions for the existence of a subgame φ-maxmin strategy. Finally, we show the possibly surprising result that the existence of subgame φ-maxmin strategies for every positive tolerance function φ is equivalent to the existence of a subgame maxmin strategy.

Dynamic Competition on Social Networks

We provide an analytical approach to the problem of influence maximization in a social network when two players compete by means of dynamic targeting strategies. We formulate the problem as a two-player zero-sum stochastic game. We prove the existence of the uniform value: if the players are sufficiently patient, both players can guarantee the same mean-average opinion without knowing the exact discount factor. Further, we put forward some elements for the characterization of equilibrium strategies. In general, players must implement a trade-off between a forward-looking perspective, according to which they shall aim at maximizing the future spread of their opinion in the network, and a backward-looking perspective, according to which they shall aim at counteracting their opponent's previous actions. When the influence potential of players is small, an equilibrium strategy is to systematically target the agent with the largest eigenvector centrality.

A multi-channel transmission schedule for remote state estimation under DoS attacks

This paper considers a cyber-physical system (CPS) under denial-of-service (DoS) attacks. The measurements of a sensor are transmitted to a remote estimator over a multi-channel network, which may be congested by a malicious attacker. Among these multiple communication paths with different characteristics and properties at each time step, the sensor needs to choose a single channel for sending data packets while reducing the probability of being attacked. In the meanwhile, the attacker needs to decide the target channel to jam under an energy budget constraint. To model this interactive decision-making process between the two sides, we formulate a two-player zero-sum stochastic game framework. A Nash Q-learning algorithm is proposed to tackle the computation complexity when solving the optimal strategies for both players. Numerical examples are provided to illustrate the obtained results.

A stochastic game approach to the security issue of networked control systems under jamming attacks

Securing networked control systems (NCSs) from cyber attacks has been a very important issue to keep NCSs reliable and stable. Most existing efforts tackling this issue treat cyber attacks as model-based disturbances to NCSs. But the reality is that intelligent attackers will not follow any prescribed models and in fact they are able to change their attack strategies dynamically and randomly. In this paper, we address this problem and present an optimal defense mechanism for the NCS under jamming attacks based on the stochastic game theory. A two-player zero-sum stochastic game is formulated to model the dynamic interactions between a jammer (attacker) and a sensor transmitter (defender) in the NCS. In this stochastic game, the cost function includes not only the resource costs used to conduct cyber-layer defense and attack actions, but also the possible degraded dynamic performance (indexed by a quadratic state error) of the NCS. With this cost function, the impacts of the interactions between the attacker and the defender on the dynamic performance of the NCS are taken into account when the two players design/change their cyber-layer strategies. The optimal defense mechanism is obtained by solving a stochastic dynamic programming (SDP) problem. Simulation and comparison studies show that the packet-loss rate of the communication channel of the NCS has been greatly reduced and the dynamic performance of the NCS being attacked by an intelligent jammer is much improved when the proposed defense mechanism is deployed.

Modeling Attacker-Defender Interaction as a Zero-Sum Stochastic Game

Game-theoretic modeling of computer security views security attack scenarios as an optimization game comprising of multiple players notably the attackers and the defenders (system administrators). This paper first presents theoretically, a two-player zero-sum stochastic game model of the interaction between malicious users and network administrators and secondly introduces a hypothetical network of a typical scenario to show the applicability of our model within that scenario. State games are encoded using a binary scheme in order to properly capture components of the underlying network environment. Our solution involves reducing each state game into a min and max linear programming problems for both the defender and attacker respectively. Game costs, rewards and outcomes are modeled to closely match real world measurements. We propose the use of a combination of the pivotal algorithm and a custom stochastic algorithm to compute the optimal (best-response) strategies for the players at each state. We also describe how the results can be analyzed to show how the optimal strategies can be used by the network administrators to predict adversary's actions, determine vulnerable network assets and suggest optimal defense strategies.

A Stochastic Security Game for Kalman Filtering in Networked Control Systems (NCSs) under Denial of Service (DoS) Attacks

In this paper, we investigate the problem how to design optimal security mechanism for the Kalman filtering of Networked Control Systems (NCSs) with a DoS attacker under limited resources. A two-player zero-sum stochastic game is formulated to model not only the dynamic interaction between the network administrator (as a defender) and the DoS attacker (as an attacker), but also the transitions between the two states (on/off) of the communication link. In specific, we take the degraded dynamic performance of physical systems into consideration when we design the network security mechanism. In this stochastic game, the cost function consists of two parts: one component is the network resource cost, the other one is the degraded dynamic performance of the Kalman filtering of the NCS, indexing by the trace of the estimation error covariance matrix. The optimal defense and attack strategies are obtained by using stochastic dynamic programming (SDP). A linear NCS is used to illustrate the procedure of the proposed stochastic game formulation and solving it by SDP.

A survey of partial-observation stochastic parity games

We consider two-player zero-sum stochastic games on graphs with ?-regular winning conditions specified as parity objectives. These games have applications in the design and control of reactive systems. We survey the complexity results for the problem of deciding the winner in such games, and in classes of interest obtained as special cases, based on the information and the power of randomization available to the players, on the class of objectives and on the winning mode. On the basis of information, these games can be classified as follows: (a) partial-observation (both players have partial view of the game); (b) one-sided partial-observation (one player has partial-observation and the other player has complete-observation); and (c) complete-observation (both players have complete view of the game). The one-sided partial-observation games have two important subclasses: the one-player games, known as partial-observation Markov decision processes (POMDPs), and the blind one-player games, known as probabilistic automata. On the basis of randomization, (a) the players may not be allowed to use randomization (pure strategies), or (b) they may choose a probability distribution over actions but the actual random choice is external and not visible to the player (actions invisible), or (c) they may use full randomization. Finally, various classes of games are obtained by restricting the parity objective to a reachability, safety, Buchi, or coBuchi condition. We also consider several winning modes, such as sure-winning (i.e., all outcomes of a strategy have to satisfy the winning condition), almost-sure winning (i.e., winning with probability 1), limit-sure winning (i.e., winning with probability arbitrarily close to 1), and value-threshold winning (i.e., winning with probability at least ?, where ? is a given rational).

Algorithms for uniform optimal strategies in two-player zero-sum stochastic games with perfect information

We deal with zero-sum two-player stochastic games with perfect information. We propose two algorithms to find the uniform optimal strategies and one method to compute the optimality range of discount factors. We prove the convergence in finite time for one algorithm. The uniform optimal strategies are also optimal for the long run average criterion and, in transient games, for the undiscounted criterion as well.

Stochastic games with information lag

Two-player zero-sum stochastic games with finite state and action spaces are known to have undiscounted values. We study such games under the assumption that one or both players observe the actions of their opponent after some time-dependent delay. We develop criteria for the rate of growth of the delay such that a player subject to such an information lag can still guarantee himself in the undiscounted game as much as he could have with perfect monitoring. We also demonstrate that the player in the Big Match with the absorbing action subject to information lags that grows too rapidly will not be able to guarantee as much as he could have in the game with perfect monitoring.

Solving Simple Stochastic Games with Few Random Vertices

Simple stochastic games are two-player zero-sum stochastic games with turn-based moves, perfect information, and reachability winning conditions. We present two new algorithms computing the values of simple stochastic games. Both of them rely on the existence of optimal permutation strategies, a class of positional strategies derived from permutations of the random vertices. The "permutation-enumeration" algorithm performs an exhaustive search among these strategies, while the "permutation-improvement" algorithm is based on successive improvements, \`a la Hoffman-Karp. Our algorithms improve previously known algorithms in several aspects. First they run in polynomial time when the number of random vertices is fixed, so the problem of solving simple stochastic games is fixed-parameter tractable when the parameter is the number of random vertices. Furthermore, our algorithms do not require the input game to be transformed into a stopping game. Finally, the permutation-enumeration algorithm does not use linear programming, while the permutation-improvement algorithm may run in polynomial time.

Computing uniformly optimal strategies in two-player stochastic games

We provide a computable algorithm to calculate uniform e-optimal strategies in two-player zero-sum stochastic games. Our approach can be used to construct algorithms that calculate uniform e-equilibria and uniform correlated e-equilibria in various classes of multi-player non-zero-sum stochastic games.