Two-person Zero-sum Markov Games Research Articles

Two-person zero-sum stochastic games with varying discount factors

<abstract><p>In this paper, two-person zero-sum Markov games with Borel state space and action space, unbounded reward function and state-dependent discount factors are studied. The optimal criterion is expected discount criterion. Firstly, sufficient conditions for the existence of optimal policies are given for the two-person zero-sum Markov games with varying discount factors. Then, the existence of optimal policies is proved by Banach fixed point theorem. Finally, we give an example for reservoir operations to illustrate the existence results.</p></abstract>

Value set iteration for two-person zero-sum Markov games

We present a novel exact algorithm called “value set iteration” (VSI) for solving two-person zero-sum Markov games (MGs) as a generalization of value iteration (VI) and as a general framework of combining multiple solution methods. We introduce a novel operator in the value function space and iteratively apply the operator with any sequence of the set of policies, extending Chang’s VSI for MDPs into the MG setting. We show that VSI for MGs converges to the equilibrium value function with at least linear convergence rate and establish that VSI can potentially improve the convergence speed in terms of the number of iterations by proper setting of the sequence of the set of policies.

Average optimal strategies for zero-sum Markov games with poorly known payoff function on one side

We are concerned with two-person zero-sum Markov games with Borel spaces under a long-run average criterion. The payoff function is possibly unbounded and depends on a parameter which is unknown to one of the players. The parameter and the payoff function can be estimated by implementing statistical methods. Thus, our main objective is to combine such estimation procedure with a variant of the so-called vanishing discount approach to construct an average optimal pair of strategies for the game. Our results are applied to a class of zero-sum semi-Markov games.

Turnpike Theorems for Markov Games

This paper has a two-folded purpose. First, we attempt to outline the development of the turnpike theorems in the the last several decades. Second, we study turnpike theorems in finite-horizon two-person zero-sum Markov games on a general Borel state space. Utilising the Bellman (or Shapley) operator defined for this game, we prove the stochastic versions of the early turnpike theorem on the set of optimal strategies and the middle turnpike theorem on the distribution of the state space.

Adaptive Adversarial Multi-Armed Bandit Approach to Two-Person Zero-Sum Markov Games

This technical note presents a recursive sampling-based algorithm for finite horizon two-person zero-sum Markov games (MGs) based on the Exp3 algorithm developed by Auer et al <i xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">.</i> for adaptive adversarial multi-armed bandit problems. We provide a finite-iteration bound to the equilibrium value of the induced ¿sample average approximation game¿ of a given MG and prove asymptotic convergence to the equilibrium value of the given MG. The time and space complexities of the algorithm are independent of the state space of the game.

Converging Coevolutionary Algorithm for Two-Person Zero-Sum Discounted Markov Games With Perfect Information

This note proposes a novel population-based coevolutionary algorithm, called the ldquolocal-equilibrium-based coevolutionary algorithmrdquo (LECA), for solving infinite-horizon-discounted two-person zero-sum Markov games with perfect information. LECA ldquorandomizesrdquo a simplified variant of Raghavan and Syed's algorithm, which adapted Howard's policy improvement algorithm for solving Markov decision processes into a ldquonegotiation processrdquo algorithm between two players via a lexicographical search. LECA runs over relatively small sets of policies such that the negotiation process proceeds with the sets via ldquopolicy switching,rdquo rather than with the entire policy spaces of the players. The use of policy switching eliminates the action spaces manipulation in value iteration and policy iteration-type algorithms and derives a novel concept of ldquolocal equilibrium.rdquo With the concept incorporated into the LECA, a ldquolocalrdquo equilibrium policy pair is identified as an ldquoelitist,rdquo and kept in the next population, directing the LECA toward finding an equilibrium or a near-equilibrium policy pair. The algorithm is especially targeted to problems where the state space is small but the action spaces of the players are extremely large, and converges with probability one to an equilibrium policy pair for a given game.

Perfect information two-person zero-sum markov games with imprecise transition probabilities

Based on an extension of the controlled Markov set-chain model by Kurano et al. (in J Appl Prob 35:293–302, 1998) into competitive two-player game setting, we provide a model of perfect information two-person zero-sum Markov games with imprecise transition probabilities. We define an equilibrium value for the games formulated with the model in terms of a partial order and then establish the existence of an equilibrium policy pair that achieves the equilibrium value. We further analyze finite-approximation error bounds obtained from a value iteration-type algorithm and discuss some applications of the model.

A Model for Multi-timescaled Sequential Decision-making Processes with Adversary

Extending the multi-timescale model proposed by the author et al. in the context of Markov decision processes, this paper proposes a simple analytical model called M timescale two-person zero-sum Markov Games (MMGs) for hierarchically structured sequential decision-making processes in two players' competitive situations where one player (the minimizer) wishes to minimize their cost that will be paid to the adversary (the maximizer). In this hierarchical model, for each player, decisions in each level in the M-level hierarchy are made in M different discrete timescales and the state space and the control space of each level in the hierarchy are non-overlapping with those of the other levels, respectively, and the hierarchy is structured in a "pyramid" sense such that a decision made at level m (slower timescale) state and/or the state will affect the evolutionary decision making process of the lower-level m+1 (faster timescale) until a new decision is made at the higher level but the lower-level decisions themselves do not affect the transition dynamics of higher levels. The performance produced by the lower-level decisions will affect the higher level decisions for each player. A hierarchical objective function for the minimizer and the maximizer is defined, and from this we define "multi-level equilibrium value function" and derive a "multi-level equilibrium equation". We also discuss how to solve hierarchical games exactly.

Two-person zero-sum markov games: receding horizon approach

We consider a receding horizon approach as an approximate solution to two-person zero-sum Markov games with infinite horizon discounted cost and average cost criteria. We first present error bounds from the optimal equilibrium value of the game when both players take "correlated" receding horizon policies that are based on exact or approximate solutions of receding finite horizon subgames. Motivated by the worst-case optimal control of queueing systems by Altman, we then analyze error bounds when the minimizer plays the (approximate) receding horizon control and the maximizer plays the worst case policy. We finally discuss some state-space size independent methods to compute the value of the subgame approximately for the approximate receding horizon control, along with heuristic receding horizon policies for the minimizer.

Successive approximations for average reward Markov games

This paper considers two-person zero-sum Markov games with finitely many states and actions with the criterion of average reward per unit time. Two special situations are treated and it is shown that in both cases the method of successive approximations yields ane-band for the value of the game as well as stationarye-optimal strategies. In the first case all underlying Markov chains of pure stationary optimal strategies are assumed to be unichained. In the second case it is assumed that the functional equation Uv=v+ge has a solution.