Commutative Stochastic Games

Abstract

We are interested in the convergence of the value of n-stage games as n goes to infinity and the existence of the uniform value in stochastic games with a general set of states and finite sets of actions where the transition is commutative. This means that playing an action profile a1 followed by an action profile a2, leads to the same distribution on states as playing first the action profile a2 and then a1. For example, absorbing games can be reformulated as commutative stochastic games. When there is only one player and the transition function is deterministic, we show that the existence of a uniform value in pure strategies implies the existence of 0-optimal strategies. In the framework of two-player stochastic games, we study a class of games where the set of states is ℝm and the transition is deterministic and 1-Lipschitz for the L1-norm, and prove that these games have a uniform value. A similar proof shows the existence of an equilibrium in the nonzero-sum case. These results remain true if one considers a general model of finite repeated games, where the transition is commutative and the players observe the past actions but not the state.