Single-controller Stochastic Game Research Articles

The Price of History-Independent Strategies in Games with Inter-Temporal Externalities

In this paper, we compare the value of zero-sum stochastic games under optimal strategies (that are, for single-controller stochastic games, stationary) to the commonly used time-independent strategies (“static strategies”). Our findings are summarized in a series of theorems which provide the lower bound on the optimality of the static strategy under different assumptions. These bounds can be used to assess whether the additional computational complexity is worth the extra payoff gain or, symmetrically, assess the price of playing sub-optimal but simple strategies when stationary ones are forbidden.

Jamming and anti‐jamming in interference channels: a stochastic game approach

This study considers jamming and anti-jamming in interference channels by way of smart hopping; and differs from the existing research studies that focus on random hopping in dedicated channels. This study additionally differs from the existing works by way of utilising inference and logical reasoning by both the jammer and target-transmitter to predict the opponent's strategy and choose the most effective counter strategy. The jammer uses a novel strategy to degrade the target-user's signal-to-interference-plus-noise-ratio by directly jamming the target-transmitter's channel, and forcing other transmitters to shift to the target-transmitter's channel. Two scenarios are considered for a target transmitter to response: a high transmit-power target-user utilises smart channel-hopping to force low transmit-power users to concurrently use a channel not hopped by the target-user to confuse the jammer, and a low transmit-power target-user evades the jammer by channel-hopping. The authors propose two-controller and single-controller stochastic games for the first and latter scenarios, respectively, to model interactions with the jammer; and propose a Q-learning algorithm to obtain smart channel hopping sequences for both the jammer and the target-transmitter. Simulations show that the proposed anti-jamming scheme outperforms alternative strategies, including random channel-hopping and transmit-power control in reducing the probability of successful jamming.

Completely Mixed Strategies for Generalized Bimatrix and Switching Controller Stochastic Game

In this paper, we revisit a result by Jurg et al. (Linear Algebra Appl 141:61–74, 1990) where the necessary and sufficient condition for a bimatrix game to be weakly completely mixed is given. We present an alternate proof of this result using linear complementarity approach. We extend this result to a generalization of bimatrix game introduced by Gowda and Sznajder (Int J Game Theory 25:1–12, 1996) via a generalization of linear complementarity problem introduced by Cottle and Dantzig (J Comb Theory 8:79–90, 1970). We further study completely mixed switching controller stochastic game (in which transition structure is a natural generalization of the single controller games) and extend the results obtained by Filar (Proc Am Math Soc 95:585–594, 1985) for completely mixed single controller stochastic game to completely mixed switching controller stochastic game. A numerical method is proposed to compute a completely mixed strategy for a switching controller stochastic game.

Completely mixed strategies for single controller unichain semi-Markov games with undiscounted payoffs

Zero-sum two-person finite undiscounted (limiting ratio average) semi-Markov games are considered where the transition probabilities and the transition times are controlled by a fixed player in all states. We prove that if such a game is unichain, the value and optimal stationary strategies can be obtained from an optimal solution of a linear programming algorithm for the associated undiscounted unichain single controller stochastic game (obtained by a data transformation method). The single controller undiscounted unichain semi-Markov games have been formulated as a linear complementarity problem and solved using a stepwise principal pivoting algorithm. We provide necessary and sufficient conditions for such games to be completely mixed (i.e., every optimal stationary strategy for each player assigns a positive probability to every action in every state). Some properties analogous to completely mixed matrix games are also established in this paper.

Downlink Power Control in Two-Tier Cellular Networks With Energy-Harvesting Small Cells as Stochastic Games

Energy harvesting in cellular networks is an emerging technique to enhance the sustainability of power-constrained wireless devices. This paper considers the co-channel deployment of a macrocell overlaid with small cells. The small cell base stations (SBSs) harvest energy from environmental sources whereas the macrocell base station (MBS) uses conventional power supply. Given a stochastic energy arrival process for the SBSs, we derive a power control policy for the downlink transmission of both MBS and SBSs such that they can achieve their objectives (e.g., maintain the signal-to-interference-plus-noise ratio (SINR) at an acceptable level) on a given transmission channel. We consider a centralized energy harvesting mechanism for SBSs, i.e., there is a central energy storage (CES) where energy is harvested and then distributed to the SBSs. When the number of SBSs is small, the game between the CES and the MBS is modeled as a single-controller stochastic game and the equilibrium policies are obtained as a solution of a quadratic programming problem. However, when the number of SBSs tends to infinity (i.e., a highly dense network), the centralized scheme becomes infeasible, and therefore, we use a mean field stochastic game to obtain a distributed power control policy for each SBS. By solving a system of partial differential equations, we derive the power control policy of SBSs given the knowledge of mean field distribution and the available harvested energy levels in the batteries of the SBSs.

A near-optimal polynomial time algorithm for learning in certain classes of stochastic games

We present a new algorithm for polynomial time learning of optimal behavior in single-controller stochastic games. This algorithm incorporates and integrates important recent results of Kearns and Singh (Proc. ICML-98, 1998) in reinforcement learning and of Monderer and Tennenholtz (J. Artif. Intell. Res. 7, 1997, p. 231) in repeated games. In stochastic games, the agent must cope with the existence of an adversary whose actions can be arbitrary. In particular, this adversary can withhold information about the game matrix by refraining from (or rarely) performing certain actions. This forces upon us an exploration versus exploitation dilemma more complex than in Markov decision processes in which, given information about particular parts of a game matrix, the agent must decide how much effort to invest in learning the unknown parts of the matrix. We present a polynomial time algorithm that addresses these issues in the context of the class of single controller stochastic games, providing the agent with near-optimal return.

Quadratic programming and the single-controller stochastic game

In this paper we show that an optimal solution to an appropriately constructed quadratic program provides a stationary Nash equilibrium point of a two-person, general-sum, single-controller stochastic game. Stochastic games with both the limiting average and the discounted payoff criteria are considered. For the latter, the converse statement also holds; that is, every stationary equilibrium point provides an optimal solution to the quadratic program. The above results include as special cases the known quadratic/linear programming formulations of bimatrix games, matrix games, Markovian decision processes, and single-controller zero-sum stochastic games.

The Completely Mixed Single-Controller Stochastic Game

We consider a zero-sum stochastic game with finitely many states and actions. Further we assume that the transition probabilities depend on the actions of only one player (player II, in our case), and that the game is completely mixed. That is, every optimal stationary strategy for either player assigns a positive probability to every action in every state. For these games, properties analogous to those derived by Kaplansky [4] for the completely mixed matrix games, are established in this paper. These properties lead to the counterintuitive conclusion that the controller need not know the law of motion in order to play optimally, but his opponent does not have this luxury.

The completely mixed single-controller stochastic game

We consider a zero-sum stochastic game with finitely many states and actions. Further we assume that the transition probabilities depend on the actions of only one player (player II, in our case), and that the game is completely mixed. That is, every optimal stationary strategy for either player assigns a positive probability to every action in every state. For these games, properties analogous to those derived by Kaplansky [4] for the completely mixed matrix games, are established in this paper. These properties lead to the counterintuitive conclusion that the controller need not know the law of motion in order to play optimally, but his opponent does not have this luxury.

On stationary equilibria of a single-controller stochastic game

We consider a two-person, general-sum, rational-data, undiscounted stochastic game in which one player (player II) controls the transition probabilities. We show that the set of stationary equilibrium points is the union of a finite number of sets such that, every element of each of these sets can be constructed from a finite number of extreme equilibrium strategies for player I and from a finite number of pseudo-extreme equilibrium strategies for player II. These extreme and pseudo-extreme strategies can themselves be constructed by finite (but inefficient) algorithms. Analogous results can also be established in the more straightforward case of discounted single-controller games.