Repeated Matrix Games Research Articles

Decoupled Monte Carlo Tree Search for Cooperative Multi-Agent Planning

The number of agents exponentially increases the complexity of a cooperative multi-agent planning problem. Decoupled planning is one of the viable approaches to reduce this complexity. By integrating decoupled planning with Monte Carlo Tree Search, we present a new scalable planning approach. The search tree maintains the updates of the individual actions of each agent separately. However, this separation brings coordination and action synchronization problems. When the agent does not know the action of the other agent, it uses the returned reward to deduce the desirability of its action. When a deterministic action selection policy is used in the Monte Carlo Tree Search algorithm, the actions of agents are synchronized. Of all possible action combinations, only some of them are evaluated. We show the effect of action synchronization on different problems and propose stochastic action selection policies. We also propose a combined method as a pruning step in centralized planning to address the coordination problem in decoupled planning. We create a centralized search tree with a subset of joint actions selected by the evaluation of decoupled planning. We empirically show that decoupled planning has a similar performance compared to a central planning algorithm when stochastic action selection is used in repeated matrix games and multi-agent planning problems. We also show that the combined method improves the performance of the decoupled method in different problems. We compare the proposed method to a decoupled method in regard to a warehouse commissioning problem. Our method achieved more than 10% improvement in performance.

A Web Gaming Facility for Research and Teaching

This essay considers the potential for utilizing web games for research and teaching. It discusses a specific gaming facility that has been constructed and utilized. The gaming facility can be made available for use for those interested in utilizing it for teaching and/or research purposes. The goal is to have this facility be of use for both single play and repeated matrix games. Much of the discussion here is aimed at single play games as a desirable benchmark preliminary to the study of repeated games. Properties of the one stage games are discussed and instructions for the use of the system are supplied. Extensions to multistage games and incomplete information are noted.

Multiagent learning in the presence of memory-bounded agents

In recent years, great strides have been made towards creating autonomous agents that can learn via interaction with their environment. When considering just an individual agent, it is often appropriate to model the world as being stationary, meaning that the same action from the same state will always yield the same (possibly stochastic) effects. However, in the presence of other independent agents, the environment is not stationary: an action’s effects may depend on the actions of the other agents. This non-stationarity poses the primary challenge of multiagent learning and comprises the main reason that it is best considered distinctly from single agent learning. The multiagent learning problem is often studied in the stylized settings provided by repeated matrix games. The goal of this article is to introduce a novel multiagent learning algorithm for such a setting, called Convergence with Model Learning and Safety (or CMLeS), that achieves a new set of objectives which have not been previously achieved. Specifically, CMLeS is the first multiagent learning algorithm to achieve the following three objectives: (1) converges to following a Nash equilibrium joint-policy in self-play; (2) achieves close to the best response when interacting with a set of memory-bounded agents whose memory size is upper bounded by a known value; and (3) ensures an individual return that is very close to its security value when interacting with any other set of agents. Our presentation of CMLeS is backed by a rigorous theoretical analysis, including an analysis of sample complexity wherever applicable.

A Web Gaming Facility for Research and Teaching

This essay considers the potential for utilizing web games for research and teaching. It discusses a specific gaming facility that has been constructed and utilized. The gaming facility can be made available for use for those interested in utilizing it for teaching and/or research purposes. The goal is to have this facility be of use for both single play and repeated matrix games. Much of the discussion here is aimed at single play games as a desirable benchmark preliminary to the study of repeated games. Properties of the one stage games are discussed and instructions for the use of the system are supplied. Extensions to multistage games and incomplete information are noted.

Regret minimization in repeated matrix games with variable stage duration

Regret minimization in repeated matrix games has been extensively studied ever since Hannan's seminal paper [Hannan, J., 1957. Approximation to Bayes risk in repeated play. In: Dresher, M., Tucker, A.W., Wolfe, P. (Eds.), Contributions to the Theory of Games, vol. III. Ann. of Math. Stud., vol. 39, Princeton Univ. Press, Princeton, NJ, pp. 97–193]. Several classes of no-regret strategies now exist; such strategies secure a long-term average payoff as high as could be obtained by the fixed action that is best, in hindsight, against the observed action sequence of the opponent. We consider an extension of this framework to repeated games with variable stage duration, where the duration of each stage may depend on actions of both players, and the performance measure of interest is the average payoff per unit time. We start by showing that no-regret strategies, in the above sense, do not exist in general. Consequently, we consider two classes of adaptive strategies, one based on Blackwell's approachability theorem and the other on calibrated play, and examine their performance guarantees. We further provide sufficient conditions for existence of no-regret strategies in this model.

Dynamic fictitious play, dynamic gradient play, and distributed convergence to Nash equilibria

We consider a continuous-time form of repeated matrix games in which player strategies evolve in reaction to opponent actions. Players observe each other's actions, but do not have access to other player utilities. Strategy evolution may be of the best response sort, as in fictitious play, or a gradient update. Such mechanisms are known to not necessarily converge. We introduce a form of "dynamic" fictitious and gradient play strategy update mechanisms. These mechanisms use derivative action in processing opponent actions and, in some cases, can lead to behavior converging to Nash equilibria in previously nonconvergent situations. We analyze convergence in the case of exact and approximate derivative measurements of the dynamic update mechanisms. In the ideal case of exact derivative measurements, we show that convergence to Nash equilibrium can always be achieved. In the case of approximate derivative measurements, we derive a characterization of local convergence that shows how the dynamic update mechanisms can converge if the traditional static counterparts do not. We primarily discuss two player games, but also outline extensions to multiplayer games. We illustrate these methods with convergent simulations of the well known Shapley and Jordan counterexamples.

The Empirical Bayes Envelope and Regret Minimization in Competitive Markov Decision Processes

This paper proposes an extension of the regret minimizing framework from repeated matrix games to stochastic game models, under appropriate recurrence conditions. A decision maker, P1, who wishes to maximize his long-term average reward is facing a Markovian environment, which may also be affected by arbitrary actions of other agents. The latter are collectively modeled as a second player, P2, whose strategy is arbitrary. Both states and actions are fully observed by both players. While P1 may obviously secure the min-max value of the game, he may wish to improve on that when the opponent is not playing a worst-case strategy. For repeated matrix games, an achievable goal is presented by the Bayes envelope, that traces P1's best-response payoff against the observable frequencies of P2's actions. We propose a generalization to the stochastic game framework, under recurrence conditions that amount to fixed-state reachability. The empirical Bayes envelope (EBE) is defined as P1's best-response payoff against the stationary strategies of P2 that agree with the observed state-action frequencies. Because the EBE may not be attainable in general, we consider its lower convex hull, the convex Bayes envelope (CBE), which is proved to be achievable by P1. The analysis relies on Blackwell's approachability theory. The CBE is lower bounded by the value of the game and for irreducible games turns out to be strictly above the value whenever P2's frequencies deviate from a worst-case strategy. In the special case of single-controller games where P2 alone affects the state transitions, the EBE itself is shown to be attainable.

Adaptation in games: Two solutions to the Crawford puzzle

When agents repeatedly face the same problem, when there is a unique equilibrium, and when other circumstances are favorable, it is commonly supposed that adaptive behavior will lead the agents to the equilibrium. Crawford has presented a striking counterexample in the context of endlessly repeated matrix games. In his model, plausible adaptation leads to endless cycling rather than to the Nash equilibrium. The puzzle is to find adaptive rules which are as plausible as Crawford's rule but which nonetheless lead to the equilibrium. Two solutions to the puzzle are proposed, one based on forecasting considerations and one based on satisficing.

Repeated matrix games

A matrix game is played repeatedly, with the actions taken at each stage determining both a reward paid to Player I and the probability of continuing to the next stage. An infinite history of play determines a sequence (Rn ) of such rewards, to which we assign the payoff lim supn (R 1 + · ·· + Rn ). Using the concept of playable strategies, we slightly generalize the definition of the value of a game. Then we find sufficient conditions for the existence of a value and for the existence of stationary optimal strategies (Theorems 8 and 9). An example shows that the game need not have a value (Example 4).

Repeated matrix games

A matrix game is played repeatedly, with the actions taken at each stage determining both a reward paid to Player I and the probability of continuing to the next stage. An infinite history of play determines a sequence (Rn) of such rewards, to which we assign the payoff lim supn (R1 + · ·· + Rn). Using the concept of playable strategies, we slightly generalize the definition of the value of a game. Then we find sufficient conditions for the existence of a value and for the existence of stationary optimal strategies (Theorems 8 and 9). An example shows that the game need not have a value (Example 4).

Infinitely repeated matrix games for which pure strategies suffice

Infinitely repeated matrix games for which pure strategies suffice