Pure Strategy Profiles Research Articles

A stochastic variant of replicator dynamics in zero-sum games and its invariant measures

We study the behavior of a stochastic variant of replicator dynamics in two-agent zero-sum games. Inspired by the well studied deterministic replicator equation, this model gives arguably a more realistic description of real world settings and, as we demonstrate here, exhibits totally distinct behavior when compared to its deterministic counterpart which is known to be recurrent. In more detail, we characterize the statistics of such systems by their invariant measures which can be shown to be entirely supported on the boundary of the space of mixed strategies. Depending on the noise strength we can furthermore characterize these invariant measures by finding accumulation of mass at specific parts of the boundary. In particular, regardless of the magnitude of noise, we show that any invariant probability measure is a convex combination of Dirac measures on pure strategy profiles, which correspond to vertices/corners of the agents’ simplices. Thus, in the presence of stochastic perturbations, even in the most classic zero-sum settings, such as Matching Pennies, we observe a stark disagreement between the axiomatic prediction of Nash equilibrium and the evolutionary emergent behavior derived by an assumption of stochastically adaptive, learning agents.

Coalitional strategic games

In pursuit of games played by groups of individuals (each group itself being a player), we develop a theory of strategic games in which each player is rational in the sense of expected utility theory, except that her preferences may fail to be transitive. Two natural solution concepts are defined, Nash equilibrium and the equilibrium in beliefs, depending on the interpretation of mixed strategies. We provide sufficient conditions for the existence of both equilibrium concepts. For instance, it turns out that an equilibrium is sure to exist if each player possesses two pure strategies (and may have cyclic preferences across pure and mixed strategy profiles), without any further qualifications. To go beyond equilibrium existence, we use the coalitional expected utility representation by Hara et al. (2019) and characterize the set of Nash equilibria in terms of this representation. We also study rationalizability in such games (without transitivity), as well as some equilibrium refinements, and compare our findings with those of standard game theory. Our investigation is meant to be a step toward understanding the nature of strategic interaction across groups of individuals and clarifying the role of transitivity in game theory.

Frequency competition among airlines on coordinated airports network

Frequency competition is critical for a full-service airline in gaining market share, and adopting a proper strategy can improve an airline’s profits. This study proposes a new equilibrium programming model with flow balance to address frequency competition on airports network with time slot constraints. We first show that a pure-strategy Nash equilibrium may not always exist, and thus forming a pure strategy profile in frequency competition among airlines may naturally lead to deviation from current frequency. Therefore, we formulate the problem as a programming model with a mixed-strategy Nash equilibrium. To avoid shocks from dramatic frequency changes across the network, airlines tend to fine-tune frequencies on select segments during each adjustment. We propose a procedure to generate a computationally tractable amount of representative strategies from a finite set of feasible strategies to demonstrate mixed-strategy Nash equilibrium. We conduct an empirical analysis using an example in which industry profitability increased by as much as 7.89%. We then extend the model to formulate frequency competition among metal-neutral alliances. The results show that forming metal-neutral alliances can improve total industry profits by 10.59%. In particular, a sensitivity analysis with real data on the tolerance of flow imbalance demonstrates that deducting the potential costs due to the relaxation of flow balance between congested airports may earn additional total industry profits in a frequency competition.

\u03b1-Rank: Multi-Agent Evaluation by Evolution

We introduce α-Rank, a principled evolutionary dynamics methodology, for the evaluation and ranking of agents in large-scale multi-agent interactions, grounded in a novel dynamical game-theoretic solution concept called Markov-Conley chains (MCCs). The approach leverages continuous-time and discrete-time evolutionary dynamical systems applied to empirical games, and scales tractably in the number of agents, in the type of interactions (beyond dyadic), and the type of empirical games (symmetric and asymmetric). Current models are fundamentally limited in one or more of these dimensions, and are not guaranteed to converge to the desired game-theoretic solution concept (typically the Nash equilibrium). α-Rank automatically provides a ranking over the set of agents under evaluation and provides insights into their strengths, weaknesses, and long-term dynamics in terms of basins of attraction and sink components. This is a direct consequence of the correspondence we establish to the dynamical MCC solution concept when the underlying evolutionary model’s ranking-intensity parameter, α, is chosen to be large, which exactly forms the basis of α-Rank. In contrast to the Nash equilibrium, which is a static solution concept based solely on fixed points, MCCs are a dynamical solution concept based on the Markov chain formalism, Conley’s Fundamental Theorem of Dynamical Systems, and the core ingredients of dynamical systems: fixed points, recurrent sets, periodic orbits, and limit cycles. Our α-Rank method runs in polynomial time with respect to the total number of pure strategy profiles, whereas computing a Nash equilibrium for a general-sum game is known to be intractable. We introduce mathematical proofs that not only provide an overarching and unifying perspective of existing continuous- and discrete-time evolutionary evaluation models, but also reveal the formal underpinnings of the α-Rank methodology. We illustrate the method in canonical games and empirically validate it in several domains, including AlphaGo, AlphaZero, MuJoCo Soccer, and Poker.

Individual upper semicontinuity and subgame perfect epsilon -equilibria in games with almost perfect information

We study games with almost perfect information and an infinite time horizon. In such games, at each stage, the players simultaneously choose actions from finite action sets, knowing the actions chosen at all previous stages. The payoff of each player is a function of all actions chosen during the game. We define and examine the new condition of individual upper semicontinuity on the payoff functions, which is weaker than upper semicontinuity. We prove that a game with individual upper semicontinuous payoff functions admits a subgame perfect epsilon -equilibrium for every epsilon >0, in eventually pure strategy profiles.

Metastability of the Logit Dynamics for Asymptotically Well-Behaved Potential Games

Convergence rate and stability of a solution concept are classically measured in terms of “eventually” and “forever,” respectively. In the wake of recent computational criticisms to this approach, we study whether these timeframes can be updated to have states computed “quickly” and stable for “long enough”. Logit dynamics allows irrationality in players’ behavior and may take time exponential in the number of players n to converge to a stable state (i.e., a certain distribution over pure strategy profiles). We prove that every potential game, for which the behavior of the logit dynamics is not chaotic as n increases, admits distributions stable for a super-polynomial number of steps in n no matter the players’ irrationality and the starting profile of the dynamics. The convergence rate to these metastable distributions is polynomial in n when the players are not too rational. Our proofs build upon the new concept of partitioned Markov chains , which might be of independent interest, and a number of involved technical contributions.

Individual Upper Semicontinuity and Subgame Perfect ϵ-Equilibria in Games with Almost Perfect Information

We study games with almost perfect information and an infinite time horizon. In such games, at each stage, the players simultaneously choose actions from finite action sets, knowing the actions chosen at all previous stages. The payoff of each player is a function of all actions chosen during the game. We define and examine the new condition of individual upper semicontinuity on the payoff functions, which is weaker than upper semicontinuity. We prove that a game with individual upper semicontinuous payoff functions admits a subgame perfect ϵ-equilibrium for every ϵ > 0, in eventually pure strategy profiles.

Bidirectional constraint satisfaction in rational strategic decision making

We describe the properties of a constraint satisfaction network that is able to reason and decide rationally in strategic games. We use the structure of Bidirectional Associative Memory (BAM), a minimal two-layer recurrent neural network, and assume that network layers represent self and other strategies, whereas connection weights encode best responses. We apply BAM to finite-strategy two-player games, and show that network activation in the long run is restricted to the set of rationalizable strategies. The network is not guaranteed to reach a stable activation state, but any pure strategy profile that constitutes a stable state in the network must be a pure strategy Nash equilibrium. We illustrate the properties of the network using the traveler’s dilemma, the rock–paper–scissors game, and coordination games. The network’s behavior also depends on starting activation states, and we show how biases in these starting states can resolve equilibrium selection problems. Strategic decision making is a key part of complex social behavior, and our results illustrate how bidirectional constraint satisfaction networks can perform rational computations in this domain.

Alpha cores of games with nonatomic asymmetric information

In this paper, we ask under what reasonable conditions a game with asymmetric information on a continuum of states admits a non-empty α-core. Players examine various private information-constrained contracts f (pure strategy profiles) for ex-ante efficiency by evaluating ex-ante expected payoffs and by forming ex-ante coalitions. Once the players agree on a contract, they implement it faithfully in the interim stage. Roughly speaking, our conclusion states that if players hold fine (non-atomic) and independent information, there exists an ex-ante efficient set of contracts (an ex-ante α-core pure strategy profile) that is implementable in the interim stage. To prove that α-cores are non-empty, we need a variant of Lyapunov’s theorem for Young measures that preserves private information. We apply an iterated integral version of Lyapunov’s theorem for Young measures to derive such a variant.

Matrix games with missing, interval, and ambiguous lottery payoffs of pure strategy profiles and compound strategy profiles

In a matrix game, the interactions among players are based on the assumption that each player has accurate information about the payoffs of their interactions and the other players are rationally self-interested. As a result, the players should definitely take Nash equilibrium strategies. However, in real-life, when choosing their optimal strategies, sometimes the players have to face missing, imprecise (i.e., interval), ambiguous lottery payoffs of pure strategy profiles and even compound strategy profile, which means that it is hard to determine a Nash equilibrium. To address this issue, in this paper we introduce a new solution concept, called ambiguous Nash equilibrium, which extends the concept of Nash equilibrium to the one that can handle these types of ambiguous payoff. Moreover, we will reveal some properties of matrix games of this kind. In particular, we show that a Nash equilibrium is a special case of ambiguous Nash equilibrium if the players have accurate information of each player's payoff sets. Finally, we give an example to illustrate how our approach deals with real-life game theory problems.

Best-Response Cycles in Perfect Information Games

We consider n-player perfect information games with payoff functions having a finite image. We do not make any further assumptions, so in particular we refrain from making assumptions on the cardinality or the topology of the set of actions and assumptions like continuity or measurability of payoff functions. We show that there exists a best response cycle of length four, that is, a sequence of four pure strategy profiles where every successive element is a best response to the previous one. This result implies the existence of point-rationalizable strategy profiles. When payoffs are only required to be bounded, we show the existence of an ε-best response cycle of length four for every ε > 0.

The query complexity of correlated equilibria

We consider the complexity of finding a correlated equilibrium of an n-player game in a model that allows the algorithm to make queries on players' payoffs at pure strategy profiles. Randomized regret-based dynamics are known to yield an approximate correlated equilibrium efficiently, namely, in time that is polynomial in the number of players n. Here we show that both randomization and approximation are necessary: no efficient deterministic algorithm can reach even an approximate correlated equilibrium, and no efficient randomized algorithm can reach an exact correlated equilibrium. The results are obtained by bounding from below the number of payoff queries that are needed.

Pareto-optimal Nash equilibrium: Sufficient conditions and existence in mixed strategies

This paper considers the Nash equilibrium strategy profiles that are Pareto optimal with respect to the rest Nash equilibrium strategy profiles. The sufficient conditions for the existence of such pure strategy profiles are established. These conditions employ the Germeier convolutions of the payoff functions. For the non-cooperative games with compact strategy sets and continuous payoff functions, the existence of the Pareto optimal Nash equilibria in mixed strategies is proved.

Remarks on the existence and stability of some relaxed Nash equilibrium in strategic form games

We introduce a relaxed Nash equilibrium concept for strategic form games, called quasi-Nash equilibrium, which exists in every diagonally transfer quasi-concave games. The quasi-Nash equilibrium has the property to be stable with respect to perturbations on both payoff functions and mistakes in playing pure strategy profiles. We compare our relaxed equilibrium with the Reny equilibrium (introduced in a recent paper) and give stability properties for the Reny equilibrium.

Best Response Cycles in Perfect Information Games

We consider n–player perfect information games with payoff functions having a finite image. We do not make any further assumptions, so in particular we refrain from making assumptions on the cardinality or the topology of the set of actions and assumptions like continuity or measurability of payoff functions. We show that there exists a best response cycle of length four, that is, a sequence (σ0, σ1, σ2, σ3, σ0) of pure strategy profiles where every successive element is a best response to the previous one. This result implies the existence of point-rationalizable strategy profiles. When payoffs are only required to be bounded, we show the existence of an ϵ–best response cycle of length four for every ϵ > 0.

An Algorithm for Computing All Berge Equilibria

An algorithm is presented in this note for determining all Berge equilibria for an n-person game in normal form. This algorithm is based on the notion of disappointment, with the payoff matrix (PM) being transformed into a disappointment matrix (DM). The DM has the property that a pure strategy profile of the PM is a BE if and only if (0,…,0) is the corresponding entry of the DM. Furthermore, any (0,…,0) entry of the DM is also a more restrictive Berge-Vaisman equilibrium if and only if each player’s BE payoff is at least as large as the player’s maximin security level.

Nonconvergence to saddle boundary points under perturbed reinforcement learning

For several reinforcement learning models in strategic-form games, convergence to action profiles that are not Nash equilibria may occur with positive probability under certain conditions on the payoff function. In this paper, we explore how an alternative reinforcement learning model, where the strategy of each agent is perturbed by a strategy-dependent perturbation (or mutations) function, may exclude convergence to non-Nash pure strategy profiles. This approach extends prior analysis on reinforcement learning in games that addresses the issue of convergence to saddle boundary points. It further provides a framework under which the effect of mutations can be analyzed in the context of reinforcement learning.

Multiagent Reinforcement Learning With Unshared Value Functions

One important approach of multiagent reinforcement learning (MARL) is equilibrium-based MARL, which is a combination of reinforcement learning and game theory. Most existing algorithms involve computationally expensive calculation of mixed strategy equilibria and require agents to replicate the other agents' value functions for equilibrium computing in each state. This is unrealistic since agents may not be willing to share such information due to privacy or safety concerns. This paper aims to develop novel and efficient MARL algorithms without the need for agents to share value functions. First, we adopt pure strategy equilibrium solution concepts instead of mixed strategy equilibria given that a mixed strategy equilibrium is often computationally expensive. In this paper, three types of pure strategy profiles are utilized as equilibrium solution concepts: pure strategy Nash equilibrium, equilibrium-dominating strategy profile, and nonstrict equilibrium-dominating strategy profile. The latter two solution concepts are strategy profiles from which agents can gain higher payoffs than one or more pure strategy Nash equilibria. Theoretical analysis shows that these strategy profiles are symmetric meta equilibria. Second, we propose a multistep negotiation process for finding pure strategy equilibria since value functions are not shared among agents. By putting these together, we propose a novel MARL algorithm called negotiation-based Q-learning (NegoQ). Experiments are first conducted in grid-world games, which are widely used to evaluate MARL algorithms. In these games, NegoQ learns equilibrium policies and runs significantly faster than existing MARL algorithms (correlated Q-learning and Nash Q-learning). Surprisingly, we find that NegoQ also performs well in team Markov games such as pursuit games, as compared with team-task-oriented MARL algorithms (such as friend Q-learning and distributed Q-learning).

A Scalar Compromise Equilibrium for N-Person Prescriptive Games

A scalar equilibrium (SE) is defined for n-person prescriptive games in normal form. When a decision criterion (notion of rationality) is either agreed upon by the players or prescribed by an external arbiter, the resulting decision process is modeled by a suitable scalar transformation (utility function). Each n-tuple of von Neumann-Morgenstern utilities is transformed into a nonnegative scalar value between 0 and 1. Any n-tuple yielding a largest scalar value determines an SE, which is always a pure strategy profile. SEs can be computed much faster than Nash equilibria, for example; and the decision criterion need not be based on the players’ selfishness. To illustrate the SE, we define a compromise equilibrium, establish its Pareto optimality, and present examples comparing it to other solution concepts.

Improving Nash by coarse correlation

We consider a class of symmetric two-person quadratic games where coarse correlated equilibria – CCE – (Moulin and Vial [16]) can strictly improve upon the Nash equilibrium payoffs, while correlated equilibrium – CE – (Aumann [3,4]) cannot, because these games are potential games with concave potential functions. We compute the largest feasible total utility in any CCE in those games and show that it is achieved by a CCE involving only two pure strategy profiles. Applications include the Cournot duopoly and the game of public good provision, where the improvement over and above the Nash equilibrium payoff can be substantial.