Mixed-strategy Equilibrium Research Articles

Credible Negotiation for Multi-agent Reinforcement Learning in Long-term Coordination

The coordination of multi-agent is one of the critical problems in Multi-Agent Reinforcement Learning (MARL). The traditional methods of MARL focus on finding a stochastically acceptable solution called Nash Equilibrium (NE) for all agents from the Markov game in which multiple equilibria exist. However, learning a fair equilibrium is crucial for the sustainability and stability of collaboration in the long-term coordination game, especially when the leadership competition exists. In this paper, we propose the bi-level reinforcement learning method N-Bi-AC, whose solution is a Pareto improvement for traditional NE, to choose a fair Equilibrium. There are two parts in our method, the first is that we propose the Negotiator to determine the leader in stage game, and the other is to update the Q-value of agents in the game by using a bi-level actor-critic learning method based on the Joint Mixed Strategy Equilibrium Q-learning algorithm (JMSE Q-learning). The convergence proof is given, and the learning algorithm is compared with the state-of-the-art algorithms. We found that the proposed N-Bi-AC method successfully converged to a fair Nash Equilibrium, which guarantees the fairness of agents in different matrix game environments.

The Future of the CFA Franc: A Game Theory Approach

Aims: Based on the theory of non-cooperative games, this paper aims to determine whether the African Franc Zone (AFZ) countries should leave or remain in monetary cooperation with France. Methodology: Empirically, we implement a non-cooperative game pitting France against the AFZ countries. The strategies defined for the two players are "Maintain (M)" and "Exit (E)." The potential benefit is the preservation of the pooling of 50% of the AFZ countries' foreign exchange reserves in the French Treasury. Results: Using Nash equilibrium, the study analyzes strategic decisions in monetary cooperation. At the end of technical manipulations, we obtain two results based on two assumptions: (i) Assuming that the AFZ is the first player, and the decision being taken by political leaders without consulting the other sociological components, the optimal solution would be (M, M), reflecting the preservation of 50% of foreign exchange reserves within the French Treasury; (ii) Then, when the leaders of the AFZ countries incorporate the will of the people and the aspirations of the African Union, we arrive at two Nash equilibria, (E, E) and (E, M) respectively, giving gains of 50% for the AFZ and 0% for the French Treasury, significantly implying an exit decision for AFZ countries. These results show that, beyond the positions of technocrats and leaders, the people's will is a determining factor in the economic and monetary conditions of a country or group of countries. Conclusion: However, these equilibria could be sensitive to the order of the game (who plays first), political shocks such as coups d'état, types of threat, grouped or solitary exit strategies, but also to the bargaining power of the players, mixed-strategy equilibrium and external geopolitical influences. The countries of the AFZ need to strengthen their monetary sovereignty by reforming cooperation mechanisms with France, while also incorporating the aspirations of their populations into the decision-making process. This will promote a more legitimate and contextually relevant monetary policy that aligns with current economic and social realities.

Think twice before attacking: Effort, restraint, and sanctions in war conflicts

There are contrasting empirical findings about how third parties mitigate conflict and few theoretical results about war efforts and restraint. Applying a new approach that distinguishes between quantity and effectiveness of effort, we explain countries’ restraint incentives. In our setting, restraint is manifested in reducing the effectiveness of military efforts when third parties impose costs (sanctions) on unrestrained behavior. We show that intermediate sanctions may lead to a mixed-strategy equilibrium or pure strategy equilibria in which only one country reduces its effectiveness. We present the conditions that yield pure-strategy (in which, surprisingly, restraint is a corner solution) and mixed-strategy (MS) equilibria in our multi-stage model. Interestingly, a country's effort decreases with its or its rival's self-restraint, allowing a third party to reduce efforts by sanctioning only one country. Our analysis enriches the classical results, particularly those obtained in the context of war conflicts, which focus on the extent of effort expended and on means of mitigating the conflict. The theoretical study lays the groundwork for future research into war contests and suggests various avenues for further exploration.

An analysis of the equilibrium strategies for route-choosing customers in a two-station queueing system

We investigate a two-station queueing system where strategic customers must be sequentially serviced at both stations. We prove that an established property — that the distribution of the total time spent in a two-station system is independent of the chosen route when services times are exponentially distributed — is not a general one by providing a counterexample with deterministic service times. In doing so, we also prove a concomitant property — that any routing strategy is an equilibrium — is peculiar to a system with a particular assumption of an exponential service time distribution. Using simulations, we show that — depending on the distribution of service times — there can be (1) cases with three equilibria, (2) cases with one pure strategy equilibrium, and (3) cases with one mixed strategy equilibrium.

Cournot Policy Model: Rethinking centralized training in multi-agent reinforcement learning

This work studies Centralized Training and Decentralized Execution (CTDE), which is a powerful mechanism to ease multi-agent reinforcement learning. Although the centralized evaluation ensures unbiased estimates of Q-value, peers with unknown policies make the decentralized policy far from the expectation. To make progress in more stabilized and effective joint policy, we develop a novel game framework, termed Cournot Policy Model, to enhance the CTDE-based multi-agent learning. Combining the game theory and reinforcement learning, we regard the joint decision-making in a single time step as a Cournot duopoly model, and then design a Hetero Variational Auto-Encoder to model the policies of peers in the decentralized execution. With a conditional policy, each agent is guided to a stable mixed-strategy equilibrium even though the joint policy evolves over time. We further demonstrate that such an equilibrium must exist in the case of centralized evaluation. We investigate the improvement of our method on existing centralized learning methods. The experimental results on a comprehensive collection of benchmarks indicate our approach consistently outperforms baseline methods.

Equilibrium balking strategies in unobservable queues with multiple vacations and an optional service

This paper examines equilibrium mixed strategies in unobservable Markovian queues featuring a second optional service with server vacations, where arriving customers may choose to join or balk the system. All customers arriving at the system receive the essential service, and some customers opt for the second service after the first service has been completed. Once all customers in the system have been served, the server takes the first of multiple vacations. If no customers are waiting upon from the vacation, then the server takes another vacation. In unobservable queues, arriving customers cannot know the queue length; however, the information pertaining to the server state may be available. By examining unobservable queues (fully unobservable and almost unobservable cases), it is possible to formulate an equilibrium joining strategy as well as the socially optimal probability of joining a fully unobservable queue. This paper also presents numerical examples illustrating how system parameters affect mixed equilibrium and socially optimal balking strategies.

Efficient equilibria in common interest voting games

We develop a unified derivation of asymmetric pure strategy equilibria and their optimality in the canonical common interest voting model of Austen-Smith and Banks (Am Polit Sci Rev 90(1):34–45, 1996). We also study the relationship between the most efficient equilibria, which have a remarkably simple and intuitive structure, and the symmetric mixed strategy equilibrium that has been commonly studied in the literature. In particular, while the efficiency in the symmetric mixed strategy equilibrium under unanimity rule is known to be decreasing in the number of voters, the efficiency does not depend on the number of voters above a threshold in the most efficient equilibria.

Game theory, compliance, and corporate criminal liability: Insights from a three-player inspection game

The economic analysis of crime has contributed significantly to our understanding of corporate criminal activity. To derive implications for legal policy, most theoretical papers, however, take public enforcement as exogenous, i.e. non-compliance with laws is punished with a given probability by a mechanical enforcer. We apply a game-theoretic perspective to corporate criminal behavior, self-disclosure, and policing. In this paper, we model corporate compliance as a three-person inspection game in order to analyze the impact of different liability regimes on rule breaking behavior by the firm’s agents, monitoring effort by corporate management and enforcement effort by authorities. Our analysis shows that one out of two equilibria in mixed strategies can be desirable for policy makers: while the first equilibrium requires a delicate regime of composite liability to induce self-disclosure, the second outcome is less prone to specification errors and does not rely on self-disclosure by firms. Which equilibrium is socially more desirable regarding social costs depends largely on the relative difference in monitoring costs between the corporation and authorities. We also find that a composite liability regime is more vulnerable to the reallocation of punishments by the agents through contracts or civil litigation. Moreover, we also consider the effects of whistleblower protection programs and a negligence-based liability regime to induce efforts to prevent crime ex-ante.

Endogenous price discrimination with asymmetric firms

We explore a third-degree spatial price discrimination model within the context of asymmetric duopolies. We find that if the degree of asymmetry is small, both firms charge uniform price is the unique sub-game perfect Nash equilibrium. As the degree of asymmetry reaches intermediate, there exists no pure strategy sub-game perfect Nash equilibrium. In the mixed strategy equilibrium, both firms price discriminate with certain probability. When the degree of asymmetry is large, there exists one or two asymmetric equilibria: the advantaged firm price discriminating and the disadvantaged firm uniformly pricing IN and vice versa in the other NI. In both equilibria, the price-discriminating firm consistently benefits, and the uniform-pricing firm also experiences higher profits when the degree of asymmetry is large, despite being driven out of the smaller market. Moreover, when the asymmetry is large enough, social welfare can also be higher.

The Marriage Game Revisited: New Insights About Domestic Labor, Divorces and Men’s Expectations About Marriage

In the Marriage Game, women decide whether to get married and afterward men decide whether to cooperate in the domestic tasks. Then, if a husband is not cooperative, a wife can divorce or remain married. The sequence of decisions can be formalized as a game in extensive form, in which the strategic interaction between partners turns out evident. A husband may deceive, hoping that the divorce threat is not realistic, and a woman may decide not to get married from the beginning. If players had perfect knowledge about partners’ attitude, then divorce would not exist. Unfortunately, partners’ type is revealed only after the wedding. So, it is assumed that the game is played by different type of players, both men and women, and they interact with incomplete information. The model is solved by calculating all the perfect Bayes–Nash equilibria and it is shown that they depend on a complex interaction between model parameters, as multiple or mixed strategy equilibria can be found same parameters configuration. Comparing equilibria, we argue that the model can describe the marriage evolution that has been experienced by contemporary society. Following the increasing women’s participation in the job market, the number of marriages started to decline, but surprisingly, we are observing its resurgence in the most advanced societies. Our solution to the Marriage Game points to both the active role of autonomous women and the men’s expectations about marriage. As the number of autonomous women increases, men realize that divorce is credible and they change their expectations on women.

Mixed-Strategy Equilibria and Gender Differences:The Soccer Penalty Kick Game

Mixed-Strategy Equilibria and Gender Differences:The Soccer Penalty Kick Game

Pure strategy Nash equilibria for bargaining models of collective choice

This paper considers pure strategy Nash equilibria of non-cooperative legislative bargaining models. In contrast to existing legislative bargaining models, we derive legislators behavior from stochastic utility maximization. This approach allows us to prove the existence of a stationary Pure Local and Global Nash Equilibrium under rather general settings. The mathematical proof is based on a fixed point argument, which can also be used as a numerical method to determine an equilibrium. We characterize the equilibrium outcome as a lottery of legislators’ proposals and prove a Mean Voter Theorem, i.e., proposals result dimension-by-dimension as a weighted mean of legislators’ ideal points and are Pareto-optimal. Based on a simple example, we illustrate different logic of our model compared to mixed strategy equilibrium of the legislative bargaining model suggested by Banks and Duggan (Am Polit Sci Rev 94(1):73–88. https://doi.org/10.2307/2586381, 2000).

Hide-and-Seek Game with Capacitated Locations and Imperfect Detection

We consider a variant of the hide-and-seek game in which a seeker inspects multiple hiding locations to find multiple items hidden by a hider. Each hiding location has a maximum hiding capacity and a probability of detecting its hidden items when an inspection by the seeker takes place. The objective of the seeker (respectively, hider) is to minimize (respectively, maximize) the expected number of undetected items. This model is motivated by strategic inspection problems, where a security agency is tasked with coordinating multiple inspection resources to detect and seize illegal commodities hidden by a criminal organization. To solve this large-scale zero-sum game, we leverage its structure and show that its mixed-strategy Nash equilibria can be characterized using their unidimensional marginal distributions, which are pure equilibria of a lower dimensional continuous zero-sum game. This leads to a two-step approach for efficiently solving our hide-and-seek game: First, we analytically solve the continuous game and derive closed-form expressions of the equilibrium marginal distributions. Second, we design a combinatorial algorithm to coordinate the players’ resources and compute equilibrium mixed strategies that satisfy the marginal distributions. We show that this solution approach computes a Nash equilibrium of the hide-and-seek game in quadratic time with linear support. Our analysis reveals novel equilibrium behaviors driven by a complex interplay between the game parameters, captured by our closed-form solutions. Funding: This work was supported by the Georgia Tech Stewart Fellowship and the Georgia Tech New Faculty Start Up Grant. Supplemental Material: The online appendix is available at https://doi.org/10.1287/deca.2023.0012 .

Mixed-strategy equilibrium of the symmetric production in advance game: The missing case

The mixed-strategy equilibrium of the symmetric production-in-advance type capacity-constrained Bertrand-Edgeworth duopoly game has not been derived analytically over the entire range of intermediate capacities in the literature. In this paper we derive for the missing region a symmetric mixed-strategy equilibrium analytically.

Revisiting the Asymmetric Matching Pennies Contradiction in China.

The asymmetric matching pennies contradiction posits that contrary to the prediction of mixed-strategy Nash equilibrium, experimental subjects' choices are, in practice, based heavily on the magnitudes of their own payoffs. Own-payoff effects are robustly confirmed in the literature. Closely following the experimental setups in the literature which support the contradiction, we conduct a series of asymmetric matching pennies games in China, hypothesizing play which is closer to equilibrium frequencies than previously found. Contrary to previous experiments which were conducted in the United States, we find that there are essentially no own-payoff effects among Row players who face large payoff asymmetry. In a Quantal Response Equilibrium framework allowing for altruism or spite, the behavior of our subjects corresponded to a positive spite parameter, whereas the results of previous studies corresponded to altruism. Our results may be consistent with recent psychology literature that finds people from collectivist cultures are substantially more adept at taking the perspective of others compared with people from individualist cultures, a feature of the reasoning needed to obtain mixed-strategy equilibrium.

Private provision of price excludable public goods by rivals

We uniquely study the private provision of price excludable public goods in a duopoly. Simultaneous price competition generates only a mixed strategy equilibrium. Price leadership generates a pure strategy equilibrium with the leader setting a lower price and serving most consumers. This leadership game is the endogenous timing choice and improves welfare relative to monopoly provision. We re-examine these results under production in advance. The leadership game no longer remains a unique timing choice but the profit under production in advance is strictly larger.

Information sharing decisions in all-pay auctions with correlated types

In many real-life competitions, contestants may not be aware of their own type (e.g., value or ability) prior to the contest. Furthermore, contestants’ types, which are observed privately after entering the competition, are frequently correlated with one another. We examine a two-stage competition that involves two players with correlated (binary) types. In the first stage, players decide simultaneously or sequentially on the probabilities they use to disclose or conceal their type, which will become their private information later on. In the second stage, each player privately observes their own type and commits to disclosing or concealing it, after which they compete in an all-pay auction. We discover that information sharing does not occur when players’ types are negatively correlated. However, when players’ types are positively correlated, information is partially shared in all equilibria examined in this study. In an asymmetric pure strategy equilibrium, one player shares his information with probability one and the other player with probability zero. In a symmetric mixed strategy equilibrium, each player shares his information with the same positive probability.

Replicator dynamics of evolutionary games with different delays on costs and benefits

The replicator dynamics of two-player evolutionary games with delays are examined. In contrast to situations with a common delay, we examine the resulting stability for the situation where the delays on the cost contributions to the payoff may be different from the delays on the benefit contributions to the payoff. In such scenarios we show that increasing one delay, while holding the other constant, can lead to conditions where the increasing delay causes instability, yet further increases in the delay can bring back stability, while still further increases in the delay can lead to instability again, and so on. This is in contrast to the case where costs and benefits share a common delay, in which case we show that only a simple destabilization can occur with an increasing delay. We develop a Hopf bifurcation analysis of this phenomenon for a general version of evolutionary games that have a stable mixed strategy equilibrium in the nominal (no delay) case. From the Hopf bifurcation analysis we show conditions under which the stabilizing and destabilizing phenomena occur. We then illustrate the results by applying them to specific examples of both a snowdrift game and a division of labor game. Numerical examples are presented to validate the analysis claims.

Optimal strategy analysis of a Markovian queue with variable vacation and vacation interruptions under unobservable cases

ABSTRACT This paper makes a game-theoretic analysis of an M/M/1 queue with variable vacation and vacation interruptions. The Nash equilibrium mixed strategy and social utility maximization are derived based on a non-cooperative game theory and an optimization theory under different information precision levels, namely almost unobservable and fully unobservable cases. The explicit solutions of the entrance probabilities and the steady-state probability distribution of the system are investigated using the probability generating function and nonhomogeneous linear difference equations. Furthermore, the effects of the information levels and diverse system parameters on the equilibrium strategies, the arrival rates, and the expected benefits are explicitly illustrated by numerical comparisons. The research results can provide a theoretical basis and performance analysis tool for the optimal design in the wireless transmission and network communication system.

Differential N-players game: Nash equilibria and Mather measures

Abstract We study Nash equilibria for the deterministic ergodic N-players game. We introduce pure strategies, mixed strategies and Nash equilibria associated with those. We show that a Nash equilibrium in mixed strategies exists and it is a Mather measure for the Lagrangian system defined by the cost functional. In conclusion, we show that the mean field limit of the N-players game is described by the ergodic partial differential equation’s system for a continuum of players.