Markov Perfect Equilibrium Research Articles

The battle of opinion: dynamic information revelation by ideological senders

Two informed and interested parties (senders) repeatedly send messages to an uninformed party (public). Senders face a trade-off between propagating their favoured opinions, possibly by lying, and maintaining a high audience (or market share), as the state is occasionally revealed and lies cause audiences to switch to the competitor. We fully characterize a focal Markov perfect equilibrium of this game and discuss the impact of exogenous parameters on the truthfulness of equilibrium reporting. In particular, we find that senders’ lying propensities are strategic complements so that increasing one sender’s bias decreases both senders’ truthfulness. We also analyse the role of polarization across senders.

On the complexity of computing Markov perfect equilibrium in general-sum stochastic games.

Similar to the role of Markov decision processes in reinforcement learning, Markov games (also called stochastic games) lay down the foundation for the study of multi-agent reinforcement learning and sequential agent interactions. We introduce approximate Markov perfect equilibrium as a solution to the computational problem of finite-state stochastic games repeated in the infinite horizon and prove its PPAD-completeness. This solution concept preserves the Markov perfect property and opens up the possibility for the success of multi-agent reinforcement learning algorithms on static two-player games to be extended to multi-agent dynamic games, expanding the reign of the PPAD-complete class.

Layered Networks, Equilibrium Dynamics, and Stable Coalitions

An important aspect of network dynamics that has been missing from our understanding of network dynamics in various applied settings is the influence of strategic behavior in determining equilibrium network dynamics. Our main objective hear to say what we can regarding the emergence of stable club networks—and therefore, stable coalition structures—based on the stability properties of strategically determined equilibrium network formation dynamics. Because club networks are layered networks, our work here can be thought of as a first work on the strategic dynamics of layered networks. In addition to constructing a discounted stochastic game model (i.e., a DSG model) of club network formation, (1) we show that our DSG of network formation possesses a stationary Markov perfect equilibrium in players’ membership-action strategies; (2) we identify the assumptions on primitives which ensure that the induced equilibrium Markov process of layered club network formation satisfies the Tweedie Stability Conditions (Tweedie in Stoch Process Appl 92:345–354, 2001) and (3) we show that, as a consequence, the equilibrium Markov network formation process generates a unique decomposition of the set of state-network pairs into a transient set together with finitely many basins of attraction. Moreover, we show that if there is a basin containing a vio set (a visited infinitely often set) of club networks sufficiently close together, then the coalition structures across club networks in the vio set will be the same (i.e., closeness across networks in a vio set leads to invariance in coalition structure across networks in a vio set).

The effect of access to clean technology on pollution reduction: An experiment

We use a laboratory experiment to study decisions in a dynamic game where firms' private production leads to accumulation of a public bad, such as pollution. Firms have an option to invest in clean technology, which proportionally lowers their emissions, or contributions to the public bad. The main treatment variable is the type of access to clean technology, or benefits from such investment, which can be private or common. Under private access, investment reduces the firm's own propensity to pollute. Under common access, each firm's investment reduces all firms' propensity to pollute. For each treatment, we characterize the Markov perfect equilibrium and social optimum. The observed level of the public bad is lower with common access. This result remains in the presence of communication. The option to communicate induces coordination of investments in clean technology at a higher level, leading to lower average pollution levels in both treatments.

Profit Effects of Consumers’ Identity Management: A Dynamic Model

We consider a nondurable good monopolist that collects data on its customers in order to profile them and subsequently practice price discrimination on returning customers. The monopolist’s price discrimination scheme is leaky in the sense that an endogenous fraction of consumers choose to incur a privacy cost to conceal their identity when they return in the following periods. We characterize the Markov perfect equilibrium of the game under two alternative customer profiling regimes: full information acquisition (FIA) and purchase history information (PHI). In both cases, we find that, contrary to what could be expected, the monopolist’s aggregate profit is not monotonically increasing in the level of the privacy cost, but a U-shaped function of it, leading to ambiguous profit effects: a reduction in privacy costs increases the fraction of customers who choose to be anonymous (detrimental profit effect), but it also softens the firm’s introductory price, reducing the pace at which prices targeted to new customers fall over time (positive profit effect). When comparing results under FIA and PHI, we find that market expansion is faster, and more customers conceal their identity under FIA than under PHI. Equilibrium profits are also higher in the FIA case. Although equilibrium profits are U-shaped functions of the privacy cost in both profiling regimes, they tend to be globally decreasing with the privacy cost under PHI and globally increasing under FIA. This paper was accepted by Eric Anderson, marketing. Funding: This work was supported by Fundação para a Ciência e a Tecnologia [Grants NORTE-01-0145-FEDER-028540 and POCI-01-0145-FEDER-006890]. Supplemental Material: The online appendix is available at https://doi.org/10.1287/mnsc.2022.4511 .

Strategic investments in distributed computing: A stochastic game perspective

We study a stochastic game with a dynamic set of players, for modeling and analyzing their computational investment strategies in distributed computing. Players obtain a certain reward for solving a problem, while incurring a certain cost based on the invested time and computational power. We present our framework while considering a contemporary application of blockchain mining, and show that the framework is applicable to certain other distributed computing settings as well. For an in-depth analysis, we consider a particular yet natural scenario where the rate of solving the problem is proportional to the total computational power invested by the players. We show that, in Markov perfect equilibrium, players with cost parameters exceeding a certain threshold, do not invest; while those with cost parameters less than this threshold, invest maximal power. We arrive at an interesting conclusion that the players need not have information about the system state as well as each others' parameters, namely, cost parameters and arrival/departure rates. With extensive simulations and insights through mean field approximation, we study the effects of players' arrival/departure rates and the system parameters on the players' utilities.

Social security as Markov equilibrium in OLG models: Clarifications and some new insights

This paper studies the politico-economic sustainability of pay-as-you-go social security in OLG models under Markovian strategies as first studied by Forni (2005). Under logarithmic utility, the paper shows that equilibria with social security can only exist if the underlying economy is dynamically inefficient. The paper also derives the exact parametric conditions that allow for the existence of equilibria and shows that among all the admissible (arbitrary) constants that produce a Markov perfect equilibrium, the maximum constant in such set yields the only equilibrium that solves dynamic inefficiency.

Equilibrium in a Symmetric Game of Resource Extraction with Coalitional Structure

The game of resource extraction / capital accumulation is a stochastic nonzero-sum infinite horizon game, obtained as an extension of the well-known optimal growth model to m strategically competing players, who jointly posses a renewable resource. The existence of a Nash equilibrium in different, often symmetric, frameworks of the game received a significant attention in the scientific literature on the topic. The focus of this paper is to introduce the coalitional component to the symmetric problem. Specifically, we examine whether the game with a fixed coalitional structure admits stability against profitable coalitional deviations.It is assumed that the set of all players is partitioned into coalitions which do not intersect and remain consistent throughout the game. The members of each coalition are able to coordinate their actions and perform joint deviations in a cooperative manner. Such setting incorporates a natural concept of established social ties, which may reflect a potential context appearing in practical applications. The corresponding notion of equilibrium in the paper is expressed as a position, from which none of the set coalitions can deviate in a manner to increase a total reward of its members. Its existence is studied in the context of a certain symmetric resource extraction game model with unbounded utilities of the players. This model was studied in [12; 13], concluding a Stationary Markov Perfect Equilibrium existence in both symmetric and non-symmetric game structure. The first feature of the model is that the preferences of the players are considered to be isoelastic in the form of strictly concave power functions. Furthermore, the law of motion between states is set to follow a geometric random walk in relation to players' joint investments. We prove that the game within the formulated settings admits stability against profitable coalitional deviations for any partition on the set of agents. The method provides an algorithm for building the corresponding stationary strategies, which can be useful for practical purposes. Finally, we use two examples with different numerical configurations to illustrate possible patterns of how the individual rewards of the players vary depending on a coalitional structure, which is set at the beginning of the game.

Coalitional bargaining games: A new concept of value and coalition formation

We propose a new solution for coalition bargaining problems among n players that can form coalitions c generating heterogenous coalitional values ▪. The players' values vi and probability of coalition formation ▪ are given by: [Display omitted] where coalition c is chosen only if it maximizes the average gain ▪ and ▪. This solution is the strong Markov perfect equilibrium of a non-cooperative coalition bargaining game where players choose simultaneously the coalition they want to join followed by negotiations to split the surplus. The solution does not rely on the specification of a proposer recognition protocol. For majority voting games, the solution exhibits more inequality among the values of large and small parties and a concentrated equilibrium coalition formation distribution.

To Fight or to Give Up? Dynamic Contests with a Deadline

We study dynamic contests between two players whose performance is determined jointly by effort and luck. The players observe each other’s positions in real time. There is a fixed deadline, and the player with a higher performance at the deadline wins the contest. We fully characterize the Markov perfect equilibrium for heterogeneous players. Effort is high when the players are tied but collapses quickly when one of them assumes a lead, due to a dynamic momentum effect. Therefore, total expected effort does not necessarily increase in the prize or in the players’ abilities. We discuss implications for contest design and propose splitting the contest to cool off competition and introducing optimal head-starts for heterogeneous players as possible solutions. This paper was accepted by Ilia Tsetlin, behavioral economics and decision analysis.

Uncertainty-driven symmetry-breaking and stochastic stability in a generic differential game of lobbying

We study a 2-player stochastic differential game of lobbying. Players invest in lobbying activities to alter the legislation in her own benefit. The payoffs are quadratic and uncertainty is driven by a Wiener process. We consider the Nash symmetric game where players face the same cost and extract symmetric payoffs, and we solve for Markov Perfect Equilibria (MPE) in the class of affine functions. First, we prove a general sufficient (catching up) optimality condition for two-player stochastic games with uncertainty driven by Wiener processes. Second, we prove that the number and nature of MPE depend on the extent of uncertainty (i.e. the variance of the Wiener processes). In particular, we prove that while a symmetric MPE always exists, two asymmetric MPE emerge if and only if uncertainty is large enough. Third, we study the stochastic stability of all the equilibria. We notably find, that the state converges to a stationary invariant distribution under asymmetric MPE. Fourth, we study the implications for rent dissipation asymptotically and compare the outcomes of symmetric vs asymmetric MPE in this respect, ultimately enhancing again the role of uncertainty.

On a special case of non-symmetric resource extraction games with unbounded payoffs

The game of resource extraction/capital accumulation is a stochastic infinite-horizon game, which models a joint utilization of a productive asset over time. The paper complements the available results on pure Markov perfect equilibrium existence in the non-symmetric game setting with an arbitrary number of agents. Moreover, we allow that the players have unbounded utilities and relax the assumption that the stochastic kernels of the transition probability must depend only on the amount of resource before consumption. This class of the game has not been examined beforehand. However, we could prove the Markov perfect equilibrium existence only in the specific case of interest. Namely, when the players have constant relative risk aversion (CRRA) power utilities and the transition law follows a geometric random walk in relation to the joint investment. The setup with the chosen characteristics is motivated by economic considerations, which makes it relevant to a certain range of real-word problems.

Equilibrium Defaultable Corporate Debt and Investment

In dynamic capital structure models with an investor break-even condition, the firm's Bellman equation may not generate a contraction mapping, so the standard existence and uniqueness conditions do not apply. First, we provide an example showing the problem in a classical trade-off model. The firm can issue one-period defaultable debt, invest in capital and pay a dividend. If the firm cannot meet the required debt payment, it is liquidated. Second, we show how to use a dual to the original problem and a change of measure, such that existence and uniqueness can be proved. In the unique Markov-perfect equilibrium, firm decisions reflect state-dependent capital and debt targets. Our approach may be useful for other dynamic firm models that have an investor break-even condition.

Attaining herd immunity to a new infectious disease through multi-stage policies incentivising voluntary vaccination

Vaccine hesitancy, as well as propensity to free-ride, can pose serious challenges to mass vaccination schemes aimed at controlling spread of infectious diseases. If vaccination is not mandatory, the individuals decide whether to vaccinate, or not, comparing their expected utilities, net of costs, from the respective options. In our proposed model, to encourage and incentivise vaccination, the government launches a 4-stage policy implementation plan, involving awareness building, vaccine distribution intensification, subsidising the vaccine, and offering free-vaccination along with a reward for getting vaccinated. While the government distributes and subsidises one vaccine, it decides whether to approve private distribution of another vaccine. Modelling the scenario as a population game, wherein the individuals may want to free-ride by avoiding vaccine while others get vaccinated, we determine the vaccine coverage in the Markov perfect equilibrium (MPE) of the game at each stage. Through a detailed numerical simulation, we observe the effects of exogenous variables, policy variables and efficiency parameters, on coverage. Our results indicate that the government should approve the other vaccine and focus on intensifying vaccine distribution. The impact of awareness development is limited. If the partial subsidy in the penultimate stage is higher, a smaller fraction of the population needs to be covered under the rewarded free-vaccination scheme. However, the high partial subsidy in the penultimate stage may come at an additional cost burden on the government, and hence a trade-off may be necessary.

Markov Perfect Equilibria in Multi-Mode Differential Games with Endogenous Timing of Mode Transitions

We study Markov perfect equilibria (MPE) of two-player multi-mode differential games with controlled state dynamics, where one player controls the transition between modes. Different types of MPE are characterized distinguishing between delay equilibria, inducing for some initial conditions mode switches after a positive finite delay, and now or never equilibria, under which, depending on the initial condition, a mode switch occurs immediately or never. These results are applied to analyze the MPE of a game capturing the dynamic interaction between two incumbent firms among which one has to decide when to extend its product range by introducing a new product. The market appeal of the new product can be influenced over time by both firms through costly investments. Under a wide range of market introduction costs a now or never equilibrium coexists with a continuum of delay equilibria, each of them inducing a different time of product introduction.

Overcoming Free-Riding in Bandit Games

AbstractThis article considers a class of experimentation games with Lévy bandits encompassing those of Bolton and Harris (1999, Econometrica, 67, 349–374) and Keller, Rady, and Cripps (2005, Econometrica, 73, 39–68). Its main result is that efficient (perfect Bayesian) equilibria exist whenever players’ payoffs have a diffusion component. Hence, the trade-offs emphasized in the literature do not rely on the intrinsic nature of bandit models but on the commonly adopted solution concept (Markov perfect equilibrium). This is not an artefact of continuous time: we prove that efficient equilibria arise as limits of equilibria in the discrete-time game. Furthermore, it suffices to relax the solution concept to strongly symmetric equilibrium.

Computing Equilibria with Two-Player Zero-Sum Continuous Stochastic Games with Switching Controller

Equilibrium computation with continuous games is currently a challenging open task in artificial intelligence. In this paper, we design an iterative algorithm that finds an ε-approximate Markov perfect equilibrium with two-player zero-sum continuous stochastic games with switching controller. When the game is polynomial (i.e., utility and state transitions are polynomial functions), our algorithm converges to ε = 0 by exploiting semidefinite programming. When the game is not polynomial, the algorithm exploits polynomial approximations and converges to an ε value whose upper bound is a function of the maximum approximation error with infinity norm. To our knowledge, this is the first algorithm for equilibrium approximation with arbitrary utility and transition functions providing theoretical guarantees. The algorithm is also empirically evaluated.

Reducible Markov Decision Processes and Stochastic Games

Markov decision processes (MDPs) provide a powerful framework for analyzing dynamic decision making. However, their applications are significantly hindered by the difficulty of obtaining solutions. In this study, we introduce reducible MDPs whose exact solutions can be obtained by solving simpler MDPs, termed the coordinate MDPs. The value function and an optimal policy of a reducible MDP are linear functions of those of the associated coordinate MDP. Because the coordinate MDP does not involve the multi‐dimensional endogenous state, we achieve dimension reduction on a reducible MDP. Extending the MDP framework to multiple players, we introduce reducible stochastic games. We show that these games reduce to simpler coordinate games that do not involve the multi‐dimensional endogenous state. We specify sufficient conditions for the existence of a pure‐strategy Markov perfect equilibrium in reducible stochastic games and derive closed‐form expressions for the players’ equilibrium values. The reducible framework encompasses a variety of linear and nonlinear models and offers substantial simplification in analysis and computation. We provide guidelines and illustrative examples on formulating problems as reducible models. We demonstrate the applicability and modeling flexibility of reducible models in a wide range of contexts including capacity and inventory management and duopoly competition.

A noncooperative foundation of the competitive divisions for bads

Many economic situations involve the division of bads. We study a noncooperative game model for this type of division problem. The game resembles a standard multilateral bargaining model, but in our case, perpetual disagreement is not a feasible outcome. The driving feature of the model is that a player that makes an unacceptable proposal (causing breakdown with some probability) is made to internalize all the costs in case of breakdown. We show that as the probability of exogenous breakdown goes to zero, this game implements some competitive divisions in Markov perfect equilibria: the limit of any convergent sequence of equilibrium outcomes is a competitive division, but a competitive division may not be a limit of the equilibrium outcomes.

Mean Field Equilibrium: Uniqueness, Existence, and Comparative Statics

The Uniqueness of a Mean Field Equilibrium