Markov Perfect Nash Equilibrium Research Articles

Data Poisoning to Fake a Nash Equilibria for Markov Games

We characterize offline data poisoning attacks on Multi-Agent Reinforcement Learning (MARL), where an attacker may change a data set in an attempt to install a (potentially fictitious) unique Markov-perfect Nash equilibrium for a two-player zero-sum Markov game. We propose the unique Nash set, namely the set of games, specified by their Q functions, with a specific joint policy being the unique Nash equilibrium. The unique Nash set is central to poisoning attacks because the attack is successful if and only if data poisoning pushes all plausible games inside it. The unique Nash set generalizes the reward polytope commonly used in inverse reinforcement learning to MARL. For zero-sum Markov games, both the inverse Nash set and the set of plausible games induced by data are polytopes in the Q function space. We exhibit a linear program to efficiently compute the optimal poisoning attack. Our work sheds light on the structure of data poisoning attacks on offline MARL, a necessary step before one can design more robust MARL algorithms.

Mixed Market Structure and R &D: A Differential Game Approach

We consider a dynamic model of an industry consisting of a few large firms, which can manipulate the market outcome, and a mass of small enterprises, each of which has a negligible impact on the market. The production costs of the respective firms depend on the stock of knowledge capital, which accumulates over time through research and development (R &D) investment made by large firms. The model is a variant of the differential game of voluntary provision of public goods, but in contrast to previous studies, we focus on the interaction between market competition and dynamic game outcomes. We derive both open-loop and Markov-perfect Nash equilibria. There exists a unique open-loop Nash equilibrium. By contrast, depending on the parameters of the model, there can be two linear Markov-perfect Nash equilibria. We also examine the short- and long-run effects of a change in the number of large firms. An increase in the number of large firms unambiguously harms both types of firms in the short run but may benefit them in the long run. In the open-loop Nash equilibrium, the relationship between the number of large firms and the steady-state stock of knowledge capital is inverted-U shaped. Concerning the Markov-perfect Nash equilibria, the effect of increased competition from large firms depends on the specific feedback strategy chosen in equilibrium.

On the dynamic optimality of yardstick regulation

This paper proposes a generalization of Shleifer’s (RAND J Econ 16:319–327, 1985) model of yardstick competition to a dynamic framework. In a differential game setting, we show that the yardstick mechanism effectively replicates the first-best solution if players adopt open-loop behaviour rules and are symmetric at the initial time; in the absence of initial symmetry, the social efficiency is reached only in the asymptotic steady state. On the contrary, if players adopt Markovian behaviour rules, then the yardstick pricing rule cannot achieve the first-best solution along the equilibrium path of any Markov Perfect Nash Equilibrium.

A time consistent dynamic bargaining procedure in differential games with hterogeneous discounting

We study cooperative solutions for differential games where players consume a common property resource. Players are asymmetric, in the sense that they have different preferences and, in particular, different time preferences. We propose a new time-consistent dynamic bargaining procedure for this class of games. This solution concept, which is defined as the time-consistent dynamic bargaining (TCB) solution, extends the recursive Nash bargaining solution introduced in Sorger (J Econ Dyn Control 30:2637–2659, 2006) to a continuous time setting. The underlying idea is that, in case of disagreement, the threat is that players will play a noncooperative Markov Perfect Nash equilibrium just during a very small period of time, since new negotiations can take place at every future moment and, in particular, immediately later. Conditions for interior TCB solutions are derived. To illustrate the results, two common property resource games are analyzed in detail.

Strategic pollution control under free trade

This paper designs a reciprocal dumping model to address the control of industrial pollution between two trading partners. Firms generate transboundary pollution from production and environmental taxes represent the pollution control instrument. We ask whether environmental taxes implemented in a non-cooperative setting are more stringent than the globally efficient level. Relative to the globally efficient case, we find in the linear Markov Perfect Nash Equilibrium (MPNE) context that the tax rate for both countries is smaller and individual emissions are larger. However, these results may not hold in the non-linear MPNE case depending on market structure and environmental conditions. Unlike the symmetric equilibrium case, the tax rates are always discontinuous under asymmetric MPNEs. The asymmetric equilibrium scenario can give rise to higher individual payoffs relative to the symmetric equilibrium case.

Dynamic Games in Empirical Industrial Organization

This survey is organized around three main topics: models, econometrics, and empirical applications. Section 2 presents the theoretical framework, introduces the concept of Markov Perfect Nash Equilibrium, discusses existence and multiplicity, and describes the representation of this equilibrium in terms of conditional choice probabilities. We also discuss extensions of the basic framework, including models in continuous time, the concepts of oblivious equilibrium and experience-based equilibrium, and dynamic games where firms have non-equilibrium beliefs. In section 3, we first provide an overview of the types of data used in this literature, before turning to a discussion of identification issues and results, and estimation methods. We review different methods to deal with multiple equilibria and large state spaces. We also describe recent developments for estimating games in continuous time and incorporating serially correlated unobservables, and discuss the use of machine learning methods to solving and estimating dynamic games. Section 4 discusses empirical applications of dynamic games in IO. We start describing the first empirical applications in this literature during the early 2000s. Then, we review recent applications dealing with innovation, antitrust and mergers, dynamic pricing, regulation, product repositioning, advertising, uncertainty and investment, airline network competition, dynamic matching, and natural resources. We conclude with our view of the progress made in this literature and the remaining challenges.

Dynamic Games in Empirical Industrial Organization

This survey is organized around three main topics: models, econometrics, and empirical applications. Section 2 presents the theoretical framework, introduces the concept of Markov Perfect Nash Equilibrium, discusses existence and multiplicity, and describes the representation of this equilibrium in terms of conditional choice probabilities. We also discuss extensions of the basic framework, including models in continuous time, the concepts of oblivious equilibrium and experience-based equilibrium, and dynamic games where firms have non-equilibrium beliefs. In section 3, we first provide an overview of the types of data used in this literature, before turning to a discussion of identification issues and results, and estimation methods. We review different methods to deal with multiple equilibria and large state spaces. We also describe recent developments for estimating games in continuous time and incorporating serially correlated unobservables, and discuss the use of machine learning methods to solving and estimating dynamic games. Section 4 discusses empirical applications of dynamic games in IO. We start describing the first empirical applications in this literature during the early 2000s. Then, we review recent applications dealing with innovation, antitrust and mergers, dynamic pricing, regulation, product repositioning, advertising, uncertainty and investment, airline network competition, dynamic matching, and natural resources. We conclude with our view of the progress made in this literature and the remaining challenges.

All Symmetric Equilibria in Differential Games With Public Goods

We characterise the entire set of symmetric stationary Markov-perfect Nash equilibria (MPE) in a differential game of public good investment, using the canonical problem of climate change as an example. We provide a sufficient and necessary condition for MPE and show how the entire set of MPE is constructed. The equilibrium in continuous strategies, unique in our context, is Pareto-dominated by any other equilibrium. If a Pareto- undominated steady state exists, it is sustained by trigger-like strategies, with deviations above and below the steady state leading to different re- sponses. We extend the theory of differential games to deal with payoffs under discontinuous strategies. Our methods work under general functional forms.

Mali Merkezîleşme ile Yeniden Dağıtım Kuralı altında Mali Yerelleşmenin Karşılaştırılması

This paper compares the case of fiscal decentralization (FD) with an intergovernmental transfer rule to the case of fiscal centralization (FC) from a theoretical perspective while focusing on Markov-perfect Nash equilibrium by a continuum of citizens, local governments and a central government, which interact strategically. Simulation analysis shows both the degree of spillovers and capital mobility play a role in the comparison of these two cases. In the presence of spillovers, the welfare of FD case is higher than the one of FC which is an unexpected result but points out the positive effect of a redistribution rule in FD model in terms of welfare. On the other hand, the growth rate of FD is lower than FC case when there are spillovers. So, fiscal discipline, provided by the redistribution rule, prevents inefficiently low tax rates which pull down the growth rate. In addition, when spillovers are not allowed, capital mobility determines which case is superior.

All Symmetric Equilibria in Differential Games with Public Goods

We characterise the entire set of symmetric stationary Markov-perfect Nash equilibria (MPE) in a differential game of public good investment, using the canonical problem of climate change as an example. We provide a sufficient and necessary condition for MPE and show how the entire set of MPE is constructed. The equilibrium in continuous strategies, unique in our context, is Pareto-dominated by any other equilibrium. If a Pareto-undominated steady state exists, it is sustained by trigger-like strategies, with deviations above and below the steady state leading to different responses. We extend the theory of differential games to deal with payoffs under discontinuous strategies. Our methods work under general functional forms.

Mixed duopoly: Differential game approach

This study formulates a dynamic mixed oligopoly model, in which a state‐owned public firm competes against a private firm over multiple periods. We adopt a differential game formulated by Fershtman and Kamien [Econometrica 55 (1987), pp. 1151–1164] and investigate how the dynamic competition affects the optimal privatization policy. We characterize the open‐loop Nash equilibrium (OLNE) and Markov‐perfect Nash equilibrium (MPNE). We show that in the MPNE, an increase in the degree of privatization has a nonmonotonic effect on the price, increasing it in a wide range of parameter spaces, which is in sharp contrast to the result in the OLNE or static analyses. We also find that the optimal degree of privatization is higher in the MPNE than that in the OLNE and static equilibrium. These results suggest that intertemporal strategic behavior changes the optimal privatization policy.

Markov perfect equilibria in OLG models with risk sensitive agents

In this paper, we present an overlapping generation model (OLG for short) of resource extraction with a random production function and an altruism having both paternalistic and non-paternalistic features. All generations are risk-sensitive with a constant coefficient of absolute risk aversion. The preferences are represented by a possibly dynamic inconsistent dynamic recursive utility function with non-cooperating generations. Under general conditions on the aggregator and transition probability, we examine the existence and the uniqueness of a recursive utility function and the existence of a stationary mixed Markov Perfect Nash Equilibria.

Noncooperative Dynamic Contribution to a Public Project

We consider a dynamic contribution game in which a group of agents collaborates to complete a public project. The agents exert efforts over time and get rewarded upon completion of the project, once the cumulative effort has reached a pre-specified level. We explicitly derive the cooperative solution and a noncooperative Markov-perfect Nash equilibrium. We characterize the set of socially efficient projects, i.e., projects that cooperative groups find worth completing. Comparing with the Markov-perfect Nash equilibrium, we find that noncooperative groups give up large socially efficient projects. Moreover, they take too much time to complete the projects that they undertake.

Transboundary pollution control and environmental absorption efficiency management

In this paper, we suggest a two-player differential game model of transboundary pollution that accounts for time-dependent environmental absorption efficiency, which allows the biosphere to switch from a carbon sink to a source. We investigate the impact of negative externalities resulting from a transboundary pollution non-cooperative game wherein countries are dynamically involved. Based on a linear-quadratic specification for the instantaneous revenue function, we assess differences related to both transient path and steady state between cooperative solution, open-loop and Markov perfect Nash equilibria (MPNE). Regarding the methodological contribution of the paper, we suggest a particular structure of the conjectured value function to solve MPNE problems with multiplicative interaction between state variables in one state equation, so that third-order terms that arise in the Hamilton–Jacobi–Bellman equation are made negligible. Using a collocation procedure, we confirm the validity of the particular structure of the conjectured value function. The results suggest unexpected contrasts in terms of pollution control and environmental absorption efficiency management: (i) in the long run, an open-loop Nash equilibrium (OLNE) allows equivalent emissions to the social optimum but requires greater restoration efforts; (ii) although an MPNE is likely to end up with lower emissions and greater restoration efforts than an OLNE, it has a much greater chance of falling in the emergency area; (iii) the absence of cooperation and or precommitment becomes more costly as the initial absorption efficiency decreases; (iv) more heavily discounted MPNE strategies are less robust than OLNE to prevent irreversible switching of the biosphere from a carbon sink to a source.

Nurturing an Infant Industry by Markovian Subsidy Schemes

We model a small open economy with an infant industry facing competition from imports. We derive the welfare maximizing output path and knowledge path for an infant industry under the central planner who can dictate the industry output. We next show how the social planner’s optimal path can be achieved when the infant industry is in private hand, focusing on two cases: the case where the infant industry consists of a monopoly and the case where it is a duopoly. In the case of a monopoly we show that free trade can induce the monopoly to choose the socially optimal production path. Contrary to conventional wisdom, we show that the volume of imports is large when the stock of knowledge is small, and gradually declines as this stock grows. In the case of a duopoly with knowledge spillovers we derive a family of subsidy rules capable of inducing a Markov perfect Nash equilibrium that replicates the social optimum. When the subsidy rule is linear affine in the state variable, we show that the subsidy rate per unit of output must be an increasing function of the stock. The underlying intuition is that the government should put domestic firms under a tough competition in their infancy with a promise to make their life easier as their knowledge grows.

Selection of a Markov Perfect Nash Equilibrium in a Class of Differential Games

This paper revisits the problem of how to select an equilibrium in a differential game in the case of multiplicity of Nash equilibria. Most of the previous applied dynamic games literature has considered pre-play negotiations between players, implicitly or explicitly, with the aim of reaching an agreement on the selection of the pair of strategies. The main objective of this paper is to determine what would be the equilibrium to be played without pre-play communications. We study the linear and nonlinear Markov perfect Nash equilibria for a class of well-known models in the literature if pre-play communications are eliminated. We analyze both symmetric and nonsymmetric strategies. We show that the nonlinear strategies are not always the optimal strategies implemented when pre-play communications are removed. We conclude that in the presence of multiple equilibria and without pre-play communications the equilibria actually implemented are symmetric piecewise linear Markov perfect Nash equilibria at least for a range of initial values of the state variable.

Certainty Equivalence Principle in Stochastic Differential Games: An Inverse Problem Approach

This paper aims to characterize a class of stochastic differential games which satisfy the certainty equivalence principle. This means that the Markov Perfect Nash Equilibrium is also an equilibrium of the associated deterministic game. By focusing on a model scalar game with linear dynamics in the players’ strategies, we solve an inverse problem to find strictly concave utility functions, not necessarily quadratic or logarithmic, for which certainty equivalence is fulfilled. We show a variety of models that satisfy the certainty equivalence principle in a non cooperative game of a productive asset.

An Empirical Model of Drug Detailing: Dynamic Competition and Policy Implications

The practice of detailing in the marketing of prescription drugs is undergoing significant changes. For instance, in September 2013, the Physician Payment Sunshine Act went into full effect. The accompanying transparency requirements have prompted physician practices and hospitals to severely restrict pharmaceutical sales representatives’ direct access to their physicians. Despite all the attention in the popular press, scant scholarly research has investigated how these restrictions on physician access impact physician prescription behavior and competitive detailing to physicians. To analyze the impact of these restrictions, we develop a structural model of how pharmaceutical firms compete dynamically to schedule detailing to physicians. Detailing activities are known to have significant carryover effects that are captured in a first-stage model of physicians’ demand for prescription drugs. We also specify detailing policy functions that describe each firm’s observed detailing actions. In a second stage, we estimate a model that describes costs of detailing, assuming that the observed detailing levels are consistent with a Markov perfect Nash equilibrium. The estimated structural model is used to examine the implications of restrictions on the amount of detailing via counterfactual simulations. We find that restriction policies would increase the market share of a nondrug-treatment-only option but impact firms asymmetrically; firms that are strong in detailing and/or rely more on detailing would be hurt more. Unexpectedly, a policy that imposes a ceiling on detailing frequency could significantly reduce detailing of all firms in the market, including those firms with average detailing levels below the ceiling, and effectively would soften competition between firms and enhance their profits. This paper was accepted by J. Miguel Villas-Boas, marketing.

Cooperative and Noncooperative Extraction in a Common Pool with Habit Formation

The present paper considers the exploitation of a common-property, nonrenewable resource, by individuals subject to habit formation. We formalize their behavior by means of a utility function, depending on the difference between the individuals’ current consumption and the consumption level which they aspire, the latter being a weighted average of past consumptions in the population. We derive and compare the benchmark cooperative solution and a noncooperative Markov-perfect Nash equilibrium of the differential game. We investigate how the intensity, persistence and initial level of habits shape the cooperative and noncooperative solutions. We prove that habit formation may either mitigate or worsen the tragedy of the commons.

A Strategic Dynamic Programming Method for Studying Short-Memory Equilibria of Stochastic Games with Uncountable Number of States

We study a class of infinite horizon stochastic games with uncountable number of states. We first characterize the set of all (nonstationary) short-term (Markovian) equilibrium values by developing a new (Abreu et al. in Econometrica 58(5):1041–1063, 1990)-type procedure operating in function spaces. This (among others) proves Markov perfect Nash equilibrium (MPNE) existence. Moreover, we present techniques of MPNE value set approximation by a sequence of sets of discretized functions iterated on our approximated APS-type operator. This method is new and has some advantages as compared to Judd et al. (Econometrica 71(4):1239–1254, 2003), Feng et al. (Int Econ Rev 55(1):83–110, 2014), or Sleet and Yeltekin (Dyn Games Appl doi: 10.1007/s13235-015-0139-1 , 2015). We show applications of our approach to hyperbolic discounting games and dynamic games with strategic complementarities.