Markov Strategies Research Articles

On Bellman's equations for mean and variance control of a Markov diffusion

A controlled diffusion process is considered with cost functions of ‘mean and variance’ type. A regularization is proposed for computing the value of the cost function via Bellman's equations. The latter equations are in particular useful because they imply sufficiency of markovian strategies, at least, for regularized versions of processes. For the diffusion without control, a system of two well-posed linear partial differential equations (PDEs) is derived, similar to Kac's and Dynkin's moment equations, along with an equivalent single degenerate equation which turns out to be well-posed, too.

On Optimal and Neutrally Stable Population Equilibrium in Voluntary Partnership Prisoner's Dilemma Games

In a voluntary partnership prisoner's dilemma, equilibrium discounted payoff at the origin of a partnership also serves as the worst feasible threat with which to sustain cooperation. In a population model, we introduce the Markov strategy and characterize neutrally stable equilibrium. We construct simple Markov strategies that achieve the highest discounted payoff at the origin among all subgame perfect equilibrium distributions that have either eternal cooperation or eternal (matched) alternation on their equilibrium paths. Partnerships past the first strangers’ phase never break up, and potential punishments are all performed within the partnership. With this feature, our optimal Markov equilibria prove to be neutrally stable, while many commonly known ones in the literature are not.

The Compellence Dilemma: International Disputes with Violent Groups

It is widely recognized that the security of states are strongly influenced by the internal policies of other states. Although states threatened by the internal policies of an adversary often consider the use of coercive instruments to compel a change to the policy, I argue that a dilemma arises in the use of harsh punishments that renders them unattractive and incredible. I analyze this compellence dilemma in the situation where a state is targeted by a violent non-state group that is hosted by a second state. Thus, the security of the state targeted by the group is threatened by the host state's provision of an international base for the group. I argue that although harsher punishments make cooperation with the target state against the group more attractive for the host, they are also destabilizing and make the host a less capable agent in the future. I analyze the compellence dilemma with a three-player infinitely repeated dynamic game in which the actors play stationary Markov strategies. I find that the existence of the compellence dilemma is pervasive in equilibrium and often prevents the target from utilizing a harsh punishment even when such a punishment would compel the host to cooperate. Concerns about future capacity and group power drive this result.

Dynamic demand and noncompetitive intertemporal output adjustments

This paper integrates dynamics on the demand and on the supply side, which are characteristic for many goods, in particular fuels, due to costly adjustments. Furthermore, firms often pursue a quantity strategy and decide on output adjustments, additional capacities, etc. instead of fixing prices, e.g., OPEC moved from posting prices to adjusting output in 1986. This paper provides a corresponding framework (surprisingly ignored in the literature) and derives intertemporal market equilibria: cooperative, and noncooperative in open loop and in Markov strategies. The equilibrium in Markov strategies is different, mostly opposite to the much discussed sticky price models, the qualitative properties are non-trivial and some of them are even surprising, e.g., increased demand sluggishness can raise supply.

A dynamic model for advertising and pricing competition between national and store brands

Abstract We study the relationship between the pricing and advertising decisions in a channel where a national brand is competing with a private label. We consider a differential game that incorporates the carryover effects of brand advertising over time for both the manufacturer and the retailer and we account for the complementary and competitive roles of advertising. Analysis of the obtained equilibrium Markov strategies shows that the relationship between advertising and pricing decisions in the channel depends mainly on the nature of the advertising effects. In particular, the manufacturer reacts to higher competitive retailer’s advertising levels by offering price concessions and limiting his advertising expenditures. The retailer’s optimal reaction to competitive advertising effects in the channel depends on two factors: (1) the price competition level between the store and the national brands and (2) the strength of the competitive advertising effects. For example, in case of intense price competition between the two brands combined with a strong manufacturer’s competitive advertising effect, the retailer should lower both the store and the national brands’ prices as a reaction to higher manufacturer’s advertising levels. For the retailer, the main advantage from boosting his competitive advertising investments seems to be driven by increased revenues from the private label. The retailer should however limit his investments in advertising if the latter generates considerable competitive effects on the national brand’s sales.

INTERTEMPORAL INVESTMENTS INTO SYNFUELS

Abstract This paper considers the interactions of consumers demanding energy services provided by fossil fuels delivered by a cartel and a synfuel industry (competitive, cartelized or run by a consumer cartel, say International Energy Agency [IEA]). The emphasis of this paper is to account for the necessarily sluggish build up capacities instead of the instantaneous unconstrained availability assumed in the backstop literature and to investigate intertemporal and subgame perfect equilibria (in Markov strategies).

Determination of optimal penalties for antitrust violations in a dynamic setting

We analyze a differential game describing the interactions between a firm that might be violating competition law and the antitrust authority. The objective of the authority is to minimize social costs (loss in consumer surplus) induced by an increase in prices above marginal costs. It turns out that the penalty schemes which are used now in EU and US legislation appear not to be as efficient as desired from the point of view of minimization of consumer loss from price-fixing activities of the firm. In particular, we prove that full compliance behavior is not sustainable as a Nash Equilibrium in Markovian strategies over the whole planning period, and, moreover, that it will never arise as the long-run steady-state equilibrium of the model. We also investigate the question which penalty system enables us to completely deter cartel formation in a dynamic setting. We found that this socially desirable outcome can be achieved in case the penalty is an increasing function of the degree of offence and is negatively related to the probability of law enforcement.

Tragedy of the Commons in a Stochastic Game of a Stock Externality

AbstractThis paper investigates a tragedy of the commons where an Ito‐process traces the accumulation of pollution and differentiates between reversible (i.e., clean up is feasible) and irreversible emissions (past pollution cannot be undone). The reversible case allows for an explicit analytical solution, while other means are necessary to characterize irreversible outcomes. More precisely, a different characterization of equilibria (in Markov strategies) as smooth connections between an initial and a stopping manifold is suggested.

Search and Intermediation: Toward a Model of the Merchant Trader

This paper develops a dynamic model of an economy in which homogeneous agents choose between specializing as producers or as merchants, and can change occupation at any time. Merchants operate along a decentralized search market and provide immediacy in exchange in return for a price. We characterize equilibria in symmetric Markov strategies, and derive conditions under which merchants and their clients form a repeated relationship. We analyze welfare and discuss the prospect of an endogenous rise of an institution of intermediation.

Go for broke or play it safe? Dynamic competition with choice of variance

We consider a differential game in which the joint choices of the two players influence the variance, but not the mean, of the one‐dimensional state variable. We show that a pure strategy perfect equilibrium in stationary Markov strategies (ME) exists and has the property that patient players choose to play it safe when sufficiently ahead and to take risks when sufficiently behind. We also provide a simple condition that implies both players choose risky strategies when neither one is too far ahead, a situation that ensures a dominant player emerges “quickly.”

On Limits to the Use of Linear Markov Strategies in Common Property Natural Resource Games

We derive conditions that must be satisfied by the primitives of the problem in order for an equilibrium in linear Markov strategies to exist in some common property natural resource differential games. These conditions impose restrictions on the admissible form of the natural growth function, given a benefit function, or on the admissible form of the benefit function, given a natural growth function.

Do multiple Nash equilibria in Markov strategies mitigate the tragedy of the commons?

In a seminal paper, Dockner and van Long [1993. International pollution control: cooperative versus non-cooperative strategies. Journal of Environmental Economics and Management 25, 13–29] argue that nonlinear strategies allow to protect the commons compared with the linear (and singular) strategy. This paper shows that the existence of multiple equilibria depends on preference characteristics and that multiple equilibria need not foster conservation. In particular, if the (absolute) elasticity of marginal utility is decreasing, then multiple equilibria aggravate the tragedy of the commons compared with the singular (or saddlepoint) strategy.

Evolutionary Markovian strategies in [formula omitted] spatial games

Evolutionary spatial 2 × 2 games between heterogeneous agents are analyzed using different variants of cellular automata (CA). Agents play repeatedly against their nearest neighbors 2 × 2 games specified by a rescaled payoff matrix with two parameters. Each agent is governed by a binary Markovian strategy (BMS) specified by four conditional probabilities [ p R , p S , p T , p P ] that take values 0 or 1. The initial configuration consists in a random assignment of “strategists” among the 2 4 = 16 possible BMS. The system then evolves within strategy space according to the simple standard rule: each agent copies the strategy of the neighbor who got the highest payoff. Besides on the payoff matrix, the dominant strategy—and the degree of cooperation—depend on (i) the type of the neighborhood (von Neumann or Moore); (ii) the way the cooperation state is actualized (deterministically or stochastically); and (iii) the amount of noise measured by a parameter ε . However a robust winner strategy is [1,0,1,1].

Markovian segmentation and parameter estimation on graphics hardware

In this paper, we show how Markovian strategies used to solve well-known segmentation problems such as motion estima- tion, motion detection, motion segmentation, stereovision, and color segmentation can be significantly accelerated when implemented on programmable graphics hardware. More precisely, we expose how the parallel abilities of a standard graphics processing unit usu- ally devoted to image synthesis can be used to infer the labels of a segmentation map. The problems we address are stated in the sense of the maximum a posteriori with an energy-based or proba- bilistic formulation, depending on the application. In every case, the label field is inferred with an optimization algorithm such as iterated conditional mode (ICM) or simulated annealing. In the case of probabilistic segmentation, mixture parameters are estimated with the K-means and the iterative conditional estimation (ICE) proce- dure. For both the optimization and the parameter estimation algo- rithms, the graphics processor unit's (GPU's) fragment processor is used to update in parallel every labels of the segmentation map, while rendering passes and graphics textures are used to simulate optimization iterations. The hardware results obtained with a mid- end graphics card, show that these Markovian applications can be accelerated by a factor of 4 to 200 without requiring any advanced skills in hardware programming. © 2006 SPIE and

Dynamic Voluntary Provision of Public Goods and Optimal Steady-State Subsidies

This paper examines a differential game model of voluntary provision of a public good in which private agents' contributions accumulate over time and derives subsidy rules that achieve the socially efficient steady state. It is shown that the optimal subsidy rule is a simple one when agents use the open-loop strategy, while under Markovian strategies it intricately depends on the parameters of the economy.

Capital accumulation games with a non-concave production function

We consider an economy with a single capital stock that two agents strategically exploit by choosing a consumption profile over an infinite time horizon. We analyze two different games and their corresponding equilibria. In one game firms are able to pre-commit and choose simple time functions as their strategies. In the other game agents are assumed to employ Markov strategies. It turns out that in the case of pre-commitment there exists a threshold level of the capital stock such that if the initial stock is above this threshold, equilibrium consumption converges to the efficient steady state while if the initial condition is below it, the capital stock converges to zero. In case of the Markov equilibrium there exists a unique interior steady state that is globally stable.

Extended estimator approach for2×2games and its mapping to the Ising Hamiltonian

We consider a system of adaptive self-interested agents interacting by playing an iterated pairwise prisoner's dilemma (PD) game. Each player has two options: either cooperate (C) or defect (D). Agents have no (long term) memory to reciprocate nor identifying tags to distinguish C from D. We show how their 16 possible elementary Markovian (one-step memory) strategies can be cast in a simple general formalism in terms of an estimator of expected utilities Delta*. This formalism is helpful to map a subset of these strategies into an Ising Hamiltonian in a straightforward way. This connection in turn serves to shed light on the evolution of the iterated games played by agents, which can represent a broad variety of individuals from firms of a market to species coexisting in an ecosystem. Additionally, this magnetic description may be useful to introduce noise in a natural and simple way. The equilibrium states reached by the system depend strongly on whether the dynamics are synchronous or asynchronous and also on the system connectivity.

Social security as Markov equilibrium in OLG models

This paper studies the characteristics of intergenerational transfers in a standard overlapping generations model with short lived governments that care about the welfare of young generations only. A number of authors have shown that simple intergenerational games, in which in each period the current young generation plays as a dictator, are able to deliver political equilibria with social security even if the underlying competitive equilibrium is not dynamically inefficient. These authors have either derived pure steady state results or have relied on subgame-perfectness. This paper extends these results deriving Markov subgame perfect equilibria (i.e. that depend only upon the period t state variable, which is the stock of capital). Non-Markov subgame perfect equilibria assume agents know all the past history of the game; they cannot predict when the social security system will emerge and whether or not it will eventually emerge; they prescribe that generations that never deviated may be punished. Markov equilibria, placing more restrictions on the structure of the game, are able to deliver solutions that do not suffer from these drawbacks. As the paper shows, however, Markov strategies may produce unstable dynamics.

Strategic Experimentation with Exponential Bandits

This paper studies a game of strategic experimentation with two-armed bandits whose risky arm might yield a payoff only after some exponentially distributed random time. Because of free-riding, there is an inefficiently low level of experimentation in any equilibrium where the players use stationary Markovian strategies with posterior beliefs as the state variable. After characterizing the unique symmetric Markovian equilibrium of the game, which is in mixed strategies, we construct a variety of pure-strategy equilibria. There is no equilibrium where all players use simple cut-off strategies. Equilibria where players switch finitely often between the roles of experimenter and free-rider all lead to the same pattern of information acquisition; the efficiency of these equilibria depends on the way players share the burden of experimentation among them. In equilibria where players switch roles infinitely often, they can acquire an approximately efficient amount of information, but the rate at which it is acquired still remains inefficient; moreover, the expected payoff of an experimenter exhibits the novel feature that it rises as players become more pessimistic. Finally, over the range of beliefs where players use both arms a positive fraction of the time, the symmetric equilibrium is dominated by any asymmetric one in terms of aggregate payoffs.

On Capturing Oil Rents with a National Excise Tax Revisited

In this paper the scope of Bergstrom's (1982) results is studied. Moreover, his analysis is extended assuming that extraction cost is directly related to accumulated extractions. For the case of a competitive market it is found that the optimal policy is a constant tariff if extraction is costless. However, with depletion effects, the optimal tariff must ultimately be decreasing. For the case of a monopolistic market the results depend crucially on the kind of strategies the importing country governments can play and on whether the monopolist chooses the price or extraction rate. For a price-setting monopolist it is shown that the importing countries cannot use a tariff to capture monopoly rents if they are constrained to use open-loop strategies, even if the governments sign a tariff agreement. This result is drastically modified if the importing countries in the tariff agreement use Markov (feedback) strategies. For a quantity-setting monopolist the nature of the game changes and the importing country governments find it advantageous to set a tariff on resource importations. Moreover, in this case the importing countries in a tariff agreement enjoy a strategic advantage which allows them to behave as a leader.