Stochastic Dynamic Games Research Articles

DsGameSolver: A Python Program for Computing Markov Perfect Equilibria of Dynamic Stochastic Games

dsGameSolver is based on the homotopy method developed in Eibelshaeuser and Poensgen (2019). It is the first program capable of computing a Markov perfect equilibrium of any finite dynamic stochastic game -- subject to the usual restrictions on numerical accuracy and working memory. dsGameSolver supports general games with state-player-specific numbers of actions and individual discount factors, covering virtually any game of complete information. Furthermore, it is remarkably performant and able to handle games with thousands of state-player-actions. Finally, dsGameSolver allows users with basic programming skills in Python to formulate complex games with ease and to directly use the results for subsequent illustration and analysis.

Markov Quantal Response Equilibrium and a Homotopy Method for Computing and Selecting Markov Perfect Equilibria of Dynamic Stochastic Games

We formally define Markov quantal response equilibrium (QRE) and prove existence for all finite discounted dynamic stochastic games. The special case of logit Markov QRE constitutes a mapping from precision parameter λ to sets of logit Markov QRE. The limiting points of this correspondence are shown to be Markov perfect equilibria. Furthermore, the logit Markov QRE correspondence can be given a homotopy interpretation. We prove that for all games, this homotopy contains a branch connecting the unique solution at λ = 0 to a unique limiting Markov perfect equilibrium. This result can be leveraged both for the computation of Markov perfect equilibria, and also as a selection criterion.

Protocol invariance and the timing of decisions in dynamic games

We characterize a class of dynamic stochastic games that we call separable dynamic games with noisy transitions and establish that these widely used models are protocol invariant provided that periods are sufficiently short. Protocol invariance means that the set of Markov perfect equilibria is nearly the same irrespective of the order in which players are assumed to move within a period. Protocol invariance can facilitate applied work, and renders the implications and predictions of a model more robust. Our class of dynamic stochastic games includes investment games, research and development races, models of industry dynamics, dynamic public contribution games, asynchronously repeated games, and many other models from the extant literature.

A Markovian Mechanism of Proportional Resource Allocation in the Incentive Model as a Dynamic Stochastic Inverse Stackelberg Game

This paper considers resource allocation among producers (agents) in the case where the Principal knows nothing about their cost functions while the agents have Markovian awareness about his/her strategies. We use a dynamic setup of the stochastic inverse Stackelberg game as the model. We suggest an algorithm for solving this game based on Q-learning. The associated Bellman equations contain functions of one variable for the Principal and also for the agents. The new results are illustrated by numerical examples.

Dynamic stochastic games with random moves

We reformulate the quality ladder model of Pakes and McGuire, Rand Journal of Economics, 25(4), 555–589 (1994) as a dynamic stochastic game with random moves in which each period one firm is picked at random to make an investment decision. Contrasting this model to the standard version with simultaneous moves illustrates the computational advantages of random moves. In particular, the quality ladder model with random moves avoids the curse of dimensionality in computing firms’ expectations over all possible future states and is therefore orders of magnitude faster to solve than its counterpart with simultaneous moves when there are more than just a few firms. Perhaps unexpectedly, the equilibria of the quality ladder model with random moves are practically indistinguishable from those of the model with simultaneous moves.

Mutually Quadratically Invariant Information Structures in Two-Team Stochastic Dynamic Games

We formulate a two-team linear quadratic stochastic dynamic game featuring two opposing teams each with decentralized information structures. We introduce the concept of mutual quadratic invariance (MQI), which, analogously to quadratic invariance in (single team) decentralized control, defines a class of interacting information structures for the two teams under which optimal feedback control strategies are linear and easy to compute. We show that for zero-sum two-team dynamic games with MQI information structure, structured state feedback saddle-point equilibrium strategies can be computed from equivalent structured disturbance feedforward saddle point equilibrium strategies. We also show that there is a saddle-point equilibrium in linear strategies even when the teams are allowed to use nonlinear strategies. However, for nonzero-sum games we show via a counterexample that a similar equivalence fails to hold. The results are illustrated with a simple yet rich numerical example that illustrates the importance of the information structure for dynamic games.

The benefits of international cooperation under climate uncertainty: a dynamic game analysis

AbstractThis paper investigates the benefits of international cooperation under uncertainty about global warming through a stochastic dynamic game. We analyze the benefits of cooperation both for the case of symmetric and asymmetric players. It is shown that the players’ combined expected payoffs decrease as climate uncertainty becomes larger, whether or not they cooperate. However, the benefits from cooperation increase with climate uncertainty. In other words, it is more important to cooperate when facing higher uncertainty. At the same time, more transfers will be needed to ensure stable cooperation among asymmetric players.

Joint Stochastic Geometry and Mean Field Game Optimization for Energy-Efficient Proactive Scheduling in Ultra Dense Networks

In this paper, we investigate the energy efficiency performance of optimal proactive scheduling strategies in the context of ultra-dense networks. The network consists of a superposition of homogeneous Poisson point process, whereas the users requests follow a space-time homogeneous point process. The objective is to define the optimal transmission powers at any time, that allows to completely serve every user request, while minimizing the total consumed energy. We also assume the system has predictive knowledge about the future transmission contexts. The problem is cast as a dynamic stochastic game which is hard to solve in ultra-dense networks, due to a complex coupling in the interference term, the large number of elements interacting, as well as uncertainties on the channel dynamics, interference, and future requests. Our contribution first lies in addressing the inherent complexity issue of the optimization, by transitioning into an equivalent and more tractable mean field game. Second, we propose to combine this mathematical framework with elements of stochastic geometry. The numerical simulations provide good insights on notable performance gains in terms of energy efficiency, compared to reference scheduling strategies. Additional simulations harnessing the impact of future knowledge uncertainty on the performance of the proposed strategies are also provided.

Decentralized Q-Learning for Stochastic Teams and Games

There are only a few learning algorithms applicable to stochastic dynamic teams and games which generalize Markov decision processes to decentralized stochastic control problems involving possibly self-interested decision makers. Learning in games is generally difficult because of the non-stationary environment in which each decision maker aims to learn its optimal decisions with minimal information in the presence of the other decision makers who are also learning. In stochastic dynamic games, learning is more challenging because, while learning, the decision makers alter the state of the system and hence the future cost. In this paper, we present decentralized Q-learning algorithms for stochastic games, and study their convergence for the weakly acyclic case which includes team problems as an important special case. The algorithms are decentralized in that each decision maker has access only to its own decisions and cost realizations as well as the state transitions; in particular, each decision maker is completely oblivious to the presence of the other decision makers. We show that these algorithms converge to equilibrium policies almost surely in large classes of stochastic games.

Dynamic Games With Asymmetric Information: Common Information Based Perfect Bayesian Equilibria and Sequential Decomposition

We formulate and analyze a general class of stochastic dynamic games with asymmetric information arising in dynamic systems. In such games, multiple strategic agents control the system dynamics and have different information about the system over time. Because of the presence of asymmetric information, each agent needs to form beliefs about other agents' private information. Therefore, the specification of the agents' beliefs along with their strategies is necessary to study the dynamic game. We use Perfect Bayesian equilibrium (PBE) as our solution concept. A PBE consists of a pair of strategy profile and belief system. In a PBE, every agent's strategy should be a best response under the belief system, and the belief system depends on agents' strategy profile when there is signaling among agents. Therefore, the circular dependence between strategy profile and belief system makes it difficult to compute PBE. Using the common information among agents, we introduce a subclass of PBE called common information based perfect Bayesian equilibria (CIB-PBE), and provide a sequential decomposition of the dynamic game. Such decomposition leads to a backward induction algorithm to compute CIB-PBE. We illustrate the sequential decomposition with an example of a multiple access broadcast game. We prove the existence of CIB-PBE for a subclass of dynamic games.

Strategic carbon taxation and energy pricing under the threat of climate tipping events

An appropriate design of climate mitigation policies such as carbon taxes may face a lot of challenges in reality, e.g., the strategic behavior of fossil fuel producers, and huge uncertainty surrounding the climate system. This paper investigated the effect of possible climate tipping events on optimal carbon taxation and energy pricing, taking into account the strategic behavior of energy consumers/producers and the uncertainty of tipping points through a stochastic dynamic game. The game was solved numerically to get some insights into the equilibrium carbon taxation and energy pricing strategies under the threat of possible tipping events. The results suggest that the sudden occurrence of tipping events will shift the carbon tax upwards but shift the (wellhead) fuel price downwards, which generates an overall effect of a sudden increase in the consumer price. The irreversibility of the damage caused by a tipping point implies more aggressive carbon taxation. Moreover, the design of climate policy should adjust to tipping probabilities, damage uncertainty, and types of tipping events.

Price of anarchy in a linear-state stochastic dynamic game

We consider a linear-state dynamic game played over an event tree and determine analytically the price of anarchy (PoA), which is given by the ratio of player’s outcome in the cooperative game to her outcome in the noncooperative game. Next, we illustrate our findings in the context of a pollution control problem. We conduct a sensitivity analysis to assess the impact on PoA, and its lower and upper bounds, of varying the different parameter values.

Zero-sum risk-sensitive stochastic games

In this paper we consider two-person zero-sum risk-sensitive stochastic dynamic games with Borel state and action spaces and bounded reward. The term risk-sensitive refers to the fact that instead of the usual risk neutral optimization criterion we consider the exponential certainty equivalent. The discounted reward case on a finite and an infinite time horizon is considered, as well as the ergodic reward case. Under continuity and compactness conditions we prove that the value of the game exists and solves the Shapley equation and we show the existence of optimal (non-stationary) strategies. In the ergodic reward case we work with a local minorization property and a Lyapunov condition and show that the value of the game solves the Poisson equation. Moreover, we prove the existence of optimal stationary strategies. A simple example highlights the influence of the risk-sensitivity parameter. Our results generalize findings in Basu and Ghosh (2014) and answer an open question posed there.

Ultra Dense Small Cell Networks: Turning Density Into Energy Efficiency

In this paper, a novel approach for joint power control and user scheduling is proposed for optimizing energy efficiency (EE), in terms of bits per unit energy, in ultra dense small cell networks (UDNs). Due to severe coupling in interference, this problem is formulated as a dynamic stochastic game (DSG) between small cell base stations (SBSs). This game enables to capture the dynamics of both the queues and channel states of the system. To solve this game, assuming a large homogeneous UDN deployment, the problem is cast as a mean-field game (MFG) in which the MFG equilibrium is analyzed with the aid of low-complexity tractable partial differential equations. Exploiting the stochastic nature of the problem, user scheduling is formulated as a stochastic optimization problem and solved using the drift plus penalty (DPP) approach in the framework of Lyapunov optimization. Remarkably, it is shown that by weaving notions from Lyapunov optimization and mean-field theory, the proposed solution yields an equilibrium control policy per SBS which maximizes the network utility while ensuring users' quality-of-service. Simulation results show that the proposed approach achieves up to 70.7% gains in EE and 99.5% reductions in the network's outage probabilities compared to a baseline model which focuses on improving EE while attempting to satisfy the users' instantaneous quality-of-service requirements.

Stochastic Dynamic Game between Hydropower Plant and Thermal Power Plant in Smart Grid Networks

In this paper, we study the competitive interactions between an autonomous pumped-storage hydropower plant and a thermal power plant in order to optimize power generation and storage. Each type of power plant individually tries to maximize its own profit by adjusting its strategy: both types of plants can sell their power to the market at a price depending on the demand and supply, as well as the dynamics over different periods of time; or alternatively, the thermal power plant can sell its power at a fixed price to the pumped-storage hydropower plant by storing the energy in the reservoir. A stochastic dynamic game is formulated to characterize this competition. The instantaneous market price is modeled as a Cournot duopoly game. Here, the dynamics comes from the varying water volume in the reservoir, and the stochastic aspect captures the uncertain natural inflow to and losses from the reservoir. The solutions are derived using the stochastic Hamilton–Jacobi–Bellman equations. We also show that the proposed game can converge to a feedback Nash equilibrium. The simulation results demonstrate the properties of the proposed game and suggest how to optimize the amounts of generation in hydropower and thermal power plants over time with the fluctuations of price. The proposed framework and games can reduce the peak-to-average ratio and total energy generation for the thermal plant, which helps power plant operation and reduces $\hbox{CO}_{2}$ emission.

Shadow Banks and Systemic Risks

We answer the following question: Does regulating the banking network increase systemic risk in the entire financial network in the presence of unregulated shadow banks? In order to answer this question, we introduce a formal definition of systemic risk based on the equilibrium state dynamics generated by a stochastic game of network formation with banks and shadow banks. Based on the equilibrium dynamics, we discuss the properties of equilibrium financial networks. The predictions of the model are consistent with empirical evidence in the literature. Our analysis makes clear the systemic importance of connections between commercial banks and shadow banks in determining the health of the entire financial network. Lastly, but most importantly, we give a definition of systemic risk based on equilibrium dynamics, and therefore, a definition which takes into account the strategic interactions of financial institutions when facing different regulatory policies. Given our definition of systemic risk, we discuss various policies for regulating the financial system. Then, specializing our dynamic stochastic game model to a more basic model in which banks and shadow banks form a network including connections to the real economy, we show, via numerical simulations of our stochastic game model under various regulatory regimes, that imposing bank regulations not faced by shadow banks does indeed increase systemic risk throughout the financial network.

Team Leadership and Performance: Combining the Roles of Direction and Participation

Due to the complexity and stochastic nature of knowledge-intensive projects (e.g., in software development, new product development, semiconductor design, pharmaceutical research), firms are increasingly organizing such work around teams, while appointing team leaders to act as managers as well as contributors to the project. One of the challenges that project team leaders face is how to combine their roles as individual contributors and managers. In this paper, we propose a model of team leadership and study how a contributor leader should adjust her managing effort over time and with respect to the project characteristics (e.g., team members' incentives and team size). Modeling the problem as a stochastic dynamic game, we show that it is optimal for the leader to follow one of two approaches: (i) being directive, i.e., contribute and also manage the team members' efforts to be above what they would choose to do in equilibrium, or (ii) being participative, i.e., not manage the team members' efforts and only contribute to the project. Moreover, we show that the intensity of the leader's managing effort may evolve non-monotonically over time as a result of the misalignment of incentives between the team leader and the other team members. Finally, we show that a participative leadership approach emerges when the team members have high incentives, or the team size is small, and that otherwise, the directive leadership approach emerges.

Two Monthly Continuous Dynamic Model Based on Nash Bargaining Theory for Conflict Resolution in Reservoir System

So far many optimization models based on Nash Bargaining Theory associated with reservoir operation have been developed. Most of them have aimed to provide practical and efficient solutions for water allocation in order to alleviate conflicts among water users. These models can be discussed from two viewpoints: (i) having a discrete nature; and (ii) working on an annual basis. Although discrete dynamic game models provide appropriate reservoir operator policies, their discretization of variables increases the run time and causes dimensionality problems. In this study, two monthly based non-discrete optimization models based on the Nash Bargaining Solution are developed for a reservoir system. In the first model, based on constrained state formulation, the first and second moments (mean and variance) of the state variable (water level in the reservoir) is calculated. Using moment equations as the constraint, the long-term utility of the reservoir manager and water users are optimized. The second model is a dynamic approach structured based on continuous state Markov decision models. The corresponding solution based on the collocation method is structured for a reservoir system. In this model, the reward function is defined based on the Nash Bargaining Solution. Indeed, it is used to yield equilibrium in every proper sub-game, thereby satisfying the Markov perfect equilibrium. Both approaches are applicable for water allocation in arid and semi-arid regions. A case study was carried out at the Zayandeh-Rud river basin located in central Iran to identify the effectiveness of the presented methods. The results are compared with the results of an annual form of dynamic game, a classical stochastic dynamic programming model (e.g. Bayesian Stochastic Dynamic Programming model, BSDP), and a discrete stochastic dynamic game model (PSDNG). By comparing the results of alternative methods, it is shown that both models are capable of tackling conflict issues in water allocation in situations of water scarcity properly. Also, comparing the annual dynamic game models, the presented models result in superior results in practice. Furthermore, unlike discrete dynamic game models, the presented models can significantly reduce the runtime thereby avoiding dimensionality problems.

Minimax estimation with intermittent observations

This paper considers a problem of minimax (or H∞) state estimation with intermittent observations. In this setting, the disturbance in the dynamical system and the sensor noise are controlled by adversaries, and the estimator receives the sensor measurements only sporadically, with the availability governed by an independent and identically distributed Bernoulli process. We cast this problem within the framework of stochastic zero-sum dynamic games. We first obtain a corresponding stochastic minimax state estimator (SMSE) and an associated generalized stochastic Riccati equation (GSRE) whose evolutions depend on two parameters: one that governs the random measurement arrivals and another one that quantifies the level of H∞ disturbance attenuation. We then analyze the asymptotic behavior of the sequence generated by the GSRE in the expectation sense, and its weak convergence. Specifically, we obtain threshold-type conditions above which the sequence generated by the GSRE can be bounded both below and above in the expectation sense. Moreover, we show that under some conditions, the norm of the sequence generated by the GSRE converges weakly to a unique stationary distribution. Finally, we prove that when the disturbance attenuation parameter goes to infinity, our asymptotic results are equivalent to the corresponding results from the literature on Kalman filtering with intermittent observations. We provide simulations to illustrate the results.

Stochastic game dynamics under demographic fluctuations

Frequency-dependent selection and demographic fluctuations play important roles in evolutionary and ecological processes. Under frequency-dependent selection, the average fitness of the population may increase or decrease based on interactions between individuals within the population. This should be reflected in fluctuations of the population size even in constant environments. Here, we propose a stochastic model that naturally combines these two evolutionary ingredients by assuming frequency-dependent competition between different types in an individual-based model. In contrast to previous game theoretic models, the carrying capacity of the population, and thus the population size, is determined by pairwise competition of individuals mediated by evolutionary games and demographic stochasticity. In the limit of infinite population size, the averaged stochastic dynamics is captured by deterministic competitive Lotka-Volterra equations. In small populations, demographic stochasticity may instead lead to the extinction of the entire population. Because the population size is driven by fitness in evolutionary games, a population of cooperators is less prone to go extinct than a population of defectors, whereas in the usual systems of fixed size the population would thrive regardless of its average payoff.