Nonzero-sum Stochastic Games Research Articles

Best-Response Dynamics for Evolutionary Stochastic Games

We introduce a version of best-response (BR) dynamics for stochastic games using the framework of evolutionary stochastic games proposed in [Flesch, J., Parthasarathy, T., Thuijsman, F. and Uyttendaele, P. [2013] Evolutionary stochastic games, Dyn. Games Appl. 3, 207–219]. This dynamics is defined on the simplex of all population distributions, as in classical evolutionary games. In this BR dynamics, it is shown that the set of all population distributions inducing Nash equilibria is globally asymptotically stable, for stable stochastic games. Also, if we transform the BR dynamics into the space of stationary strategies, the set of all Nash Equilibria is asymptotically stable. Another version of BR dynamics is defined in the space of stationary strategies for zero-sum stochastic games by [Leslie, D. S., Perkins, S. and Xu, Z. [2020] Best-response dynamics in zero-sum stochastic games, J. Econ. Theory 189, 105095]. To compare both versions of BR dynamics, we extend the evolutionary stochastic games model to asymmetric stochastic games and employ an equivalent definition of Nash equilibria. This comparison is illustrated using examples and it is observed that the trajectories of both versions of BR dynamics do converge to the same Nash equilibrium. The BR dynamics proposed in this paper has the advantage of being valid for nonzero sum stochastic games and conforms to both classical game dynamics and the dynamics of stationary strategies in stochastic games.

Risk-sensitive first passage stochastic games with unbounded costs

ABSTRACT This paper studies discrete-time nonzero-sum stochastic games under the risk-sensitive first passage discounted cost criterion. The state space is a countable set and the costs are allowed to be unbounded. Under the suitable optimality conditions, we prove that the risk-sensitive first passage discounted optimal value function of each player is a unique solution to the risk-sensitive first passage optimality equation via an approximation method. Moreover, by the risk-sensitive first passage discounted optimality equation, we show the existence of a randomized Markov Nash equilibrium. Finally, three examples are given to illustrate the results.

Nonzero-Sum Risk-Sensitive Continuous-Time Stochastic Games with Ergodic Costs

We study nonzero-sum stochastic games for continuous time Markov decision processes on a denumerable state space with risk-sensitive ergodic cost criterion. Transition rates and cost rates are allowed to be unbounded. Under a Lyapunov type stability assumption, we show that the corresponding system of coupled HJB equations admits a solution which leads to the existence of a Nash equilibrium in stationary strategies. We establish this using an approach involving principal eigenvalues associated with the HJB equations. Furthermore, exploiting appropriate stochastic representation of principal eigenfunctions, we completely characterize Nash equilibria in the space of stationary Markov strategies.

Nonzero-Sum Stochastic Games and Mean-Field Games with Impulse Controls

We consider a general class of nonzero-sum N-player stochastic games with impulse controls, where players control the underlying dynamics with discrete interventions. We adopt a veri?cation approach and provide su?cient conditions for the Nash equilibria (NEs) of the game. We then consider the limiting situation when N goes to in?nity, that is, a suitable mean-?eld game (MFG) with impulse controls. We show that under appropriate technical conditions, there exists a unique NE solution to the MFG, which is an ϵ-NE approximation to the N-player game, with [Formula: see text]. As an example, we analyze in detail a class of two-player stochastic games which extends the classical cash management problem to the game setting. In particular, we present numerical analysis for the cases of the single player, the two-player game, and the MFG, showing the impact of competition on the player’s optimal strategy, with sensitivity analysis of the model parameters.

Nonzero-sum Risk-Sensitive Average Stochastic Games: The Case of Unbounded Costs

In this paper, we study discrete-time nonzero-sum stochastic games under the risk-sensitive average cost criterion. The state space is a denumerable set, the action spaces of players are Borel spaces, and the cost functions are unbounded. Under suitable conditions, we first introduce the risk-sensitive first passage payoff functions and obtain their properties. Then, we establish the existence of a solution to the risk-sensitive average cost optimality equation of each player for the case of unbounded cost functions and show the existence of a randomized stationary Nash equilibrium in the class of randomized history-dependent strategies. Finally, we use a controlled population system to illustrate the main results.

Nonzero-Sum Stochastic Differential Reinsurance Games with Jump–Diffusion Processes

In this paper, a nonzero-sum stochastic differential reinsurance game is studied. A model including controls for the market share (advertising), investment, and reinsurance policies is considered. A jump–diffusion process is used to represent insurance claims. Necessary conditions that would lead to the Nash equilibrium can be found in the duopoly game we consider. Cases with and without controls on the reinsurance are discussed separately. Closed-form solutions for optimal strategies are derived by applying the Hamilton–Jacobi–Bellman equation. Numerical examples are given to validate the correctness of our results.

Stochastic differential games for optimal investment problems in a Markov regime-switching jump-diffusion market

We apply dynamic programming principle to discuss two optimal investment problems by using zero-sum and nonzero-sum stochastic game approaches in a continuous-time Markov regime-switching environment within the frame work of behavioral finance. We represent different states of an economy and, consequently, investors’ floating levels of psychological reactions by a D-state Markov chain. The first application is a zero-sum game between an investor and the market, and the second one formulates a nonzero-sum stochastic differential portfolio game as the sensitivity of two investors’ terminal gains. We derive regime-switching Hamilton–Jacobi–Bellman–Isaacs equations and obtain explicit optimal portfolio strategies with Feynman–Kac representations of value functions. We illustrate our results in a two-state special case and observe the impact of regime switches by comparative results.

An Extended Mean Field Game for Storage in Smart Grids

We consider a stylized model for a power network with distributed local power generation and storage. This system is modeled as a network connection of a large number of nodes, where each node is characterized by a local electricity consumption, has a local electricity production (photovoltaic panels for example) and manages a local storage device. Depending on its instantaneous consumption and production rate as well as its storage management decision, each node may either buy or sell electricity, impacting the electricity spot price. The objective at each node is to minimize energy and storage costs by optimally controlling the storage device. In a noncooperative game setting, we are led to the analysis of a nonzero sum stochastic game with N players where the interaction takes place through the spot price mechanism. For an infinite number of agents, our model corresponds to an extended mean field game. We are able to compare this solution to the optimal strategy of a central planner and in a linear quadratic setting, we obtain and explicit solution to the extended mean field game and we show that it provides an approximate Nash equilibrium for N-player game.

Nonzero-Sum Stochastic Games with Probability Criteria

In this paper, we consider two-person nonzero-sum discrete-time stochastic games under the probability criterion. Taking $$\lambda $$ for player 1 and $$\mu $$ for player 2 as their profit goal, the two players are concerned with the probabilities that the rewards they earn before the first passage to some target state set are more than $$\lambda $$ and $$\mu $$, respectively. We firstly give a characterization of the probabilities, and then, under a mild condition, we show that the optimal value function for each player is the unique solution to the corresponding optimality equation by an iterative approximation, and then establish the existence of Nash equilibria. Finally, a queueing system is provided to show the application of our main result.

Nash equilibrium approximation of some class of stochastic differential games: A combined Chebyshev spectral collocation method with policy iteration

This study relates to nonzero-sum stochastic games with perfect information. It proposes an efficient combined Chebyshev spectral collocation method (CSCM) with the policy iteration (PI) algorithm for solving nonlinear coupled Hamilton–Jacobi (HJ) equations. The proposed approach is comprised of two steps. First, the PI algorithm is used to reduce the nonlinear coupled HJ equations to a sequence of linear uncoupled PDEs. Then, these equations are approximated by the CSCM. The CSCM＋PI is especially useful when the CSCM fails due to the increasing number of collocation points for solving the associated system of nonlinear algebraic equations. The main advantage of the resulting method is that it converts nonlinear coupled HJ equations to the systems of linear algebraic equations, which can be solved readily. Convergence analysis of this method is also provided in detail. To confirm the accuracy and validity of the proposed computational algorithm, several illustrative examples are presented.

$N$-Person Nonzero-Sum Games for Continuous-Time Jump Processes With Varying Discount Factors

This paper is concerned with the first passage discounted payoff criterion for continuous-time $N$ -person nonzero-sum stochastic games in countable states and Borel action spaces, where the discount factor is nonconstant. For the game with unbounded reward functions and transition rates, the optimality is over the set of all randomized history-dependent policies. After showing the uniqueness of the solution to the Shapley equation, we give some suitable conditions imposed on the primitive data for the existence of Nash equilibria. Then, a financial system is introduced to illustrate the applications of the main result here.

A policy iteration algorithm for nonzero-sum stochastic impulse games

This work presents a novel policy iteration algorithm to tackle nonzero-sum stochastic impulse games arising naturally in many applications. Despite the obvious impact of solving such problems, there are no suitable numerical methods available, to the best of our knowledge. Our method relies on the recently introduced characterisation of the value functions and Nash equilibrium via a system of quasi-variational inequalities. While our algorithm is heuristic and we do not provide a convergence analysis, numerical tests show that it performs convincingly in a wide range of situations, including the only analytically solvable example available in the literature at the time of writing.

Continuous-time constrained stochastic games with average criteria

In this paper, we consider the continuous-time nonzero-sum stochastic games under the constrained average criteria. The state space is denumerable and the action space of each player is a general Polish space. The transition rates, reward and cost functions are allowed to be unbounded. The main hypotheses in this paper include the standard drift conditions, continuity-compactness condition and some ergodicity assumptions. By applying the vanishing discount method, we obtain the existence of stationary constrained average Nash equilibria.

Nonzero-Sum Risk-Sensitive Stochastic Games on a Countable State Space

The infinite horizon risk-sensitive discounted-cost and ergodic-cost nonzero-sum stochastic games for controlled Markov chains with countably many states are analyzed. For the discounted-cost game, we prove the existence of Nash equilibrium strategies in the class of Markov strategies under fairly general conditions. Under an additional weak geometric ergodicity condition and a small cost criterion, the existence of Nash equilibrium strategies in the class of stationary Markov strategies is proved for the ergodic-cost game. The key nontrivial contributions in the ergodic part are to prove the existence of a particular form of a (relative) value function solution to a player’s Bellman equation and the continuity of this solution with respect to the opponent’s strategies.

Orderfield property of mixtures of stochastic games

We consider certain mixtures, Γ, of classes of stochastic games and provide sufficient conditions for these mixtures to possess the orderfield property. For 2-player zero-sum and non-zero sum stochastic games, we prove that if we mix a set of states S1 where the transitions are controlled by one player with a set of states S2 constituting a sub-game having the orderfield property (where S1 ∩ S2 = ∅), the resulting mixture Γ with states S = S1 ∪ S2 has the orderfield property if there are no transitions from S2 to S1. This is true for discounted as well as undiscounted games. This condition on the transitions is sufficient when S1 is perfect information or SC (Switching Control) or ARAT (Additive Reward Additive Transition). In the zero-sum case, S1 can be a mixture of SC and ARAT as well. On the other hand,when S1 is SER-SIT (Separable Reward — State Independent Transition), we provide a counter example to show that this condition is not sufficient for the mixture Γ to possess the orderfield property. In addition to the condition that there are no transitions from S2 to S1, if the sum of all transition probabilities from S1 to S2 is independent of the actions of the players, then Γ has the orderfield property even when S1 is SER-SIT. When S1 and S2 are both SERSIT, their mixture Γ has the orderfield property even if we allow transitions from S2 to S1. We also extend these results to some multi-player games namely, mixtures with one player control Polystochastic games. In all the above cases, we can inductively mix many such games and continue to retain the orderfield property.

NASH EQUILIBRIA IN UNCONSTRAINED STOCHASTIC GAMES OF RESOURCE EXTRACTION

A class of nonzero-sum symmetric stochastic games of capital accumulation/resource extraction is considered. It is shown that Nash equilibria in the games with some natural constraints are also equilibrium solutions in unconstrained games and dominate in the Pareto sense an equilibrium leading to exhausting the entire resource stock in the first period of the game.

Non-Cooperative Outcomes for Stochastic Nash Games: Decision Strategies towards Multi-Attribute Performance Robustness

The advantages of compactness from logic of state-space model description and quantitativity from probabilistic knowledge of stochastic disturbances have been exploited to construct a situational awareness which then provides essential common knowledge to noncooperative decision makers about the adverse and dynamic environment within a linear-quadratic class of nonzero-sum stochastic games. It incorporates the perception of relevant attributes of the decision problem, comprehension of the meaning of the shared interaction model in combination with and in relation to various goals of non-cooperative decision makers so that future projection of higher-order characteristics of the Chi-squared random measures of performance is obtained with high confidence. New solution concepts, called the multi-cumulant Nash strategies are proposed to directly influence respective performance distributions and to effectively guarantee performance robustness for non-cooperative decision makers.

A Reinforcement Learning Model to Assess Market Power Under Auction-Based Energy Pricing

Auctions serve as a primary pricing mechanism in various market segments of a deregulated power industry. In day-ahead (DA) energy markets, strategies such as uniform price, discriminatory, and second-price uniform auctions result in different price settlements and thus offer different levels of market power. In this paper, we present a nonzero sum stochastic game theoretic model and a reinforcement learning (RL)-based solution framework that allow assessment of market power in DA markets. Since there are no available methods to obtain exact analytical solutions of stochastic games, an RL-based approach is utilized, which offers a computationally viable tool to obtain approximate solutions. These solutions provide effective bidding strategies for the DA market participants. The market powers associated with the bidding strategies are calculated using well-known indexes like Herfindahl-Hirschmann index and Lerner index and two new indices, quantity modulated price index (QMPI) and revenue-based market power index (RMPI), which are developed in this paper. The proposed RL-based methodology is tested on a sample network

Remarks on sensitive equilibria in stochastic games with additive reward and transition structure

A class of stochastic games with additive reward and transition structure is studied. For zero-sum games under some ergodicity assumptions 1-equilibria are shown to exist. They correspond to so-called sensitive optimal policies in dynamic programming. For a class of nonzero-sum stochastic games with nonatomic transitions nonrandomized Nash equilibrium points with respect to the average payoff criterion are also obtained. Included examples show that the results of this paper can not be extented to more general payoff or transition structure.

A Stochastic Game Approach for Modeling Wholesale Energy Bidding in Deregulated Power Markets

It has been noted in a recent report (Dec. 2002) from the General Accounting Office of the U.S. that various design flaws in wholesale markets and transmission services have created operational problems within and between wholesale markets. This paper presents a novel methodology to analyze and design the wholesale energy bidding aspect of a deregulated power market. We adopt a system-wide approach that considers many of the relevant features including transmission congestion. We develop a two-stage model: 1) a nonzero sum stochastic game with average reward for the wholesale energy market operation, and 2) a nonlinear programming (NLP) model for the unit-commitment (UC) and the optimal power-flow aspects. The solution approach for the two-stage model is based on a reinforcement-learning (RL) algorithm, which is designed to obtain Nash equilibrium policies. We use a three-retailer/three-supplier power network with and without congestion to test and benchmark our methodology.