Stationary Strategies Research Articles

A Note on Zero-Sum Two-Person Undiscounted One Player Control Semi-Markov Games

Zero-sum two-person finite undiscounted (limiting ratio average) semi-Markov games are considered where both the transition probabilities and the transition times are controlled by a fixed player in all states. We prove that, if such a game is unichain, the value and optimal stationary strategies of the players can be obtained from an optimal solution of the linear programming algorithm for the associated one player control stochastic game using a data transformation method. An example is provided for such class of games.

Zero-sum stochastic games in continuous-time with risk-sensitive average cost criterion on a countable state space

We consider zero-sum stochastic games in continuous time with controlled Markov chains and with risk-sensitive average cost criterion. Here the transition and the cost rates may be unbounded. We prove the existence of the value of the game and a saddle-point equilibrium in the class of all stationary strategies under a Lyapunov stability condition. This is accomplished by establishing the existence of a principal eigenpair for the corresponding Hamilton-Jacobi-Isaacs (HJI) equation. This, in turn, is established by using a nonlinear version of Krein-Rutman theorem. We then obtain a characterization of the saddle-point equilibrium in terms of the corresponding HJI equation. Finally, we use a controlled population system to illustrate our results.

On the Existence of Stationary Nash Equilibria for Mean Payoff Games on Graphs

In this paper we extend the classical concept of positional strategies for a mean payoff game to a general mixed stationary strategy approach, and prove the existence of mixed stationary Nash equilibria for an arbitrary m-player mean payoff game on graphs. Traditionally, a positional strategy represents a pure stationary strategy in a classical mean payoff game, where a Nash equilibrium in pure stationary strategies in general may not exist. Based on a constructive proof of the existence of specific equilibria for an m-player mean payoff game we propose a new approach for determining the optimal mixed stationary strategies. Additionally we characterize and extend the general problem of the existence of pure stationary Nash equilibria for some special classes of mean payoff games.

Deterministic n-person shortest path and terminal games on symmetric digraphs have Nash equilibria in pure stationary strategies

We prove that a deterministic n-person shortest path game has a Nash equlibrium in pure and stationary strategies if it is edge-symmetric (that is (u, v) is a move whenever (v, u) is, apart from moves entering terminal vertices) and the length of every move is positive for each player. Both conditions are essential, though it remains an open problem whether there exists a NE-free 2-person non-edge-symmetric game with positive lengths. We provide examples for NE-free 2-person edge-symmetric games that are not positive. We also consider the special case of terminal games (shortest path games in which only terminal moves have nonzero length, possibly negative) and prove that edge-symmetric n-person terminal games always have Nash equilibria in pure and stationary strategies. Furthermore, we prove that an edge-symmetric 2-person terminal game has a uniform (subgame perfect) Nash equilibrium, provided any infinite play is worse than any of the terminals for both players.

Best-Response Dynamics for Evolutionary Stochastic Games

We introduce a version of best-response (BR) dynamics for stochastic games using the framework of evolutionary stochastic games proposed in [Flesch, J., Parthasarathy, T., Thuijsman, F. and Uyttendaele, P. [2013] Evolutionary stochastic games, Dyn. Games Appl. 3, 207–219]. This dynamics is defined on the simplex of all population distributions, as in classical evolutionary games. In this BR dynamics, it is shown that the set of all population distributions inducing Nash equilibria is globally asymptotically stable, for stable stochastic games. Also, if we transform the BR dynamics into the space of stationary strategies, the set of all Nash Equilibria is asymptotically stable. Another version of BR dynamics is defined in the space of stationary strategies for zero-sum stochastic games by [Leslie, D. S., Perkins, S. and Xu, Z. [2020] Best-response dynamics in zero-sum stochastic games, J. Econ. Theory 189, 105095]. To compare both versions of BR dynamics, we extend the evolutionary stochastic games model to asymmetric stochastic games and employ an equivalent definition of Nash equilibria. This comparison is illustrated using examples and it is observed that the trajectories of both versions of BR dynamics do converge to the same Nash equilibrium. The BR dynamics proposed in this paper has the advantage of being valid for nonzero sum stochastic games and conforms to both classical game dynamics and the dynamics of stationary strategies in stochastic games.

Zero-sum semi-Markov games with state-action-dependent discount factors

Semi-Markov model is one of the most general models for stochastic dynamic systems. This paper deals with a two-person zero-sum game for semi-Markov processes. We focus on the expected discounted criterion with state-action-dependent discount factors. The state and action spaces are both Polish spaces, and the reward rate function is ω-bounded. We first construct a fairly general model of semi-Markov games under a given semi-Markov kernel and a pair of strategies. Next, based on the standard regularity condition and the continuity-compactness condition for semi-Markov games, we derive a “drift condition” on the semi-Markov kernel and suppose that the discount factors have a positive lower bound, under which the existence of the value function and an optimal pair of stationary strategies of our semi-Markov game are proved by using a general Shapley equation. Moreover, in the scenario of finite state and action spaces, a value iteration-type algorithm for approximating the value function and an optimal pair of stationary strategies is developed. The convergence and the error bound of the algorithm are also proved. Finally, we conduct numerical examples to demonstrate the main results.

Spatial behavior of red deer (Cervus elaphus) in Northern Apennines: are we managing them correctly?

BackgroundThroughout their range, red deer are a well-studied species. In Italy, this species occupies two ecologically different ranges: the Alps and the Apennines. Although several studies have described the spatial behaviour of red deer in the Italian Alps, no data are available for the Apennine population.ResultsThe spatial behaviours of 13 deer from the Northern Apennines range were analysed for the first time using GPS-GSM telemetry from 2011 to 2017. Red deer displayed two coexisting strategies, i.e., migratory and stationary. In our sample, females tended to migrate more than males. We found a high level of interindividual variability in the date of migration/return, while each migratory deer was very conservative during the study period. The migration ranges were on average 12 ± 4.2 km from the resident range. Both migratory and resident deer displayed high site fidelity. No switch from the migratory to stationary strategy was observed for any deer during the study period; however, the period could have been too short to detect any switch. At the management level, over 18 management cycles occurred during the study period, and a spatial mismatch was found between deer range and management units (districts) in 38.9% (7) of the cases. Merging the districts belonging to each province to obtain an area of approximately 1000 km2 would partially address such spatial mismatch, reducing its occurrence to 22%.ConclusionsDespite the small sample size, these results can guide future management actions. However, an in-depth study with a larger sample size is required to better understand and manage the red deer Apennines population.

Equilibria in Pure Strategies for a Two-Player Zero-Sum Average Stochastic Positional Game

The problem of the existence and determining equilibria in pure stationary strategies for a two-player zero-sum average stochastic positional game is considered. We show that for such a game there exists the value and players may achieve the value by applying pure stationary strategies of choosing the actions in their positions. Based on a constructive proof of these results we propose an algorithmic approach for determining the optimal pure stationary strategies of the players.

Nonzero-Sum Risk-Sensitive Continuous-Time Stochastic Games with Ergodic Costs

We study nonzero-sum stochastic games for continuous time Markov decision processes on a denumerable state space with risk-sensitive ergodic cost criterion. Transition rates and cost rates are allowed to be unbounded. Under a Lyapunov type stability assumption, we show that the corresponding system of coupled HJB equations admits a solution which leads to the existence of a Nash equilibrium in stationary strategies. We establish this using an approach involving principal eigenvalues associated with the HJB equations. Furthermore, exploiting appropriate stochastic representation of principal eigenfunctions, we completely characterize Nash equilibria in the space of stationary Markov strategies.

Equilibrium in a Symmetric Game of Resource Extraction with Coalitional Structure

The game of resource extraction / capital accumulation is a stochastic nonzero-sum infinite horizon game, obtained as an extension of the well-known optimal growth model to m strategically competing players, who jointly posses a renewable resource. The existence of a Nash equilibrium in different, often symmetric, frameworks of the game received a significant attention in the scientific literature on the topic. The focus of this paper is to introduce the coalitional component to the symmetric problem. Specifically, we examine whether the game with a fixed coalitional structure admits stability against profitable coalitional deviations.It is assumed that the set of all players is partitioned into coalitions which do not intersect and remain consistent throughout the game. The members of each coalition are able to coordinate their actions and perform joint deviations in a cooperative manner. Such setting incorporates a natural concept of established social ties, which may reflect a potential context appearing in practical applications. The corresponding notion of equilibrium in the paper is expressed as a position, from which none of the set coalitions can deviate in a manner to increase a total reward of its members. Its existence is studied in the context of a certain symmetric resource extraction game model with unbounded utilities of the players. This model was studied in [12; 13], concluding a Stationary Markov Perfect Equilibrium existence in both symmetric and non-symmetric game structure. The first feature of the model is that the preferences of the players are considered to be isoelastic in the form of strictly concave power functions. Furthermore, the law of motion between states is set to follow a geometric random walk in relation to players' joint investments. We prove that the game within the formulated settings admits stability against profitable coalitional deviations for any partition on the set of agents. The method provides an algorithm for building the corresponding stationary strategies, which can be useful for practical purposes. Finally, we use two examples with different numerical configurations to illustrate possible patterns of how the individual rewards of the players vary depending on a coalitional structure, which is set at the beginning of the game.

An Interior-Point Differentiable Path-Following Method to Compute Stationary Equilibria in Stochastic Games

The subgame perfect equilibrium in stationary strategies (SSPE) is the most important solution concept in applications of stochastic games, making it imperative to develop efficient methods to compute an SSPE. For this purpose, this paper develops an interior-point differentiable path-following method (IPM), which establishes a connection between an artificial logarithmic barrier game and the stochastic game of interest by adding a homotopy variable. IPM brings several advantages over the existing methods for stochastic games. On the one hand, IPM provides a bridge between differentiable path-following methods and interior-point methods and remedies several issues of an existing homotopy method called the stochastic linear tracing procedure (SLTP). First, the starting stationary strategy profile can be arbitrarily chosen. Second, IPM does not need switching between different systems of equations. Third, the use of a perturbation term makes IPM applicable to all stochastic games rather than generic games only. Moreover, a well-chosen transformation of variables reduces the number of equations and variables by roughly one half. Numerical results show that the proposed method is more than three times as efficient as SLTP. On the other hand, the stochastic game can be reformulated as a mixed complementarity problem and solved by the PATH solver. We employ the proposed IPM and the PATH solver to compute SSPEs. Numerical results evince that for some stochastic games the PATH solver may fail to find an SSPE, whereas IPM is successful in doing so for all stochastic games, which confirms the reliability and stability of the proposed method. Summary of Contribution: This paper incorporates the interior-point methods into a differentiable path-following method for computing stationary equilibria for stochastic games. This novel method brings excellent computational advantages and remedies several issues with the existing methods for stochastic games. We prove the global convergence of the proposed method and employ this method to solve numerous randomly generated stochastic games with different scales. Numerical results further confirm the high efficiency, stability, and universality of this method for stochastic games.

Equilibrium in the problem of choosing the meeting time for N persons

A game-theoretic model of competitive decision on a meet time is considered. There are n players who are negotiating the meeting time. The objective is to find a meet time that satisfies all participants. The players' utilities are represented by linear unimodal functions ui(x), x∈ [0, 1], i=1,2,...,n. The maximum values of the utility functions are located at the points i/(n-1), i=0,...,n-1. Players take turns 1 → 2→ 3 → ...→ (n-1) → n → 1→... . Players can indefinitely insist on a profitable solution for themselves. To prevent this from happening, a discounting factor δ is greater than 1 is introduced to limit the duration of negotiations. We will assume that after each negotiation session, the utility functions of all players will decrease proportionally to δ. Thus, if the players have not come to a decision before time t, then at time t their utilities are represented by the functions δt-1ui(x), i = 1, 2,..., n. We will look for a solution in the class of stationary strategies, when it is assumed that the decisions of the players will not change during the negotiation time, i. e. the player i will make the same offer at step i and at subsequent steps n+i, 2n+i, ... . This will allow us to limit ourselves to considering the chain of sentences 1 → 2 → 3 → ... →(n-1) → n→ 1. We will use the method of backward induction. To do this, assume that player n is looking for his best responce, knowing player 1's proposal, then player (n-1) is looking for his best responce, knowing player n's solution, etc. In the end, we find the best responce of the player 1, and it should coincide with his offer at the beginning of the procedure. Thus, the reasoning in the method of backward induction has the form 1 ← 2← 3← ... ←(n-1)← n← 1. The subgame perfect equilibrium in the class of stationary strategies is found in analytical form. It is shown that when δ changes from 1 to 0, the optimal offer of player 1 changes from 1/2 to 1. That is, when the value of δ is close to 1, the players have a lot of time to negotiate, so the offer of player 1 should be fair to everyone. If the discounting factor is close to 0, the utilities of the players decreases rapidly and they must quickly make a decision that is beneficial to player 1.

Solution of the Meeting Time Choice Problem for n Persons

We consider a game-theoretic model of negotiations of n persons about a meeting time. The problem is to determine the time of the meeting, with the consensus of all players required to make a final decision. The solution is found by backward induction in the class of stationary strategies. Players' wins are represented by piecewise linear functions having one peak. An subgame perfect equilibrium for the problem in the case of δ ≤ 1/2 is found in analytical form.

RISK MANAGEMENT IN MAQISTRAL GAS AND OIL PIPELINE SYSTEMS WITH POTENTIAL COST CONSTRAINTS

The problem of managing technical and technological risks in main gas and oil pipeline systems, subject to the possibility of limited funds allocated for the prevention and elimination of the consequences of accidents, is considered in the form of a semi-Markov decision-making model for a controlled Markov process in continuous time with the criterion of the maximum average discounted income. To find the optimal nonrandomized Markov stationary strategy, a procedure is proposed based on reducing the formulated fuzzy problem to an equivalent Boolean programming problem with deterministic constraints. To solve the resulting system of inequality constraints, an algorithm has been developed for finding basic solutions for an arbitrary number of accidents and measures to eliminate them. The numerical implementation of the proposed approach is implemented for the real problem of risk management in the main gas pipeline with unclear cost constraints. Keywords: semi-Markov decision-making process, Markov stationary strategy, feasibility optimization, basic solutions to inequalities.

A differentiable path-following method to compute subgame perfect equilibria in stationary strategies in robust stochastic games and its applications

As an effective paradigm to address uncertainty in payoffs and transition probabilities, robust stochastic games have been formulated in the literature. This paper is concerned with the computation of subgame perfect equilibria in stationary strategies (SSPEs) in robust stochastic games. To tackle this problem, we develop in this paper a globally convergent differentiable path-following method by exploiting the structures of the games. Incorporating a logarithmic-barrier term into each player’s payoff function with an extra variable between zero and one, we constitute a logarithmic-barrier robust stochastic game in which each player solves in each state a convex optimization problem. An application of the optimality conditions to the barrier game together with a fixed-point argument yields a polynomial equilibrium system for the barrier game. As a result of this system, we establish the existence of a smooth path that starts from an arbitrary mixed strategy profile and ends at an SSPE as the extra variable descends from one to zero. As an alternative scheme, we make up a convex-quadratic-penalty robust stochastic game and attain a globally convergent convex-quadratic-penalty differentiable path-following method for SSPEs in robust stochastic games. Numerical comparisons show that the logarithmic-barrier path-following method significantly outperforms the convex-quadratic-penalty path-following method. To further evince the value of the proposed methods, we apply the logarithmic-barrier path-following method to solve a supply chain configuration problem and a market entry problem from medical waste recycling.

Costless delay in negotiations

We study bargaining models in discrete time with a finite number of players, stochastic selection of the proposing player, endogenously determined sets and orders of responders, and a finite set of feasible alternatives. The standard optimality conditions and system of recursive equations may not be sufficient for the existence of a subgame perfect equilibrium in stationary strategies (SSPE) in case of costless delay. We present a characterization of SSPE that is valid for both costly and costless delay. We address the relationship between an SSPE under costless delay and the limit of SSPEs under vanishing costly delay. An SSPE always exists when delay is costly, but not necessarily so under costless delay, even when mixed strategies are allowed for. This is surprising as a quasi SSPE, a solution to the optimality conditions and the system of recursive equations, always exists. The problem is caused by the potential singularity of the system of recursive equations, which is intimately related to the possibility of perpetual disagreement in the bargaining process.

Discrete stop-or-go games

Dubins and Savage (How to gamble if you must: inequalities for stochastic processes, McGraw-Hill, New York, 1965) found an optimal strategy for limsup gambling problems in which a player has at most two choices at every state x at most one of which could differ from the point mass delta (x). Their result is extended here to a family of two-person, zero-sum stochastic games in which each player is similarly restricted. For these games we show that player 1 always has a pure optimal stationary strategy and that player 2 has a pure epsilon -optimal stationary strategy for every epsilon > 0. However, player 2 has no optimal strategy in general. A generalization to n-person games is formulated and epsilon -equilibria are constructed.

Simple collective equilibria in stopping games

At each moment in time, an alternative from a finite set is selected by a stochastic process. Players observe the selected alternative and sequentially cast a yes or a no vote. If the set of players casting a yes vote is decisive for the selected alternative, it is accepted and the game ends. Otherwise the next period begins. We refer to this class of problems as stopping games. Collective choice games, quitting games, and coalition formation games are particular examples. When the core of a stopping game is non-empty, a subgame perfect equilibrium in pure stationary strategies is shown to exist. But in general, even subgame perfect equilibria in mixed stationary strategies may not exist. We show that aggregate voting behavior can be summarized by a collective strategy. We insist on pure strategies, allow for simple forms of punishment, and provide a constructive proof to show that so-called two-step simple collective equilibria always exist. This implies the existence of a pure strategy subgame perfect equilibrium. We apply our approach to the case with three alternatives exhibiting a Condorcet cycle and to a model of redistributive politics.

Trust and betrayals: Reputational payoffs and behaviors without commitment

I study a repeated game in which a patient player wants to win the trust of some myopic opponents, but can strictly benefit from betraying them. His benefit from betrayal is strictly positive and is his persistent private information. I characterize every type of patient player's highest equilibrium payoff and construct equilibria that attain this payoff. Since the patient player's Stackelberg action is mixed and motivating the lowest‐benefit type to play mixed actions is costly, every type's highest equilibrium payoff is strictly lower than his Stackelberg payoff. In every equilibrium where the patient player approximately attains his highest equilibrium payoff, no type of the patient player plays stationary strategies or completely mixed strategies.

On the Existence and Determining Stationary Nash Equilibria for Switching Controller Stochastic Games

In this paper we consider the problem of the existence and determining stationary Nash equilibria for switching controller stochastic games with discounted and average payoffs. The set of states and the set of actions in the considered games are assumed to be finite. For a switching controller stochastic game with discounted payoffs we show that all stationary equilibria can be found by using an auxiliary continuous noncooperative static game in normal form in which the payoffs are quasi-monotonic (quasi-convex and quasi-concave) with respect to the corresponding strategies of the players. Based on this we propose an approach for determining the optimal stationary strategies of the players. In the case of average payoffs for a switching controller stochastic game we also formulate an auxiliary noncooperative static game in normal form with quasi-monotonic payoffs and show that such a game possesses a Nash equilibrium if the corresponding switching controller stochastic game has a stationary Nash equilibrium.