Optimal Stationary Strategies Research Articles

A Note on Zero-Sum Two-Person Undiscounted One Player Control Semi-Markov Games

Zero-sum two-person finite undiscounted (limiting ratio average) semi-Markov games are considered where both the transition probabilities and the transition times are controlled by a fixed player in all states. We prove that, if such a game is unichain, the value and optimal stationary strategies of the players can be obtained from an optimal solution of the linear programming algorithm for the associated one player control stochastic game using a data transformation method. An example is provided for such class of games.

Optimal Per-Loss Reinsurance for a Risk Model with a Thinning-Dependence Structure

In this paper, we consider the optimal reinsurance problem for a risk model with a thinning-dependence structure, where the stochastic sources related to claim occurrence are classified into different groups, and each group may cause a claim in each insurance class with some probability. We assume that the insurer can manage the risk by purchasing per-loss reinsurance, and their aim is to maximize the expected utility of the terminal wealth. By using the technique of stochastic control, we obtain the corresponding Hamilton–Jaccobi–Bellman equation. From the perspective of game theory, we derive the closed-form expression of the optimal strategy for each class of business, which is actually the best response to other given strategies. We also investigate the necessary conditions for optimal strategies and transfer the original optimization problem into a system of equations. Furthermore, we prove that the solution of the system of equations always exists, but may not be unique, and we also study some features of the optimal strategies in special cases and derive several interesting results. Finally, some numerical examples are given to show the impacts of some important parameters on the optimal strategies.

Constrained Discounted Stochastic Games

In this paper, we consider a large class of constrained non-cooperative stochastic Markov games with countable state spaces and discounted cost criteria. In one-player case, i.e., constrained discounted Markov decision models, it is possible to formulate a static optimisation problem whose solution determines a stationary optimal strategy (alias control or policy) in the dynamical infinite horizon model. This solution lies in the compact convex set of all occupation measures induced by strategies, defined on the set of state–action pairs. In case of n-person discounted games the occupation measures are induced by strategies of all players. Therefore, it is difficult to generalise the approach for constrained discounted Markov decision processes directly. It is not clear how to define the domain for the best-response correspondence whose fixed point induces a stationary equilibrium in the Markov game. This domain should be the Cartesian product of compact convex sets in a locally convex topological vector space. One of our main results shows how to overcome this difficulty and define a constrained non-cooperative static game whose Nash equilibrium induces a stationary Nash equilibrium in the Markov game. This is done for games with bounded cost functions and positive initial state distribution. An extension to a class of Markov games with unbounded costs and arbitrary initial state distribution relies on an approximation of the unbounded game by bounded ones with positive initial state distributions. In the unbounded case, we assume the uniform integrability of the discounted costs with respect to all probability measures induced by strategies of the players, defined on the space of plays (histories) of the game. Our assumptions are weaker than those applied in earlier works on discounted dynamic programming or stochastic games using the so-called weighted norm approach.

Optimal control of probabilistic Boolean control networks: A scalable infinite horizon approach

AbstractOne of the major issues in systems biology is developing control theory for gene regulatory networks (GRNs). Particularly, an important objective is to develop therapeutic intervention strategies to alter regulatory dynamics of GRNs to avoid undesired or diseased states. Several optimal control strategies have been developed to find therapeutic intervention strategies for small (or medium) sized GRNs modeled as probabilistic Boolean control networks (PBCNs). However, due to the humongous nature of GRNs, we require an optimal control strategy that scales to large systems without posing any constraints on the network dynamics. In this article, we formulate the infinite horizon discounted cost optimal control problem by leveraging the Markov decision process (MDP) based PBCN structure to model GRNs. Further, we design a stationary optimal control strategy, thereby avoiding the states associated with a disease. An augmented state space is defined to facilitate the control design method. By exploiting Kullback–Leibler (KL) divergence and the log transformation of the cost function referred to as the desirability function, the optimal control is devised using the path integral (PI) approach. We propose a sampling‐based technique for the approximation of the PI and hence the optimal control of PBCNs. The sampling‐based strategy is amicable to parallel implementation, thereby addressing the optimal control problem of large‐scale PBCNs. Finally, the viability of the results in the article is shown by some illustrative examples.

RISK MANAGEMENT IN MAQISTRAL GAS AND OIL PIPELINE SYSTEMS WITH POTENTIAL COST CONSTRAINTS

The problem of managing technical and technological risks in main gas and oil pipeline systems, subject to the possibility of limited funds allocated for the prevention and elimination of the consequences of accidents, is considered in the form of a semi-Markov decision-making model for a controlled Markov process in continuous time with the criterion of the maximum average discounted income. To find the optimal nonrandomized Markov stationary strategy, a procedure is proposed based on reducing the formulated fuzzy problem to an equivalent Boolean programming problem with deterministic constraints. To solve the resulting system of inequality constraints, an algorithm has been developed for finding basic solutions for an arbitrary number of accidents and measures to eliminate them. The numerical implementation of the proposed approach is implemented for the real problem of risk management in the main gas pipeline with unclear cost constraints. Keywords: semi-Markov decision-making process, Markov stationary strategy, feasibility optimization, basic solutions to inequalities.

Discrete stop-or-go games

Dubins and Savage (How to gamble if you must: inequalities for stochastic processes, McGraw-Hill, New York, 1965) found an optimal strategy for limsup gambling problems in which a player has at most two choices at every state x at most one of which could differ from the point mass delta (x). Their result is extended here to a family of two-person, zero-sum stochastic games in which each player is similarly restricted. For these games we show that player 1 always has a pure optimal stationary strategy and that player 2 has a pure epsilon -optimal stationary strategy for every epsilon > 0. However, player 2 has no optimal strategy in general. A generalization to n-person games is formulated and epsilon -equilibria are constructed.

On the Existence and Determining Stationary Nash Equilibria for Switching Controller Stochastic Games

In this paper we consider the problem of the existence and determining stationary Nash equilibria for switching controller stochastic games with discounted and average payoffs. The set of states and the set of actions in the considered games are assumed to be finite. For a switching controller stochastic game with discounted payoffs we show that all stationary equilibria can be found by using an auxiliary continuous noncooperative static game in normal form in which the payoffs are quasi-monotonic (quasi-convex and quasi-concave) with respect to the corresponding strategies of the players. Based on this we propose an approach for determining the optimal stationary strategies of the players. In the case of average payoffs for a switching controller stochastic game we also formulate an auxiliary noncooperative static game in normal form with quasi-monotonic payoffs and show that such a game possesses a Nash equilibrium if the corresponding switching controller stochastic game has a stationary Nash equilibrium.

Linear programming approach to optimal impulse control problems with functional constraints

This paper considers an optimal impulse control problem of dynamical systems generated by a flow. The performance criteria are total costs over the infinite time horizon. Apart from the main performance to be minimized, there are multiple constraints on performance functionals of a similar type. Under a natural set of compactness-continuity conditions on the system primitives, we establish a linear programming approach, and prove the existence of a stationary optimal control strategy out of a more general class of randomized strategies. This is done by making use of the tools from Markov decision processes.

On the existence of stationary Nash equilibria in average stochastic games with finite state and action spaces

We consider infinite n-person stochastic games with limiting average payoffs criteria for the players. The main results of the paper are concerned with the existence of stationary Nash equilibria and determining the optimal strategies of the players in the games with finite state and action spaces. We present conditions for the existence of stationary Nash equilibria in the considered games and propose an approach for determining the optimal stationary strategies of the players if such strategies exist.

Defensive deception against reactive jamming attacks in remote state estimation

This paper considers a synthetic counter-measure, combining transmission scheduling and defensive deception, to defend against jamming attacks in remote state estimation. In the setup studied, an attacker sabotages packet transmissions from a sensor to a remote estimator by congesting the communication channel between them. In order to efficiently degrade the estimation accuracy, the intelligent attacker tailors its jamming strategy by reacting to the real-time information it collects. In response to the jamming attacks, the sensor with a long-term goal will select the transmission power level at each stage. In addition, by modifying the real-time information intentionally, the sensor creates asymmetric uncertainty to mislead the attacker and thus mitigate attacks. Considering the dynamic nature of the process, we model the strategic interaction between the sensor and the attacker by a general stochastic game with asymmetric information structure. To obtain stationary optimal strategies for each player, we convert this game into a belief-based dynamic game and analyze the existence of its optimal solution. For a tractable implementation, we present an algorithm that finds equilibrium strategies based on multi-agent reinforcement learning for symmetric-information stochastic games. Numerical examples illustrate properties of the proposed algorithm.

Markov games with unknown random state‐actions‐dependent discount factors: Empirical estimation

AbstractThe work deals with a class of discrete‐time zero‐sum Markov games under a discounted optimality criterion with random state‐action‐dependent discount factors of the form , where xn,an,bn, and ξn+1 are the state, the actions of players, and a random disturbance at time n, respectively, taking values in Borel spaces. The one‐stage payoff is assumed to be possibly unbounded. In addition, the process {ξn} is formed by observable, independent, and identically distributed random variables with common distribution θ, which is unknown to players. By using the empirical distribution to estimate θ, we introduce a procedure to approximate the value V∗ of the game; such a procedure yields construction schemes of stationary optimal strategies and asymptotically optimal Markov strategies.

An accretive operator approach to ergodic zero-sum stochastic games

We study some ergodicity property of zero-sum stochastic games with a finite state space and possibly unbounded payoffs. We formulate this property in operator-theoretical terms, involving the solvability of an optimality equation for the Shapley operators (i.e., the dynamic programming operators) of a family of perturbed games. The solvability of this equation entails the existence of the uniform value, and its solutions yield uniform optimal stationary strategies. We first provide an analytical characterization of this ergodicity property, and address the generic uniqueness, up to an additive constant, of the solutions of the optimality equation. Our analysis relies on the theory of accretive mappings, which we apply to maps of the form $Id - T$ where $T$ is nonexpansive. Then, we use the results of a companion work to characterize the ergodicity of stochastic games by a geometrical condition imposed on the transition probabilities. This condition generalizes classical notion of ergodicity for finite Markov chains and Markov decision processes.

Supply Chain Management Local Model for Unreliable Suppliers

A supply chain management mathematical model for an isolated warehouse with two suppliers of different reliability is considered. Optimal inventory control strategy synthesis algorithm is constructed. The transfer to stationary strategies is investigated. We developed C++ software package to simulate algorithm for two different probabilistic distributions of demand: discrete Gauss-like and another one, discrete too, close to uniform. These examples are used to investigate the optimal stationary strategy parameters and costs dependency of econometrical attributes (among these–discount factor).

An epistemic approach to stochastic games

In this paper we focus on stochastic games with finitely many states and actions. For this setting we study the epistemic concept of common belief in future rationality, which is based on the condition that players always believe that their opponents will choose rationally in the future. We distinguish two different versions of the concept—one for the discounted case with a fixed discount factor delta , and one for the case of uniform optimality, where optimality is required for all discount factors close enough to 1” . We show that both versions of common belief in future rationality are always possible in every stochastic game, and always allow for stationary optimal strategies. That is, for both versions we can always find belief hierarchies that express common belief in future rationality, and that have stationary optimal strategies. We also provide an epistemic characterization of subgame perfect equilibrium for two-player stochastic games, showing that it is equivalent to mutual belief in future rationality together with some “correct beliefs assumption”.

Characterization and simplification of optimal strategies in positive stochastic games

Abstract We consider positive zero-sum stochastic games with countable state and action spaces. For each player, we provide a characterization of those strategies that are optimal in every subgame. These characterizations are used to prove two simplification results. We show that if player 2 has an optimal strategy then he/she also has a stationary optimal strategy, and prove the same for player 1 under the assumption that the state space and player 2's action space are finite.

Simplifying optimal strategies in limsup and liminf stochastic games

We consider two-player zero-sum stochastic games with the limsup and with the liminf payoffs. For the limsup payoff, we prove that the existence of an optimal strategy implies the existence of a stationary optimal strategy. Our construction does not require the knowledge of an optimal strategy, only its existence. The main technique of the proof is to analyze the game with specific restricted action spaces. For the liminf payoff, we prove that the existence of a subgame-optimal strategy (i.e. a strategy that is optimal in every subgame) implies the existence of a subgame-optimal strategy under which the prescribed mixed actions only depend on the current state and on the state and the actions chosen at the previous period. In particular, such a strategy requires only finite memory. The proof relies on techniques that originate in gambling theory.

Markov decision processes and stochastic games with total effective payoff

We consider finite Markov decision processes with undiscounted total effective payoff. We show that there exist uniformly optimal pure and stationary strategies that can be computed by solving a polynomial number of linear programs. This implies that in a two-player zero-sum stochastic game with perfect information and with total effective payoff there exists a stationary best response to any stationary strategy of the opponent. From this, we derive the existence of a uniformly optimal pure and stationary saddle point. Finally we show that mean payoff can be viewed as a special case of total payoff.

Dynamic Risk Control in Multidimensional Markov Models

The authors consider a typical model used in many fields of applied mathematics regarding selection of optimal decisions in stochastic systems. The multidimensional death process controlled on an interval of random length, which is analyzed in the paper, can be used to describe phenomena that appear in economical, biological, medical, engineering, and other applications. Several approaches are proposed to determine the optimality criteria, the properties of corresponding risk function are investigated, and a computing algorithm is developed for construction of optimal stationary strategies.

On zero-sum two-person undiscounted semi-Markov games with a multichain structure

Abstract Zero-sum two-person finite undiscounted (limiting ratio average) semi-Markov games (SMGs) are considered with a general multichain structure. We derive the strategy evaluation equations for stationary strategies of the players. A relation between the payoff in the multichain SMG and that in the associated stochastic game (SG) obtained by a data-transformation is established. We prove that the multichain optimality equations (OEs) for an SMG have a solution if and only if the associated SG has optimal stationary strategies. Though the solution of the OEs may not be optimal for an SMG, we establish the significance of studying the OEs for a multichain SMG. We provide a nice example of SMGs in which one player has no optimal strategy in the stationary class but has an optimal semistationary strategy (that depends only on the initial and current state of the game). For an SMG with absorbing states, we prove that solutions in the game where all players are restricted to semistationary strategies are solutions for the unrestricted game. Finally, we prove the existence of stationary optimal strategies for unichain SMGs and conclude that the unichain condition is equivalent to require that the game satisfies some recurrence/ergodicity/weakly communicating conditions.

Generic uniqueness of the bias vector of finite zero-sum stochastic games with perfect information

Mean-payoff zero-sum stochastic games can be studied by means of a nonlinear spectral problem. When the state space is finite, the latter consists in finding an eigenpair (u,λ) solution of T(u)=λe+u, where T:Rn→Rn is the Shapley (or dynamic programming) operator, λ is a scalar, e is the unit vector, and u∈Rn. The scalar λ yields the mean payoff per time unit, and the vector u, called bias, allows one to determine optimal stationary strategies in the mean-payoff game. The existence of the eigenpair (u,λ) is generally related to ergodicity conditions. A basic issue is to understand for which classes of games the bias vector is unique (up to an additive constant). In this paper, we consider perfect-information zero-sum stochastic games with finite state and action spaces, thinking of the transition payments as variable parameters, transition probabilities being fixed. We show that the bias vector, thought of as a function of the transition payments, is generically unique (up to an additive constant). The proof uses techniques of nonlinear Perron–Frobenius theory. As an application of our results, we obtain an explicit perturbation scheme allowing one to solve degenerate instances of stochastic games by policy iteration.