Compact Action Sets Research Articles

Absorbing games with a clock and two bits of memory

An absorbing game is a two-person zero-sum repeated game. Some of the entries are “absorbing” in the sense that, following the play of an absorbing entry, with positive probability all future payoffs are equal to that entry's payoff. The outcome of the game is the long-run average payoff. We prove that a two-person zero-sum absorbing game, with either finite or compact action sets, has, for each ε>0, ε-optimal strategies with finite memory. In fact, we show that there is an ε-optimal strategy that depends on the clock and three states of memory.

Communicating zero-sum product stochastic games

We study two classes of zero-sum stochastic games with compact action sets and a finite product state space. These two classes assume a communication property on the state spaces of the players. For strongly communicating on one side games, we prove the existence of the uniform value. For weakly communicating on both sides games, we prove that the asymptotic value, and therefore the uniform value, may fail to exist.

The Vanishing Discount Approach in a class of Zero-Sum Finite Games with Risk-Sensitive Average Criterion

This work concerns zero-sum stochastic games with finite state space and compact action sets. The game is driven by two players and it is assumed that player 1 has a nonull and constant risk-sensitivity coefficient, so that a random cost is assessed via an exponential disutility function. The main objective of the paper is to show that as the discount factor increases to 1, an appropriate normalization of the risk-sensitive discounted value function converges to the risk-sensitive average value function. Besides standard continuity-compactness conditions, that result is obtained assuming that (i) the state space is communicating under the action of any pair of stationary deterministic policies and (ii) a minorization property holds at some state. Additionally, it is shown that, if as in the classical risk-neutral context, the risk-sensitive discounted value function is multiplied by 1 minus the discount factor, then the resulting normalization converges to the arithmetic mean of the risk-sensitive average value over the interval joining 0 and the risk-sensitivity coefficient.

Interactive epistemology in simple dynamic games with a continuum of strategies

We extend the epistemic analysis of dynamic games of Battigalli and Siniscalchi (J Econ Theory 88:188–230, 1999, J Econ Theory 106:356–391, 2002, Res Econ 61:165–184, 2007) from finite dynamic games to all simple games, that is, finite and infinite-horizon multistage games with finite action sets at nonterminal stages and compact action sets at terminal stages. We prove a generalization of Lubin’s (Proc Am Math Soc 43:118–122, 1974) extension result to deal with conditional probability systems and strong belief. With this, we can provide a short proof of the following result: in every simple dynamic game, strong rationalizability characterizes the behavioral implications of rationality and common strong belief in rationality.

Nash equilibrium in games with quasi-monotonic best-responses

This paper proposes a new general class of strategic games and develops an associated new existence result for pure-strategy Nash equilibrium. For a two-player game with scalar and compact action sets, existence entails that one reaction curve be increasing and continuous and the other quasi-increasing (i.e., not have any downward jumps). The latter property amounts to strategic quasi-complementarities. The paper provides a number of ancillary results of independent interest, including sufficient conditions for a quasi-increasing argmax (or non-monotone comparative statics), and new sufficient conditions for uniqueness of fixed points. For maximal accessibility of the results, the main results are presented in a Euclidean setting. We argue that all these results have broad and elementary applicability by providing simple illustrations with commonly used models in economic dynamics and industrial organization.

Uniform value for recursive games with compact action sets

Mertens, Neyman and Rosenberg (Mertens et al., 2009) used the Mertens and Neyman theorem (Mertens and Neyman, 1981) to prove the existence of a uniform value for absorbing games with a finite state space and compact action sets. We provide an analogous proof for another class of stochastic games, recursive games with a finite state space and compact action sets. Moreover, both players have stationary ε-optimal strategies.

Conditions for the Solvability of the Linear Programming Formulation for Constrained Discounted Markov Decision Processes

We consider a discrete-time constrained discounted Markov decision process (MDP) with Borel state and action spaces, compact action sets, and lower semi-continuous cost functions. We introduce a set of hypotheses related to a positive weight function which allow us to consider cost functions that might not be bounded below by a constant, and which imply the solvability of the linear programming formulation of the constrained MDP. In particular, we establish the existence of a constrained optimal stationary policy. Our results are illustrated with an application to a fishery management problem.

Sample-Path Optimal Stationary Policies in Stable Markov Decision Chains with the Average Reward Criterion

This paper concerns discrete-time Markov decision chains with denumerable state and compact action sets. Besides standard continuity requirements, the main assumption on the model is that it admits a Lyapunov function ℓ. In this context the average reward criterion is analyzed from the sample-path point of view. The main conclusion is that if the expected average reward associated to ℓ2 is finite under any policy then a stationary policy obtained from the optimality equation in the standard way is sample-path average optimal in a strong sense.

Sample-Path Optimal Stationary Policies in Stable Markov Decision Chains with the Average Reward Criterion

This paper concerns discrete-time Markov decision chains with denumerable state and compact action sets. Besides standard continuity requirements, the main assumption on the model is that it admits a Lyapunov function ℓ. In this context the average reward criterion is analyzed from the sample-path point of view. The main conclusion is that if the expected average reward associated to ℓ2is finite under any policy then a stationary policy obtained from the optimality equation in the standard way is sample-path average optimal in a strong sense.

A Zero-Sum Stochastic Game with Compact Action Sets and no Asymptotic Value

We give an example of a zero-sum stochastic game with four states, compact action sets for each player, and continuous payoff and transition functions, such that the discounted value does not converge as the discount factor tends to 0, and the value of the n-stage game does not converge as n goes to infinity.

Discounted Approximations for Risk-Sensitive Average Criteria in Markov Decision Chains with Finite State Space

This work concerns Markov decision processes with finite state space and compact action set. The performance of a control policy is measured by a risk-sensitive average cost criterion and, under standard continuity-compactness conditions, it is shown that the discounted approximations converge to the optimal value function, and that the superior and inferior limit average criteria have the same optimal value function. These conclusions are obtained for every nonnull risk-sensitivity coefficient, and regardless of the communication structure induced by the transition law.

Absorbing Games with Compact Action Spaces

We prove that games with absorbing states with compact action sets have a value.

Optimization of a special case of continuous-time Markov decision processes with compact action set

Performance optimization is considered for average-cost multichain Markov decision processes (MDPs) with compact action set. Since, for a general compact multichain model, the optimality equation system may have no solution, and also a policy iteration algorithm may yield a suboptimal policy rather than an optimal one, we concentrate only on a special case of multichain models in this paper, where we assume that the classifications of states are fixed identically rather than varying with policies. By using the concept of performance potentials, the existence of solutions to the optimality equation system is established, and then a potential-based policy iteration algorithm is supposed to solve this system. In addition, the optimality convergence, for recurrent classes, of the algorithm has been proved. Finally, a numerical example is provided.

Uniqueness and Stability of Optimal Policies of Finite State Markov Decision Processes

In this paper we consider infinite horizon discrete-time optimal control of Markov decision processes (MDPs) with finite state spaces and compact action sets. We restrict attention to unicost MDPs, which form a class that contains all the weakly communicating MDPs. The unicost MDPs are characterized as those MDPs for which there exists a solution to the single optimality equation. We address the problem of uniqueness and stability of minimizing Markov actions. Our main result asserts that when we endow the set of unicost MDPs with a certain natural metric, under which it is complete, then the class of MDPs with essentially unique and stable minimizing Markov actions contains the intersection of countably many open dense sets (hence is itself dense). Thus, the property of having essentially unique and stable minimizing Markov actions is generic for unicost MDPs.

Nonstationary value iteration in controlled Markov chains with risk-sensitive average criterion

This work concerns Markov decision chains with finite state spaces and compact action sets. The performance index is the long-run risk-sensitive average cost criterion, and it is assumed that, under each stationary policy, the state space is a communicating class and that the cost function and the transition law depend continuously on the action. These latter data are not directly available to the decision-maker, but convergent approximations are known or are more easily computed. In this context, the nonstationary value iteration algorithm is used to approximate the solution of the optimality equation, and to obtain a nearly optimal stationary policy.

Nonstationary value iteration in controlled Markov chains with risk-sensitive average criterion

This work concerns Markov decision chains with finite state spaces and compact action sets. The performance index is the long-run risk-sensitive average cost criterion, and it is assumed that, under each stationary policy, the state space is a communicating class and that the cost function and the transition law depend continuously on the action. These latter data are not directly available to the decision-maker, but convergent approximations are known or are more easily computed. In this context, the nonstationary value iteration algorithm is used to approximate the solution of the optimality equation, and to obtain a nearly optimal stationary policy.

Performance optimization algorithms based on potentials for semi-Markov control processes

Optimization algorithms are studied for a class of semi-Markov control processes (SMCPs) with compact action set. Both the long-run average and discounted cost problems are considered. Formulas of performance potentials and optimality equations for SMCPs are derived. A policy iteration algorithm and a value iteration algorithm are proposed, which can lead to an optimal or suboptimal stationary policy in a finite number of iterations. The convergence of these algorithms is established, without the assumption of the corresponding iteration operator being a span-contraction. In addition, an online policy iteration optimization algorithm based on a single sample path is provided to escape ‘curse of dimensionality’. In the end, a numerical example is provided to illustrate the application of the algorithms.

Efficient Contract Design in Multi-Principal Multi-Agent Supply Chains

We consider a general multi-principal multi-agent contracting game in a complete-information supply-chain setting and determine coordinating equilibrium transfer schedules in closed form. The resulting contracts manage to align incentives for decentralized decision-making and achieve first-best channel solutions. We allow for multidimensional actions and arbitrary payoff externalities between all members of the supply chain. For the coordinating contracts to exist it suffices that all payoff functions are continuous on the compact action sets in a general sense that accommodates discrete action sets. Our approach unifies and generalizes a significant portion of the extant supply-chain literature. It can be applied to a very large class of many-to-many supply-chain settings.

An Algorithm to Identify and Compute Average Optimal Policies in Multichain Markov Decision Processes

This paper concerns discrete-time, finite state multichain MDPs with compact action sets. The optimality criterion is long-run average cost. Simple examples illustrate that optimal stationary Markov policies do not always exist. We establish the existence of ε-optimal policies that are stationary Markovian, and develop an algorithm that computes these approximate optimal policies. We establish a necessary and sufficient condition for the existence of an optimal policy that is stationary Markovian, and in case that such an optimal policy exists the algorithm computes it.

Performance optimization of continuous-time Markov control processes based on performance potentials

Average-cost optimization problems for a class of continuous-time Markov control processes with a compact action set have been studied. The definition of a generalized average-cost Poisson equation, which can be viewed as an extension to the standard one is first given. Markov performance potentials are defined as its unique solution. Based on the formula of performance potentials, an average-cost optimality equation is derived and the existence of its solution is established. Then, policy iteration and value iteration algorithms are proposed and their convergence discussed. A numerical example for controlled closed queuing networks illustrates the application of the proposed value iteration algorithm.