Borel State Spaces Research Articles

On a new class of nonzero-sum discounted stochastic games having stationary Nash equilibrium points

A new class of nonzero-sum Borel state space discounted stochastic games having stationary Nash equilibria is presented. Some applications to economic theory are also included.

The critical discount factor for finite Markovian decision processes with an absorbing set

This paper deals with a Markovian decision process with an absorbing set J0. We are interested in the largest number β*≥1, called the critical discount factor, such that for all discount factors β smaller than β* the limit V of the N-stage value function VN for N →∞ exists and is finite for each choice of the one-stage reward function. Several representations of β* are given. The equality of 1/β* with the maximal Perron/Frobenius eigenvalue of the MDP links our problem and our results to topics studied intensively (mostly for β=1) in the literature. We derive in a unified way a large number of conditions, some of which are known, which are equivalent either to β 1. In particular, the latter is equivalent to transience of the MDP. A few of our findings are extended with the aid of results in Rieder (1976) to models with standard Borel state and action space. We also complement an algorithm of policy iteration type, due to Mandl/Seneta (1969), for the computation of β*. Finally we determine β* explicitly in two models with stochastically monotone transition law.

Zero-Sum Average Semi-Markov Games: Fixed-Point Solutions of the Shapley Equation

This paper deals with zero-sum average semi-Markov games with Borel state and action spaces, unbounded payoffs, and mean holding times. A solution to the Shapley equation is obtained via the Banach fixed-point theorem assuming that the model satisfies a Lyapunov-like condition, a growth hypothesis on the payoff function, and the mean holding times, besides standard continuity and compactness requirements.

Average optimal switching of a Markov chain with a Borel state space

We extend results on average per unit time optimality criterion in a switching model from a countable state space to a Borel state space. In the model we consider, a controller selects an increasing sequence of stopping times with respect to a Markov chain, and gets rewards and pays costs at them in an alternating order. The rewards and costs depend on the state of the chain. We find the optimal average gain and construct an optimal strategy. The basic tool is a variational problem with two obstacles that appears also in Dynkin games.

Zero-Sum Semi-Markov Games

This paper deals with Borel state and action spaces zero-sum semi-Markov games under the expected long run average payoff criterion. The transition probabilities are assumed to satisfy some generalized geometric ergodicity conditions. The main result states that the optimality equation has a solution, which is approximated by the solutions of some $\varepsilon$-perturbed semi-Markov games. As a corollary, the existence of value and average optimal strategies for the players is established.

Adaptive policies for time-varying stochastic systems under discounted criterion

We consider a class of time-varying stochastic control systems, with Borel state and action spaces, and possibly unbounded costs. The processes evolve according to a discrete-time equation xn + 1=Gn (xn , an , ξn), n=0, 1, … , where the ξn are i.i.d. ℜk-valued random vectors whose common density is unknown, and the Gn are given functions converging, in a restricted way, to some function G∞ as n→∞. Assuming observability of ξn, we construct an adaptive policy which is asymptotically discounted cost optimal for the limiting control systemxn+1=G∞ (xn , an , ξn).

On the optimality equation for zero-sum ergodic stochastic games

We consider zero-sum stochastic games with Borel state spaces satisfying a generalized geometric ergodicity condition. We prove under fairly general assumptions that the optimality equation has a solution which is unique up to an additive constant.

An approximation approach to ergodic semi-Markov control processes

We consider semi-Markov control models (SMCMs) with a Borel state space satisfying certain stochastic stability assumptions on the transition structure which imply the so-called V-uniform geometric ergodicity of the state process. We deal with a class of e-perturbations of transition probability functions of the original model. First, we determine the rate of convergence of the optimal expected costs in in perturbed models to the optimal expected cost in the orginal SMCM. Next, we present a new algorithm for finding the solution to the average cost optimality equation (ACOE). The algorithm makes use of a sequence of solutions to the ACOE for the perturbed models, which can be found by a simple iterative procedure.

Limiting Average Criteria For Nonstationary Markov Decision Processes

This paper deals with the so-called limiting average criteria for nonstationary Markov decision processes with (possibly unbounded) rewards and Borel state space. A new set of conditions is provided, under which the existence of both a solution to the optimality equations and the limiting average $\epsilon (\geq 0)$-optimal Markov policies is derived. Also, a rolling horizon algorithm for computing limiting average $\epsilon (>0)$-optimal Markov policies is developed. Furthermore, the results in this paper are illustrated by several examples such as the water regulation problem.

Nonhomogeneous Markov Decision Processes with Borel State Space—The Average Criterion with Nonuniformly Bounded Rewards

This paper deals with nonhomogeneous Markov decision processes with Borel state space and nonuniformly bounded rewards under the average criterion. First, under the minorant-type assumption we prove the existence of an appropriate solution to the optimality equations. Second, from the optimality equations we also establish the existence of ε(≥0)-optimal Markov policies under the additional conditions. Third, some sufficient conditions for the validity of the assumptions in this paper and several examples such as inventory/production systems are provided. Finally, as an application of the optimality equations, a rolling horizon algorithm is given.

Blackwell optimality in the class of all policies in Markov decision chains with a Borel state space and unbounded rewards

This paper is the second part of our study of Blackwell optimal policies in Markov decision chains with a Borel state space and unbounded rewards. We prove that a stationary policy is Blackwell optimal in the class of all history-dependent policies if it is Blackwell optimal in the class of stationary policies.

Optimal strategies in a class of zero-sum ergodic stochastic games

In this paper we study zero-sum stochastic games with Borel state spaces. We make some stochastic stability assumptions on the transition structure of the game which imply the so-called w-uniform geometric ergodicity of Markov chains induced by stationary strategies of the players. Under such assumptions and some regularity conditions on the primitive data, we prove the existence of optimal stationary strategies for the players in the expected average payoff stochastic games. We also provide a first result on overtaking optimality in zero-sum stochastic games.

Nonparametric adaptive control for discrete-time Markov processes with unbounded costs under average criterion

We introduce average cost optimal adaptive policies in a class of discrete-time Markov control processes with Borel state and action spaces, allowing unbounded costs. The processes evolve according to the system equations $x_{t+1}=F(x_t,a_t,ξ _t)$, t=1,2,

Sample-path average optimality for Markov control processes

The authors consider a Markov control process with Borel state and actions spaces, unbounded costs, and under the long-run sample-path average cost criterion. They prove that under very weak assumptions on the transition law and a moment assumption for the one-step cost, there exists a stationary policy with invariant probability distribution v, that is sample-path average cost optimal for v-almost all initial states. In addition, every expected average-cost optimal stationary policy is in fact (liminf) sample-path average-cost optimal and strongly expected average-cost optimal.

Adaptive control for discrete-time Markov processes with unbounded costs: Average criterion

The paper deals with a class of discrete-time Markov control processes with Borel state and action spaces, and possibly unbounded one-stage costs. The processes are given by recurrent equations x t +1=F(x t ,a t ,ξ t ), t=1,2,… with i.i.d. ℜ k – valued random vectors ξ t whose density ρ is unknown. Assuming observability of ξ t , and taking advantage of the procedure of statistical estimation of ρ used in a previous work by authors, we construct an average cost optimal adaptive policy.

Blackwell Optimality in Borelian Continuous-in-Action Markov Decision Processes

We prove the existence of stationary Blackwell optimal policies in Markov decision processes with a Borel state space, compact action sets, and continuous-in-action and bounded transition densities and rewards, satisfying a simultaneous Doeblin-type condition. The proof is based on a compactification of the randomized stationary policy space in a weak-strong topology, on the continuity of Laurent coefficients of the discounted rewards in this topology, and on a lexicographical policy improvement. Until now similar results were obtained for the models with a denumerable state space or with a Borel state space and finite action sets.

Approximation of average cost optimal policies for general Markov decision processes with unbounded costs

The aim of the paper is to show that Lyapunov-like ergodicity conditions on Markov decision processes with Borel state space and possibly unbounded cost provide the approximation of an average cost optimal policy by solvingn-stage optimization problems (n = 1, 2, ...). The used approach ensures the exponential rate of convergence. The approximation of this type would be useful to find adaptive procedures of control and to estimate stability of an optimal control under disturbances of the transition probability.

The Compactness of a Policy Space in Dynamic Programming Via an Extension Theorem for Carathéodory Functions

The background for this paper is a dynamic programming model with a Borel state space and compact action sets. A new simple proof of the compactness of a space of measures corresponding to randomized stationary policies is given, in a topology induced by Carathéodory functions on the state-action space. For this purpose an extension theorem for Carathéodory functions is proved, which is a new version of the Tietze's extension theorem for continuous functions.

Approximations of dynamic Nash games with general state and action spaces and ergodic costs for the players

The purpose of this paper is to prove existence of an e-equilibrium point in a dynamic Nash game with Borel state space and long-run time average cost criteria for the players. The idea of the proof is first to convert the initial game with ergodic costs to an “equivalent” game endowed with discounted costs for some appropriately chosen value of the discount factor, and then to approximate the discounted Nash game obtained in the first step with a countable state space game for which existence of a Nash equilibrium can be established. From the results of Whitt we know that if for any e > 0 the approximation scheme is selected in an appropriate way, then Nash equilibrium strategies for the approximating game are also eequilibrium strategies for the discounted game constructed in the first step. It is then shown that these strategies constitute an e-equilibrium point for the initial game with ergodic costs as well. The idea of canonical triples, introduced by Dynkin and Yushkevich in the control setting, is adapted here to the game situation.

Discounted semi-markov decision process in a semi-markov environment

This paper presents the discounted semi-Markov decision process (SMDP) with Borel state space in a semi-Markov environment. It describes a system which behaves like a SMDP except that the system is influenced by its semi-Markov process environment. Following each state transition of the environment, the parameters of the SMDP changes. After present me the model, we show the validity of the optimality equation and the existence of ε-optimal policies