Policies For Markov Decision Processes Research Articles

A Note on the Existence of Optimal Policies in Total Reward Dynamic Programs with Compact Action Sets

This work deals with Markov decision processes (MDPs) with expected total rewards, discrete state spaces, and compact action sets. Within this framework, a question on the existence of optimal stationary policies, formulated by Puterman (1994, p. 326), is considered. The paper concerns the possibility of obtaining an affirmative answer when additional assumptions are imposed on the decision model. Three conditions ensuring the existence of average optimal stationary policies in finite-state MDPs are analyzed, and it is shown that only the so-called structural continuity condition is a natural sufficient assumption under which the existence of total-reward optimal stationary policies can be guaranteed. In particular, this existence result holds for unichain MDPs with finite state space, but an example is provided to show that this general conclusion does not have an extension to the denumerable state space case.

Blackwell Optimality in Borelian Continuous-in-Action Markov Decision Processes

We prove the existence of stationary Blackwell optimal policies in Markov decision processes with a Borel state space, compact action sets, and continuous-in-action and bounded transition densities and rewards, satisfying a simultaneous Doeblin-type condition. The proof is based on a compactification of the randomized stationary policy space in a weak-strong topology, on the continuity of Laurent coefficients of the discounted rewards in this topology, and on a lexicographical policy improvement. Until now similar results were obtained for the models with a denumerable state space or with a Borel state space and finite action sets.

The policy iteration algorithm for average reward Markov decision processes with general state space

The average cost optimal control problem is addressed for Markov decision processes with unbounded cost. It is found that the policy iteration algorithm generates a sequence of policies which are c-regular, where c is the cost function under consideration. This result only requires the existence of an initial c-regular policy and an irreducibility condition on the state space. Furthermore, under these conditions the sequence of relative value functions generated by the algorithm is bounded from below and "nearly" decreasing, from which it follows that the algorithm is always convergent. Under further conditions, it is shown that the algorithm does compute a solution to the optimality equations and hence an optimal average cost policy. These results provide elementary criteria for the existence of optimal policies for Markov decision processes with unbounded cost and recover known results for the standard linear-quadratic-Gaussian problem. In particular, in the control of multiclass queueing networks, it is found that there is a close connection between optimization of the network and optimal control of a far simpler fluid network model.

Notes on equivalent stationary policies in Markov decision processes with total rewards

We construct examples of Markov Decision Processes for which, for a given initial state and for a given nonstationary transient policy, there is no equivalent (randomized) stationary policy, i.e. there is no stationary policy which occupation measure is equal to the occupation measure of a given policy. We also investigate the relation between the existence of equivalent stationary policies in special models and the existence of equivalent strategies in various classes of nonstationary policies in general models.

On the convergence and ODE limit of a two-dimensional stochastic approximation

We consider a two-dimensional stochastic approximations scheme of the Robbins-Monro type which naturally arises in the study of steering policies for Markov decision processes. Making use of a decoupling change of variables, we establish its almost sure convergence by ad-hoc arguments that combine standard results on one-dimensional stochastic approximations with a version of the law of large numbers for martingale differences. We use this direct analysis to guide us in selecting the test function which appears in standard convergence results for multidimensional schemes. Furthermore, although a blind application of the ordinary differential equation (ODE) method is not possible here due to a lack of regularity properties, the aforementioned change of variables paves the way for an interpretation of the behavior of solutions to the associated limiting ODE.< <ETX xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">&gt;</ETX>

The performance of forwards induction policies

Following major theoretical advances in the study of multi-armed bandit problems, Gittins proposed a forwards induction (FI) approach to the development of policies for Markov decision processes (MDP's). Considerable computational savings are often possible over conventional dynamic programming. We describe procedures for computing such policies and give a bound on their suboptimality. This yields, inter alia, a probabilistic analysis of FI policies for families of competing MDP's. The paper concludes with a detailed study of the status of FI policies for stochastic scheduling problems with order constraints.

Comparing Policies in Markov Decision Processes: Mandl's Lemma Revisited

A general framework is developed for comparing the long-run average cost of a Markov stationary policy with that of another related policy. The underlying methodology constitutes an extension of some ideas of Mandl to randomized policies, and to Polish state and action spaces. Sufficient conditions for the applicability of the methodology are given. These conditions, which are easy to verify, have a natural probabilistic interpretation in terms of the “stability” of the chain and of the convergence of the control values. The usefulness of the general framework proposed here is illustrated on several applications. Standard results on the convergence of adaptive policies are readily recovered under conditions which are more transparent than the ones existing in the literature, and the convergence of randomized policies is handled as a special case. Finally, a novel application to “probing controls” is outlined.

Markov Decision Processes with Sample Path Constraints: The Communicating Case

We consider time-average Markov Decision Processes (MDPs), which accumulate a reward and cost at each decision epoch. A policy meets the sample-path constraint if the time-average cost is below a specified value with probability one. The optimization problem is to maximize the expected average reward over all policies that meet the sample-path constraint. The sample-path constraint is compared with the more commonly studied constraint of requiring the average expected cost to be less than a specified value. Although the two criteria are equivalent for certain classes of MDPs, their feasible and optimal policies differ for many nontrivial problems. In general, there does not exist optimal or nearly optimal stationary policies when the expected average-cost constraint is employed. Assuming that a policy exists that meets the sample-path constraint, we establish that there exist nearly optimal stationary policies for communicating MDPs. A parametric linear programming algorithm is given to construct nearly optimal stationary policies. The discussion relies on well known results from the theory of stochastic processes and linear programming. The techniques lead to simple proofs of the existence of optimal and nearly optimal stationary policies for unichain and deterministic MDPs, respectively.

Randomized and Past-Dependent Policies for Markov Decision Processes with Multiple Constraints

The Markov decision problem of locating a policy to maximize the long-run average reward subject to K long-run average cost constraints is considered. It is assumed that the state and action spaces are finite and the law of motion is unichain, that is, every pure policy gives rise to a Markov chain with one recurrent class. It is first proved that there exists an optimal stationary policy with a degree of randomization no greater than K; consequently, it is never necessary to randomize in more than K states. A linear program produces the optimal policy with limited randomization. For the special case of a single constraint, we also address the problem of finding optimal nonrandomized, but nonstationary, policies. We show that a round-robin type policy is optimal, and conjecture the same for a steering policy that depends on the entire past history of the process, but whose implementation requires essentially no more storage than that of a pure policy.

The Existence of a Minimum Pair of State and Policy for Markov Decision Processes under the Hypothesis of Doeblin

This paper studies the average-cost Markov decision process with compact state and action spaces and bounded lower semicontinuous cost functions. Following the idea of Borkar’s excellent papers [SIAMJ. Control Optim., 21 (1983), pp. 652–666; 22 (1984), pp. 965–978], the general case where irreducibility is not assumed is considered under the hypothesis of Doeblin and the existence of a minimum pair of state and policy, which attains the infimum of the average expected cost over all initial states and policies, is established. Further, it is proved that under additional weak conditions there exists an optimal stationary policy in the usual sense.

Discount-isotone policies for Markov decision processes

This paper considers infinite horizon discounted Markov decision processes and conditions under which discount-isotone optimal policies exist. Given partial orders over the state and action spaces, a set of discount-isotone optimal policies is a set of optimal policies, one for each discount factor in a given set, such that, for each state, the optimal actions are partially ordered in such a manner as to match the ordering of the discount factors. It is easier to solve problems with small discount factors and the induced partial ordering facilitates the solutions for higher discount factor levels.

Suboptimal policy determination for large-scale Markov decision processes, Part 1: Description and bounds

This paper is the first of two papers that present and evaluate an approach for determining suboptimal policies for large-scale Markov decision processes (MDP). Part 1 is devoted to the determination of bounds that motivate the development and indicate the quality of the suboptimal design approach; Part 2 is concerned with the implementation and evaluation of the suboptimal design approach. The specific MDP considered is the infinite-horizon, expected total discounted cost MDP with finite state and action spaces. The approach can be described as follows. First, the original MDP is approximated by a specially structured MDP. The special structure suggests how to construct associated smaller, more computationally tractable MDP's. The suboptimal policy for the original MDP is then constructed from the solutions of these smaller MDP's. The key feature of this approach is that the state and action space cardinalities of the smaller MDP's are exponential reductions of the state and action space cardinalities of the original MDP.

Adaptive Policies in Markov Decision Processes with Uncertain Transition Matrices

This study is concerned with Markov Decision Processes with uncertain transition matrices. In the discounted case, the Bayesian analysis of this model is studied.We define an adaptive policy and a learning policy and show that there exists, for any ∈> 0 an ∈-optimal and learning policy. In the average case, the non-Bayesian analysis of this model is studied and an optimal adaptive policy is constructed.

The Determination of Approximately Optimal Policies in Markov Decision Processes by the Use of Bounds

In the general area of Markov decision processes, a lot of attention has been given to deriving upper and lower bounds for approximating the optimal performance level. These are, in themselves, not useful unless they can be used to derive an approximately optimal policy. The existing literature does this specifically in the context of the computational methods being used at the time. However, irrespective of the method used, a straight application of one step of Howard's policy space method will give the desired results.

The Determination of Approximately Optimal Policies in Markov Decision Processes by the Use of Bounds

The Determination of Approximately Optimal Policies in Markov Decision Processes by the Use of Bounds

A note on ‘monotone optimal policies for markov decision processes’

The purpose of this short note is to correct some oversights in [1]. More precisely, we point out that stronger assumptions have to be imposed on the decision model (in order to use results in [2]) and present a counterexample to a comment to [1, Theorem 3.1].

On sets of countable non-negative matrices and Markov decision processes

Consider a set S of countable non-negative matrices satisfying the property that for any two indices i, j, for some n ≧ 1 there are matrices M 1, M 2, · · ·, Mn in S with (M 1 M 2 · · · Mn ) ij &amp;gt;0. For non-negative vectors x set Tx = sup M∈S Mx, where the supremum is taken separately in each coordinate. Assume that for each x with Tx finite in each coordinate there is a matrix in S which achieves the supremum simultaneously for all coordinates. With these two assumptions on S, the R-theory for a countable irreducible matrix is extended to the operator T. The results are used to consider the existence of stationary optimal policies for Markov decision processes with multiplicative rewards.

On sets of countable non-negative matrices and Markov decision processes

Consider a set S of countable non-negative matrices satisfying the property that for any two indices i, j, for some n ≧ 1 there are matrices M1, M2, · · ·, Mn in S with (M1M2 · · · Mn)ij &gt;0. For non-negative vectors x set Tx = supM∈SMx, where the supremum is taken separately in each coordinate. Assume that for each x with Tx finite in each coordinate there is a matrix in S which achieves the supremum simultaneously for all coordinates. With these two assumptions on S, the R-theory for a countable irreducible matrix is extended to the operator T. The results are used to consider the existence of stationary optimal policies for Markov decision processes with multiplicative rewards.

An iterative procedure for nondiscounted discrete‐time markov decisions

AbstractThis paper considers the problem of computing, by iterative methods, optimal policies for Markov decision processes. The policies computed are optimal for all sufficiently small interest rates.