Finite-state Markov Decision Processes Research Articles

Markov decision processes and regular events

Desirable properties of the infinite histories of a finite-state Markov decision process are specified in terms of a finite number of events represented as /spl omega/-regular sets. An infinite history of the process produces a reward which depends on the properties it satisfies. The authors investigate the existence of optimal policies and provide algorithms for the construction of such policies.

Markov Decision Processes with Imprecise Transition Probabilities

We present new numerical algorithms and bounds for the infinite horizon, discrete stage, finite state and action Markov decision process with imprecise transition probabilities. We assume that the transition probability mass vector for each state and action is described by a finite number of linear inequalities. This model of imprecision appears to be well suited for describing statistically determined confidence limits and/or natural language statements of likelihood. The numerical procedures for calculating an optimal max-min strategy are based on successive approximations, reward revision, and modified policy iteration. The bounds that are determined are at least as tight as currently available bounds for the case where the transition probabilities are precise.

Computationally efficient algorithms for on-line optimization of markov decision processes

Iterative algorithms are proposed for on-line optimization of finite state Markov decision processes. The optimization criterion is the minimization of long-run average cost. The transition probabilities of the underlying Markov chain are assumed to depend on certain unknown parameters. The algorithms estimate the unknown para-meters using a strongly consistent estimator, and use this estimate in place of the true parameter to generate a sequence of control policies which converge to an optimal policy. The main feature of the algorithms is that the updating step does not have to be carried out in all states at each iteration. At each iteration, the updating step may be performed in any number of states—as few as one—as long as all states are updated “often enough” as the number of iterations increases; in this sense the algorithms are asynchronous. The asynchronous nature of our algorithms allows us to update the costly states more often. As will be illustrated through simulation results, the asynchronism of our algorithms makes them feasible for on-line optimization.

Multichain Markov Decision Processes with a Sample Path Constraint

We consider finite-state finite-action Markov decision processes which accumulate both a reward and a cost at each decision epoch. We study the problem of finding a policy that maximizes the expect...

Multichain Markov Decision Processes with a Sample Path Constraint: A Decomposition Approach

We consider finite-state finite-action Markov decision processes which accumulate both a reward and a cost at each decision epoch. We study the problem of finding a policy that maximizes the expected long-run average reward subject to the constraint that the long-run average cost be no greater than a given value with probability one. We establish that if there exists a policy that meets the constraint, then there exists an ε-optimal stationary policy. Furthermore, an algorithm is outlined to locate the ε-optimal stationary policy. The proof of the result hinges on a decomposition of the state space into maximal recurrent classes and a set of transient states.

Optimal Acquisition of Automated Flexible Manufacturing Processes

We formulate the problem of converting a labor-intensive batch production process to one that incorporates flexible automation as a finite-state Markov decision process. Interest rates and the level of automated technology influence both operating and acquisition costs and are treated as random variables. The model specifies the optimal level of capacity to convert to flexible automation. The optimization criterion is the minimization of the sum of expected, discounted costs incurred over a finite planning horizon. The optimal acquisition strategy depends upon the time period, the current interest rate, the current level of technology, and a measure of the remaining capacity that is not automated. We investigate the structure of optimal acquisition strategies using mathematical analysis and simulation. Our objective is to illustrate the qualitative characteristics of optimal strategies for acquiring flexible automation. As a step toward the implementation of the model, we examine the qualitative consequences associated with specifying classes of inventory and acquisition cost functions.

Generalized Inverses in Discrete Time Markov Decision Processes

A new self-contained approach based on the Drazin generalized inverse is used to derive many basic results in discrete time, finite state Markov decision processes. A product form representation for the transition matrix of a stationary policy gives new derivations of the average reward evaluation equations, Laurent series expansions, as well as the finite test for Blackwell optimality. This representation also suggests new computational methods.

Decentralized optimal control of Markov chains with a common past information set

Decentralized dynamic (closed-loop) optimal control strategies are sought for a class of finite state Markov decision processes, characterized by the sharing of a common past after k steps of delay. The control is considered over a finite time horizon, and it is shown that a nonclassical dynamic programming procedure can be applied, based on the existence of a sufficient statistic of constant dimension. Finally, the infinite horizon case is briefly discussed, in view of an extention of existing results on the minimization of average expected cost for the centralized and decentralized control of Markov chains.

Monotonically Improving Limit-Optimal Strategies in Finite-State Decision Processes

In every finite-state leavable gambling problem and in every finite-state Markov decision process with discounted, negative or positive reward criteria there exists a Markov strategy which is monotonically improving and optimal in the limit along every history. An example is given to show that for the positive and gambling cases such strategies cannot be constructed by simply switching to a “better” action or gamble at each successive return to a state.

Reward Revision for Discounted Markov Decision Problems

We present a numerical procedure for determining the optimal total expected discounted reward vector f* for an infinite horizon, discrete stage, finite state and action Markov decision process (MDP). This procedure exploits the fact that many MDPs generate the same vector f*. The objective of the procedure is to simultaneously construct and solve one such MDP that has a computationally attractive transition structure. The construction of this MDP requires the periodic revision of its reward structure. We then present a simple, a priori method for estimating the impact of the new procedure on operation counts as compared to standard successive approximations. This method is useful for determining whether or not the new procedure should be used and for selecting an important design parameter. We also describe two extensions of the new procedure, one generalizing a standard extrapolation and the other a modified policy iteration algorithm. A numerical evaluation indicates that for MDPs having transition structures with a small number of dominant probabilities per row, the new procedure can significantly reduce CPU time.

Adaptive control of discounted Markov decision chains

In this paper, we consider discounted-reward finite-state Markov decision processes which depend on unknown parameters. An adaptive policy inspired by the nonstationary value iteration scheme of Federgruen and Schweitzer (Ref. 1) is proposed. This policy is briefly compared with the principle of estimation and control recently obtained by Schal (Ref. 4).

Connectedness conditions used in finite state Markov Decision Processes

This note considers the conditions that have been put on the set of transition matrices of finite state Markov Decision Processes in order to prove the existence of optimal policies, and the convergence of algorithms. It is shown that no two of the nine conditions considered are equivalent.

Markov ratio decision processes

A finite-state Markov decision process, in which, associated with each action in each state, there are two rewards, is considered. The objective is to optimize the ratio of the two rewards over an infinite horizon. In the discounted version of this decision problem, it is shown that the optimal value is unique and the optimal strategy is pure and stationary; however, they are dependent on the starting state. Also, a finite algorithm for computing the solution is given.

Markov Decision Processes with a New Optimality Criterion: Continuous Time

Standard finite state and action continuous time Markov decision processes with discounting are studied using a new optimality criterion called moment optimality. A policy is moment optimal if it lexicographically maximizes the sequence of signed moments of total discounted return with a positive (negative) sign if the moment is odd (even). It is shown constructively that a stationary policy is moment optimal among the class of piecewise constant policies by examining the negative of the Laplace transform of the total return random variable and its Taylor series expansion.

Markov Decision Processes with a New Optimality Criterion: Discrete Time

Standard finite state and action discrete time Markov decision processes with discounting are studied using a new optimality criterion called moment optimality. A policy is moment optimal if it lexicographically maximizes the sequence of signed moments of total discounted return with a positive (negative) sign if the moment is odd (even). This criterion is equivalent to being a little risk adverse. It is shown that a stationary policy is moment optimal by examining the negative of the Laplace transform of the total return random variable. An algorithm to construct all stationary moment optimal policies is developed. The algorithm is shown to be finite.

Markov Decision Processes with a New Optimality Criterion: Small Interest Rates

Finite state and action discrete time Markov decision processes with discounting are considered under the criterion of moment optimality. The case of small interest rates is studied, in particular the behavior of optimal policies as the interest rate approaches zero. Laurent expansions in the interest rate are developed for all moments of return for stationary policies, and a proof is given that there is a stationary policy which is moment optimal for all sufficiently small interest rates.