Markov Decision Chains Research Articles

Technical Note—A Pure Birth Model of Optimal Advertising with Word-of-Mouth

A stochastic, dynamic model of advertising, which incorporates both advertising and word-of-mouth effects, is formulated. The time between the acquisition of new customers is assumed to be random. The distribution of the time until the firm obtains a new customer depends upon the rate of advertising expenditures and upon a word-of-mouth parameter. The problem of choosing the rate of advertising expenditures so as to maximize long-run expected profit is formulated as a continuous-time Markov decision chain. The impact of changes in various parameters of the model on optimal advertising decisions is studied.

Multiplicative Markov Decision Chains

Previous treatments of multiplicative Markov decision chains (eg., Bellman [Bellman, R. 1957. Dynamic Programming. Princeton University Press, Princeton, New Jersey.], Mandl [Mandl, P. 1967. An iterative method for maximizing the characteristic root of positive matrices. Rev. Roumaine Math. Pures Appl. XII 1317–1322.], and Howard and Matheson [Howard, R. A., Matheson, J. E. 1972. Risk-sensitive Markov decision processes. Management Sci. 8 356–369.]) restricted attention to stationary policies and assumed that all transition matrices are irreducible and aperiodic. They also used a “first term” optimality criterion, namely maximizing the spectral radius of the associated transition matrix. We give a constructive proof of the existence of optimal policies among all policies under new cumulative average optimality criteria which are more sensitive than the maximization of the spectral radius. The algorithm for finding an optimal policy, first searches for a stationary policy with a nonnilpotent transition matrix, provided such a rule exists. Otherwise, the method still finds an optimal policy; though in this case the set of optimal policies usually does not contain a stationary policy! If a stationary policy with a nonnilpotent transition matrix exists, then we develop a policy improvement algorithm which finds a stationary optimal policy.

Growth Optimality for Branching Markov Decision Chains

This paper considers a (multiplicative) process called branching Markov decision chains in which the output at the end of the Nth period equals the product of N nonnegative matrices chosen at the beginning of periods 1, …, N, respectively, times a positive (fixed) terminal reward vector. It is assumed that the above transition matrices are drawn out of a finite set of matrices given in product form (i.e., the rows of the matrices can be selected independently out of finite sets of nonnegative row vectors). For each coordinate s we define the geometric and algebraic growth rates, respectively, of the sth coordinate of the stream of output. These growth rates are defined so that the magnitude of the corresponding sequence is of the order αNNk where α is the geometric growth rate and k is the algebraic growth rate. The main result of this paper is the constructive establishment of the existence of a transition matrix whose repeated use will guarantee, for each coordinate, the achievement of the best geometric growth rate and the best algebraic growth rate subject to the geometric growth rate at maximum, among all potential sequences of transition matrices.

Optimality conditions for a Markov decision chain with unbounded costs

It is known that when costs are unbounded satisfaction of the appropriate dynamic programming ‘optimality' equation by a policy is not sufficient to guarantee its average optimality. A ‘lowest-order potential' condition is introduced which, along with the dynamic programming equation, is sufficient to establish the optimality of the policy. Also, it is shown that under fairly general conditions, if the lowest-order potential condition is not satisfied there exists a non-memoryless policy with smaller average cost than the policy satisfying the dynamic programming equation.

Optimality conditions for a Markov decision chain with unbounded costs

It is known that when costs are unbounded satisfaction of the appropriate dynamic programming ‘optimality' equation by a policy is not sufficient to guarantee its average optimality. A ‘lowest-order potential' condition is introduced which, along with the dynamic programming equation, is sufficient to establish the optimality of the policy. Also, it is shown that under fairly general conditions, if the lowest-order potential condition is not satisfied there exists a non-memoryless policy with smaller average cost than the policy satisfying the dynamic programming equation.

Overtaking Optimality for Markov Decision Chains

In a finite Markov decision chain, let V(n, Π)i denote the expectation of the income earned for starting at state i and following (stationary or nonstationary) policy Π for n epochs. Call policy Π overtaking optimal if lim infn→∞ {V(n, Π)i − V(n, Λ)i} ≥ 0 for every policy A and every state i. This paper provides conditions under which a certain stationary policy is overtaking optimal. Tests for these conditions are provided. These conditions hold, for instance, when Blackwell's (Blackwell, D. 1962. Discrete dynamic programming. Ann. Math. Statist. 33 719–726.) policy iteration routine terminates unambiguously, without ties, with a single policy whose transition matrix is not cyclic.

Linear Programming and Markov Decision Chains

In this paper we show that for a finite Markov decision process an average optimal policy can be found by solving only one linear programming problem. Also the relation between the set of feasible solutions of the linear program and the set of stationary policies is analyzed.

Stochastic Dynamic Location Analysis

This research introduces methods of stochastic decision processes into location analysis. The specific model concerns making dynamic relocation decisions for a new facility (server) that must interact with existing facilities (customers) whose relocations are stochastic processes. The model is an infinite-horizon Markov decision chain whose solution gives a server relocation policy that minimizes the expected discounted sum of costs. Costs are location-dependent and are incurred in two ways: when the server makes choice relocations and when the server interacts with customers. The model captures the essence of a variety of familiar dynamic location decision situations. Some methodological developments that allow solution of large problems are reported.

部分観測可能なマルコフ決定過程とアルゴリズム

In this paper we consider an optimal control problem for partially observable Markov decision processes with finite states, signals and actions OVE,r an infinite horizon. It is shown that there are €­ optimal piecewise·linear value functions and piecl~wise-constant policies which are simple. Simple means that there are only finitely many pieces, each of which is defined on a convex polyhedral set. An algorithm based on the method of successive approximation is developed to compute €-optimal policy and €·optimal cost. Furthermore, a special class of stationary policies, called finitely transient, will be considered. It will be shown that such policies have attractive properties which enable us to convert a partially observable Markov decision chain into a usual finite state Markov one.

Normalized Markov Decision Chains. II: Optimality of Nonstationary Policies

In this paper we consider finite state and action, discrete time parameters normalized Markov decision chains, i.e., Markov decision processes with transition matrices that are nonnegative with spectral radius not exceeding one (but not necessarily substochastic). We show that the periodical reward gained in period N is bounded by a polynom, uniformly over the set of all policies. The degree of this polynom can be obtained by considering only the set of stationary policies. Extending and improving results of Sladky (1974) for the stochastic case, we obtain necessary and sufficient conditions for n discount optimality of arbitrary (not necessarily stationary) policies.

Note—A Test for Nonoptimal Actions in Undiscounted Finite Markov Decision Chains

A test for nonoptimal actions in undiscounted Markov decision chains is proposed. The test eliminates actions for one or more stages after which they may re-enter the set of possibly optimal actions, but as convergence proceeds such re-entries cease.

Markov decision chains with unbounded costs and applications to the control of queues

A discrete-time Markov decision model with a denumerable set of states and unbounded costs is considered. It is shown that the optimality equation of dynamic programming along with some additional, easily checked, conditions may be used to establish the optimality or ∊ -optimality of policies with respect to the average expected cost criterion. The results are used to derive optimal policies in two queueing examples.

Markov decision chains with unbounded costs and applications to the control of queues

A discrete-time Markov decision model with a denumerable set of states and unbounded costs is considered. It is shown that the optimality equation of dynamic programming along with some additional, easily checked, conditions may be used to establish the optimality or ∊ -optimality of policies with respect to the average expected cost criterion. The results are used to derive optimal policies in two queueing examples.

Normalized Markov Decision Chains I; Sensitive Discount Optimality

In this paper we study sensitive discount optimality criteria for finite state and action, discrete time parameter, stationary generalized Markov decision chains. We extend previous results obtained by Miller and Veinott and Veinott for substochastic transition matrices to arbitrary non-negative matrices with spectral radius not exceeding one. In particular, we generalize their policy improvement algorithm for finding a stationary policy maximizing the expected discounted reward for all sufficiently small positive interest rates.

Bounds and Transformations for Discounted Finite Markov Decision Chains

This paper develops new improved bounds on the optimal return function in finite state and action, infinite horizon, discounted stationary Markov decision chains. The bounds are obtained by solving a single-constraint, bounded-variable linear program. They can be used for algorithmic termination criteria and improved tests for suboptimal decisions. We show how to implement these tests so that little additional computational effort is required. We consider several transformations that can be used to convert a process into an equivalent one that may be easier to solve. We examine whether the transformations reduce the spectral radius and/or the norm (maximum row sum) of the process. Gauss-Seidel iteration and Jacobi iteration are shown to be special cases of the general transformations. Gauss-Seidel iteration is given additional consideration. Another special case not only preserves equality of the row sums and sparsity but, when applicable, can dramatically reduce the norm. It reduces each element in a column by that column's smallest element. Several possible computational approaches are applied to a small numerical example.

Note—A Counterexample in Continuous Markov Decision Chains

Results in another paper in this issue may raise the question of whether for finite state and action, continuous time parameter Markov decision chains there always exists an initially stationary optimal policy. We give an example in which there is do such policy.

Preferred Rules in Continuous Time Markov Decision Processes

Motivated by a planning horizon result for continuous time Markov decision chains, we study decision rules, called preferred, which may be used in the initially stationary part of nearly optimal policies. We characterize these rules and then, under conditions involving state recurrence and accessibility, consider finding such rules. We also discuss the connection between preferred rules and certain discounted process decision rules, and the role of preferred rules in optimal policies.

On Maximal Rewards and $|varepsilon$-Optimal Policies in Continuous Time Markov Decision Chains

For continuous time Markov decision chains of finite duration, we show that the vector of maximal total rewards, less a linear average-return term, converges as the duration $t \rightarrow \infty$. We then show that there are policies which are both simultaneously $\varepsilon$-optimal for all durations $t$ and are stationary except possibly for a final, finite segment. Further, the length of this final segment depends on $\varepsilon$, but not on $t$ for large enough $t$, while the initial stationary part of the policy is independent of both $\varepsilon$ and $t$.

Constrained Markov Decision Chains

We consider finite state and action discrete time parameter Markov decision chains. The objective is to provide an algorithm for finding a policy that minimizes the long-run expected average cost when there are linear side conditions on the limit points of the expected state-action frequencies. This problem has been solved previously only for the case where every deterministic stationary policy has at most one ergodic class. This note removes that restriction by applying the Dantzig-Wolfe decomposition principle.