Undiscounted Markov Decision Processes Research Articles

Economic MPC of Markov Decision Processes: Dissipativity in undiscounted infinite-horizon optimal control

Economic Model Predictive Control (MPC) dissipativity theory is central to discussing the stability of policies resulting from minimizing economic stage costs. In its current form, the dissipativity theory for economic MPC applies to problems based on deterministic dynamics or to very specific classes of stochastic problems, and does not readily extend to generic Markov decision processes. In this paper, we clarify the core reason for this difficulty, and propose a generalization of the economic MPC dissipativity theory that circumvents it. This generalization focuses on undiscounted infinite-horizon problems and is based on nonlinear stage cost functionals, allowing one to discuss the Lyapunov asymptotic stability of policies for Markov decision processes in terms of the probability measures underlying their stochastic dynamics. This theory is illustrated for the stochastic linear quadratic regulator with Gaussian process noise, for which a storage functional can be provided explicitly. For the sake of brevity, we limit our discussion to undiscounted Markov decision processes.

Undiscounted Recursive Path Choice Models: Convergence Properties and Algorithms

Traffic flow predictions are central to a wealth of problems in transportation. Path choice models can be used for this purpose, and in state-of-the-art models—so-called recursive path choice (RPC) models—the choice of a path is formulated as a sequential arc choice process using undiscounted Markov decision process (MDP) with an absorbing state. The MDP has a utility maximization objective with unknown parameters that are estimated based on data. The estimation and prediction using RPC models require repeatedly solving value functions that are solutions to the Bellman equation. Although there are several examples of successful applications of RPC models in the literature, the convergence of the value iteration method has not been studied. We aim to address this gap. For the two closed-form models in the literature—recursive logit (RL) and nested recursive logit (NRL)—we study the convergence properties of the value iteration method. In the case of the RL model, we show that the operator associated with the Bellman equation is a contraction under certain assumptions on the parameter values. On the contrary, the operator in the NRL case is not a contraction. Focusing on the latter, we study two algorithms designed to improve upon the basic value iteration method. Extensive numerical results based on two real data sets show that the least squares approach we propose outperforms two value iteration methods.

Demand-side energy management under time-varying prices

Under time-varying electricity prices, an end-user may be stimulated to delay flexible demands that can be shifted over time. In this article, we study the problem where each end-user adopts an energy management system that helps time flexible demands fulfillments. Discomfort costs are incurred if demand is not satisfied immediately upon arrival. Energy storage and trading decisions are also considered. We model the problem as a finite horizon undiscounted Markov Decision Process, and outline a tractable approximate dynamic programming approach to overcome the curse of dimensionality. Specifically, we construct an approximation for the value-to-go function such that Bellman equations are converted into mixed-integer problems with structural properties. Finally, we numerically demonstrate that our approach achieves close performance to the exact approach, while dominating the myopic policy and no-control policy. Most importantly, the proposed approach can take advantage of the price differences and efficiently shift demands.

New approximate dynamic programming algorithms for large-scale undiscounted Markov decision processes and their application to optimize a production and distribution system

Undiscounted Markov decision processes (UMDP's) can formulate optimal stochastic control problems that minimize the expected total cost per period for various systems. We propose new approximate dynamic programming (ADP) algorithms for large-scale UMDP's that can solve the curses of dimensionality. These algorithms, called simulation-based modified policy iteration (SBMPI) algorithms, are extensions of the simulation-based modified policy iteration method (SBMPIM) (Ohno, 2011) for optimal control problems of multistage JIT-based production and distribution systems with stochastic demand and production capacity. The main new concepts of the SBMPI algorithms are that the simulation-based policy evaluation step of the SBMPIM is replaced by the partial policy evaluation step of the modified policy iteration method (MPIM) and that the algorithms starts from the expected total cost per period and relative value estimated by simulating the system under a reasonable initial policy.For numerical comparisons, the optimal control problem of the three-stage JIT-based production and distribution system with stochastic demand and production capacity is formulated as a UMDP. The demand distribution is changed from a shifted binomial distribution in Ohno (2011) to a Poisson distribution and near-optimal policies of the optimal control problems with 35,973,840 states are computed by the SBMPI algorithms and the SBMPIM. The computational result shows that the SBMPI algorithms are at least 100 times faster than the SBMPIM in solving the numerical problems and are robust with respect to initial policies. Numerical examples are solved to show an effectiveness of the near optimal control utilizing the SBMPI algorithms compared with optimized pull systems with optimal parameters computed utilizing the SBOS (simulation-based optimal solutions) from Ohno (2011).

Parts quality‐based priority policy in remanufacturing environments

PurposeThe purpose of this study is to consider the acquisition of end‐of‐life products under variable quality consideration for remanufacturing so as to determine optimal control policy that minimizes per‐period expected costs that may guide future consideration by practitioners.Design/methodology/approachThe authors review recent literature on reverse supply chains and remanufacturing. They utilize an undiscounted Markov decision process methodology to ascertain the order amount of remanufacturable products using optimal control under minimum cost criterion.FindingsThe authors conclude that it makes sense for firms to focus on the cost management with production control based on quality levels with different acquisition costs of remanufacturable products.Research limitations/implicationsAlthough the Markov decision process methodology – which is well supported in literature – was diligently followed, the nature of analysis and discussion may be subject to authors’ bias. Future investigation and adoption of the methodological approach used will verify the paper findings.Practical implicationsThis study determines optimal control policy for ordering specific amount of product that minimizes per‐period expected costs for remanufacturing. Reverse supply‐chain professionals now have an easy‐to‐follow guide when acquiring end‐of‐life remanufacturable products alternatives with variable quality.Social implicationsThis study determines the optimal policy for ordering remanufacturable products. This information enables practitioners to reduce their carbon footprint in reverse supply chain through inspection/sorting before remanufacturing by processing only the type, quality, and quantity of needed product.Originality/valueFor reverse supply chain to be taken seriously by senior management in firms, it is imperative that practitioners in this field synchronize their operational‐level ordering decisions with holistic cost minimization objective (to maximize value recovery) to stay viable.

The optimal control of just-in-time-based production and distribution systems and performance comparisons with optimized pull systems

In just-in-time (JIT) production systems, there is both input stock in the form of parts and output stock in the form of product at each stage. These activities are controlled by production-ordering and withdrawal kanbans. This paper discusses a discrete-time optimal control problem in a multistage JIT-based production and distribution system with stochastic demand and capacity, developed to minimize the expected total cost per unit of time. The problem can be formulated as an undiscounted Markov decision process (UMDP); however, the curse of dimensionality makes it very difficult to find an exact solution. The author proposes a new neuro-dynamic programming (NDP) algorithm, the simulation-based modified policy iteration method (SBMPIM), to solve the optimal control problem. The existing NDP algorithms and SBMPIM are numerically compared with a traditional UMDP algorithm for a single-stage JIT production system. It is shown that all NDP algorithms except the SBMPIM fail to converge to an optimal control. Additionally, a new algorithm for finding the optimal parameters of pull systems is proposed. Numerical comparisons between near-optimal controls computed using the SBMPIM and optimized pull systems are conducted for three-stage JIT-based production and distribution systems. UMDPs with 42 million states are solved using the SBMPIM. The pull systems discussed are the kanban, base stock, CONWIP, hybrid and extended kanban.

Technical note: A computationally efficient algorithm for undiscounted Markov decision processes with restricted observations

AbstractWe present a computationally efficient procedure to determine control policies for an infinite horizon Markov Decision process with restricted observations. The optimal policy for the system with restricted observations is a function of the observation process and not the unobservable states of the system. Thus, the policy is stationary with respect to the partitioned state space. The algorithm we propose addresses the undiscounted average cost case. The algorithm combines a local search with a modified version of Howard's (Dynamic programming and Markov processes, MIT Press, Cambridge, MA, 1960) policy iteration method. We demonstrate empirically that the algorithm finds the optimal deterministic policy for over 96% of the problem instances generated. For large scale problem instances, we demonstrate that the average cost associated with the local optimal policy is lower than the average cost associated with an integer rounded policy produced by the algorithm of Serin and Kulkarni Math Methods Oper Res 61 (2005) 311–328. © 2008 Wiley Periodicals, Inc. Naval Research Logistics 2009

Optimal threshold probability in undiscounted Markov decision processes with a target set

We consider risk minimizing problems in undiscounted Markov decisions processes with a target set. We formulate the problem as an infinite horizon case with a recurrent class. We show that an optimal value function is a unique solution to an optimality equation and there exists an stationary optimal policy. Also we give several value iteration methods and a policy improvement method.

A K-step look-ahead analysis of value iteration algorithms for Markov decision processes

We introduce and analyze a general look-ahead approach for Value Iteration Algorithms used in solving both discounted and undiscounted Markov decision processes. This approach, based on the value-oriented concept interwoven with multiple adaptive relaxation factors, leads to accelerating procedures which perform better than the separate use of either the concept of value oriented or of relaxation. Evaluation and computational considerations of this method are discussed, practical guidelines for implementation are suggested and the suitability of enhancing the method by incorporating Phase 0, Action Elimination procedures and Parallel Processing is indicated. The method was successfully applied to several real problems. We present some numerical results which support the superiority of the developed approach, particularly for undiscounted cases, over other Value Iteration variants.

Mean-Variance Tradeoffs in an Undiscounted MDP: The Unichain Case

The problem analyzed here is the computation of Pareto optima in the sense of high mean and low variance of the stationary distribution in the unichain, undiscounted Markov decision process (MDP, for short).

Mean-Variance Tradeoffs in an Undiscounted MDP

A stationary policy and an initial state in an MDP (Markov decision process) induce a stationary probability distribution of the reward. The problem analyzed here is generating the Pareto optima in the sense of high mean and low variance of the stationary distribution. In the unichain case, Pareto optima can be computed either with policy improvement or with a linear program having the same number of variables and one more constraint than the formulation for gain-rate optimization. The same linear program suffices in the multichain case if the ergodic class is an element of choice.

Mean-variance criteria in an undiscounted Markov decision process

In the steady state of an undiscounted Markov decision process, we consider the problem of finding an optimal stationary probability distribution that minimizes the variance of the reward in a transition among the stationary probability distributions which give a mean not less than a specified value. The problem consists of a mathematical program with linear constraints and a non-linear objective. The solution technique replaces the non-linear part of the objective with a constant, inserts the constant as a constraint, and then parametrically analyzes the resulting linear program. Three numerical examples are discussed.

Conditions for the discovery of solution horizons

We present necessary and sufficient conditions for discrete infinite horizon optimization problems with unique solutions to be solvable. These problems can be equivalently viewed as the task of finding a shortest path in an infinite directed network. We provide general forward algorithms with stopping rules for their solution. The key condition required is that of weak reachability, which roughly requires that for any sequence of nodes or states, it must be possible from optimal states to reach states close in cost to states along this sequence. Moreover the costs to reach these states must converge to zero. Applications are considered in optimal search, undiscounted Markov decision processes, and deterministic infinite horizon optimization.

Remarks on maximal meanstandard devition ratio in undiscounted mdps

In the steady state of an undiscounted Markov decision process, we consider the problem to find an optimal stationary probability distribution that maximizes the mean standard deviation ratio among all the stationary probability distributions. The problem injects considerations in MDPs from the relative point of view

Criteria for selecting the relaxation factor of the value iteration algorithm for undiscounted Markov and semi-Markov decision processes

We present two criteria for selecting the adaptive relaxation factor being used in speeding-up the value iteration algorithm for undiscounted Markov decision processes. The criteria are: 1. Minimum Ratio, 2. Minimum Variance. For the problems tested it was found that the criterion of minimum variance is most effective for Markov models while an hybrid use of both criteria yields best results when solving semi-Markov decision problems. The advantage of using these criteria increases with the dimension of the models.

A note on maximal mean/standard deviation ratio in an undiscounted MDP

A stationary policy in an MDP (Markov decision process) induces a stationary probability distribution of the reward from each initial state. This note is related to the problem of maximizing the mean/standard deviation ratio of the stationary distribution. It concludes that a pure policy optimum exists.

A value iteration method for undiscounted multichain Markov decision processes

This paper proposes a value iteration method which finds ane-optimal policy of an undiscounted multichain Markov decision process in a finite number of iterations. The undiscounted multichain Markov decision process is reduced to an aggregated Markov decision process, which utilizes maximal gains of undiscounted Markov decision sub-processes and is formulated as an optimal stopping problem. As a preliminary, sufficient conditions are presented under which a policy ise-optimal.

Technical Note—Successive Approximations in Value Determination for a Markov Decision Process

We present a new procedure for solving the value-determination equations for an undiscounted Markov decision process by the successive approximations of a Gauss-Seidel or Jacobi iteration method. The key step is to use the Markov chain partitioning algorithm to compute the steady-state probabilities corresponding to a given policy. The gain, computed in terms of the steady-state probabilities, is substituted into the value-determination equations. After deleting one relative value and one value-determination equation, we show that the remaining system of equations satisfies the sufficient conditions for the convergence of a Gauss-Seidel or Jacobi iteration.

Asymptotic expansions for dynamic programming recursions with general nonnegative matrices

This paper is concerned with the study of the asymptotic behavior of dynamic programming recursions of the form $$x(n + 1) = \mathop {\max }\limits_{P \in \mathcal{K}} Px(n), n = 0,1,2,...,$$ where ℜ denotes a set of matrices, generated by all possible interchanges of corresponding rows, taken from a fixed finite set of nonnegative square matrices. These recursions arise in a number of well-known and frequently studied problems, e.g. in the theory of controlled Markov chains, Leontief substitution systems, controlled branching processes, etc. Results concerning the asymptotic behavior ofx(n), forn→∞, are established in terms of the maximal spectral radius, the maximal index, and a set of generalized eigenvectors. A key role in the analysis is played by a geometric convergence result for value iteration in undiscounted multichain Markov decision processes. A new proof of this result is also presented.

On the Convergence of Policy Iteration in Finite State Undiscounted Markov Decision Processes: The Unichain Case

We study the convergence of policy iteration for the undiscounted, finite state, discrete time Markov decision problem with compact action space and unichain transition structure. Using a “Newton Method type” representation for policy iteration, we establish the existence of a solution to the optimality equation. We show that to find an average optimal policy, it is sufficient to solve the optimality equation on the recurrent set of the maximizing policy. Under the additional assumption of a unique maximizing policy at each stage of the policy iteration procedure, we show that the iterates are convergent and the resulting policy is Blackwell optimal.