Discounted Cost Markov Decision Research Articles

Self-Guided Approximate Linear Programs: Randomized Multi-Shot Approximation of Discounted Cost Markov Decision Processes

Approximate linear programs (ALPs) are well-known models based on value function approximations (VFAs) to obtain policies and lower bounds on the optimal policy cost of discounted-cost Markov decision processes (MDPs). Formulating an ALP requires (i) basis functions, the linear combination of which defines the VFA, and (ii) a state-relevance distribution, which determines the relative importance of different states in the ALP objective for the purpose of minimizing VFA error. Both of these choices are typically heuristic; basis function selection relies on domain knowledge, whereas the state-relevance distribution is specified using the frequency of states visited by a baseline policy. We propose a self-guided sequence of ALPs that embeds random basis functions obtained via inexpensive sampling and uses the known VFA from the previous iteration to guide VFA computation in the current iteration. In other words, this sequence takes multiple shots at randomly approximating the MDP value function with VFA-based guidance between consecutive approximation attempts. Self-guided ALPs mitigate domain knowledge during basis function selection and the impact of the state-relevance-distribution choice, thus reducing the ALP implementation burden. We establish high-probability error bounds on the VFAs from this sequence and show that a worst-case measure of policy performance is improved. We find that these favorable implementation and theoretical properties translate to encouraging numerical results on perishable inventory control and options pricing applications, where self-guided ALP policies improve upon policies from problem-specific methods. More broadly, our research takes a meaningful step toward application-agnostic policies and bounds for MDPs. This paper was accepted by Chung Piaw Teo, optimization. Supplemental Material: The online appendix and data files are available at https://doi.org/10.1287/mnsc.2020.00038 .

Optimal admission and queuing control with reneging behavior under premature discharge decisions

AbstractQueues or waiting lines are a common phenomenon in healthcare facilities. The increasing number of admissions to hospital emergency departments (EDs) has resulted in overcrowded EDs and decreased quality of care. Patients may renege with dissatisfaction while waiting, prolonged hospital stays, or the cancelation of surgeries, especially for those with acute diseases. This paper studies patient admission and queuing control policies, considering premature discharge decisions and reneging behavior. First, we formulate this problem as an infinite‐horizon total discounted cost Markov decision process to balance bed utility and experiences of patients waiting for admission to minimize the total expected cost for the hospital. Then, we characterize an optimal admission threshold policy by analyzing the structural properties of the optimal value function. Numerical experiments are employed to discuss how the optimal dynamic policy depends on some key parameters of the system. Finally, the performance of the optimal policy is discussed through comparison with the benchmark: A myopic rule (first come, first served) mimicking the decision‐making in practice under different scenarios. The numerical results show that the reneging behavior can affect the superiority of the optimal policy compared to the benchmark, and the optimal policy can significantly reduce the number of waiting patients without increasing the total expected cost when the system is overloaded.

Optimal admission control under premature discharge decisions for operational effectiveness

AbstractThe shortage of inpatient beds causes surgery cancellations, treatment delays, and inappropriate ward arrangements. That is why studying how to make admission decisions on the wards of crowded units under premature discharge decisions is essential. In this study, we set up an infinite‐horizon total discounted cost Markov decision process model to minimize the total expected cost to the hospital. By analyzing the structural properties of the optimal value function, we find that there is an optimal admission control policy of the threshold type for low‐acuity emergency patients (type‐2). Numerical experiments are done to examine how optimal dynamic policy depends on key system parameters. Furthermore, we evaluate the performance of the optimal dynamic policy with a benchmark policy without premature discharge decisions. The results indicate that the optimal admission control policy helps hospitals in maintaining low costs and enables patients with high‐acuity diseases to enter primary wards more quickly.

Non-Asymptotic Analysis of Monte Carlo Tree Search

In this work, we consider the popular tree-based search strategy within the framework of reinforcement learning, the Monte Carlo Tree Search (MCTS), in the context of infinite-horizon discounted cost Markov Decision Process (MDP) with deterministic transitions. While MCTS is believed to provide an approximate value function for a given state with enough simulations, cf. [5, 6], the claimed proof of this property is incomplete. This is due to the fact that the variant of MCTS, the Upper Confidence Bound for Trees (UCT), analyzed in prior works utilizes "logarithmic" bonus term for balancing exploration and exploitation within the tree-based search, following the insights from stochastic multi-arm bandit (MAB) literature, cf. [1, 3]. In effect, such an approach assumes that the regret of the underlying recursively dependent non-stationary MABs concentrates around their mean exponentially in the number of steps, which is unlikely to hold as pointed out in [2], even for stationary MABs.

Optimal production decisions in biopharmaceutical fill-and-finish operations

Fill-and-finish is among the most commonly outsourced operations in biopharmaceutical manufacturing and involves several challenges. For example, fill-operations have a random production yield, as biopharmaceutical drugs might lose their quality or stability during these operations. In addition, biopharmaceuticals are fragile molecules that need specialized equipment with limited capacity, and the associated production quantities are often strictly regulated. The non-stationary nature of the biopharmaceutical demand and limitations in forecasts add another layer of challenge in production planning. Furthermore, most companies tend to “freeze” their production decisions for a limited period of time, in which they do not react to changes in the manufacturing system. Using such freeze periods helps to improve stability in planning, but comes at a price of reduced flexibility. To address these challenges, we develop a finite-horizon, discounted-cost Markov decision model, and optimize the production decisions in biopharmaceutical fill-and-finish operations. We characterize the structural properties of optimal cost and policies, and propose a new, zone-based decision-making approach for these operations. More specifically, we show that the state space can be partitioned into decision zones that provide guidelines for optimal production policies. We illustrate the use of the model with an industry case study.

Dynamic Energy Storage Control for Reducing Electricity Cost in Data Centers

As the scale of the data centers increases, electricity cost is becoming the fastest-growing element in their operation costs. In this paper, we investigate the electricity cost reduction opportunities utilizing energy storage facilities in data centers used as uninterrupted power supply units (UPS). Its basic idea is to combine the temporal diversity of electricity price and the energy storage to conceive a strategy for reducing the electricity cost. The electricity cost minimization is formulated in the framework of finite state-action discounted cost Markov decision process (MDP). We applyQ-Learning algorithm to solve the MDP optimization problem and derive a dynamic energy storage control strategy, which does not require any priori information on the Markov process. In order to address the slow-convergence problem of theQ-Learning based algorithm, we introduce a SpeedyQ-Learning algorithm. We further discuss the offline optimization problem and obtain the optimal offline solution as the lower bound on the performance of the online and learning theoretic problem. Finally, we evaluate the performance of the proposed scheme by using real workload traces and electricity price data sets. The experimental results show the effectiveness of the proposed scheme.

Q-Learning Based Energy Management Policies for a Single Sensor Node with Finite Buffer

In this paper, we consider the problem of finding optimal energy management policies in the presence of energy harvesting sources to maximize network performance. We formulate this problem in the discounted cost Markov decision process framework and apply two reinforcement learning algorithms. Prior work obtains optimal policy in the case when the conversion function mapping energy to data transmitted is linear and provides heuristic policies in the case when the same is nonlinear. Our algorithms, however, provide optimal policies regardless of the form of the conversion function. Through simulations, our policies are seen to outperform those of in the nonlinear case.

Reducing Electricity Cost of Smart Appliances via Energy Buffering Framework in Smart Grid

To reduce the long term electricity cost of smart appliances (SAs) with deferrable operation time in smart grid, we propose a novel energy buffering framework to intelligently schedule the distributed energy storage (DES) for the cost reduction of SAs in this paper. The proposed energy buffering framework determines the action policy (e.g., charging or discharging) and the power allocation policy of the DES to provide DES power to proper SAs at proper time with lower price than that of the utility grid, resulting in the reduction of the long term financial cost of SAs. Specifically, we first formulate the optimal decision problem in the energy buffering framework as a discounted cost Markov decision process (MDP) over infinite-horizon. Then, we propose an optimal scheme for the energy buffering framework to solve the discounted cost MDP based on online learning approach. Extensive simulation results show that the proposed optimal scheme for the energy buffering framework can significantly reduce the long term financial cost comparing with the baseline schemes and the myopic scheme.

Markov decision processes with exponentially representable discounting

We generalize the geometric discount of finite discounted cost Markov Decision Processes to “exponentially representable”discount functions, prove existence of optimal policies which are stationary from some time N onward, and provide an algorithm for their computation. Outside this class, optimal “ N -stationary” policies in general do not exist.

A Robust Approach to Markov Decision Problems with Uncertain Transition Probabilities

This paper considers a discrete-time infinite horizon discounted cost Markov decision problem in which the transition probability vector for each state-control pair is uncertain. A popular approach to this problem has been to find a policy that performs best in the worst-case scenario. A policy obtained in this manner, however, tends to be conservative. We construct a robust formulation for the problem, which produces a less conservative policy. We characterize the performance of the robust formulation via the probability that the optimal cost of a random instance of the problem is at most that of the robust formulation. A congestion-dependent pricing problem for network services is examined as a numerical example.

Discounted Cost Markov Decision Processes with a Constraint

We consider a discounted cost Markov decision process with a constraint. Relating this to a vector-valued Markov decision process, we prove that there exists a constrained optimal randomized semistationary policy if there exists at least one policy satisfying a constraint. Moreover, we present an algorithm by which we can find the constrained optimal randomized semistationary policy, or we can discover that there exist no policies satisfying a given constraint.

Discounted Cost Markov Decision Processes on Borel Spaces: The Linear Programming Formulation

This paper is concerned with the linear programming formulation of Markov decision processes (or stochastic dynamic programs) with Borel state and action spaces and the discounted cost criterion. The one-stage cost function may be unbounded. A linear program and its dual are introduced, for which is shown the absence of a duality gap and that a strong duality condition holds. These results are used to determine the optimal value and the existence of an optimal policy for the discounted cost problem.