Discounted Cost Markov Decision Processes Research Articles

Self-Guided Approximate Linear Programs: Randomized Multi-Shot Approximation of Discounted Cost Markov Decision Processes

Approximate linear programs (ALPs) are well-known models based on value function approximations (VFAs) to obtain policies and lower bounds on the optimal policy cost of discounted-cost Markov decision processes (MDPs). Formulating an ALP requires (i) basis functions, the linear combination of which defines the VFA, and (ii) a state-relevance distribution, which determines the relative importance of different states in the ALP objective for the purpose of minimizing VFA error. Both of these choices are typically heuristic; basis function selection relies on domain knowledge, whereas the state-relevance distribution is specified using the frequency of states visited by a baseline policy. We propose a self-guided sequence of ALPs that embeds random basis functions obtained via inexpensive sampling and uses the known VFA from the previous iteration to guide VFA computation in the current iteration. In other words, this sequence takes multiple shots at randomly approximating the MDP value function with VFA-based guidance between consecutive approximation attempts. Self-guided ALPs mitigate domain knowledge during basis function selection and the impact of the state-relevance-distribution choice, thus reducing the ALP implementation burden. We establish high-probability error bounds on the VFAs from this sequence and show that a worst-case measure of policy performance is improved. We find that these favorable implementation and theoretical properties translate to encouraging numerical results on perishable inventory control and options pricing applications, where self-guided ALP policies improve upon policies from problem-specific methods. More broadly, our research takes a meaningful step toward application-agnostic policies and bounds for MDPs. This paper was accepted by Chung Piaw Teo, optimization. Supplemental Material: The online appendix and data files are available at https://doi.org/10.1287/mnsc.2020.00038 .

Optimal admission and queuing control with reneging behavior under premature discharge decisions

AbstractQueues or waiting lines are a common phenomenon in healthcare facilities. The increasing number of admissions to hospital emergency departments (EDs) has resulted in overcrowded EDs and decreased quality of care. Patients may renege with dissatisfaction while waiting, prolonged hospital stays, or the cancelation of surgeries, especially for those with acute diseases. This paper studies patient admission and queuing control policies, considering premature discharge decisions and reneging behavior. First, we formulate this problem as an infinite‐horizon total discounted cost Markov decision process to balance bed utility and experiences of patients waiting for admission to minimize the total expected cost for the hospital. Then, we characterize an optimal admission threshold policy by analyzing the structural properties of the optimal value function. Numerical experiments are employed to discuss how the optimal dynamic policy depends on some key parameters of the system. Finally, the performance of the optimal policy is discussed through comparison with the benchmark: A myopic rule (first come, first served) mimicking the decision‐making in practice under different scenarios. The numerical results show that the reneging behavior can affect the superiority of the optimal policy compared to the benchmark, and the optimal policy can significantly reduce the number of waiting patients without increasing the total expected cost when the system is overloaded.

Optimal admission control under premature discharge decisions for operational effectiveness

AbstractThe shortage of inpatient beds causes surgery cancellations, treatment delays, and inappropriate ward arrangements. That is why studying how to make admission decisions on the wards of crowded units under premature discharge decisions is essential. In this study, we set up an infinite‐horizon total discounted cost Markov decision process model to minimize the total expected cost to the hospital. By analyzing the structural properties of the optimal value function, we find that there is an optimal admission control policy of the threshold type for low‐acuity emergency patients (type‐2). Numerical experiments are done to examine how optimal dynamic policy depends on key system parameters. Furthermore, we evaluate the performance of the optimal dynamic policy with a benchmark policy without premature discharge decisions. The results indicate that the optimal admission control policy helps hospitals in maintaining low costs and enables patients with high‐acuity diseases to enter primary wards more quickly.

Non-Asymptotic Analysis of Monte Carlo Tree Search

In this work, we consider the popular tree-based search strategy within the framework of reinforcement learning, the Monte Carlo Tree Search (MCTS), in the context of infinite-horizon discounted cost Markov Decision Process (MDP) with deterministic transitions. While MCTS is believed to provide an approximate value function for a given state with enough simulations, cf. [5, 6], the claimed proof of this property is incomplete. This is due to the fact that the variant of MCTS, the Upper Confidence Bound for Trees (UCT), analyzed in prior works utilizes "logarithmic" bonus term for balancing exploration and exploitation within the tree-based search, following the insights from stochastic multi-arm bandit (MAB) literature, cf. [1, 3]. In effect, such an approach assumes that the regret of the underlying recursively dependent non-stationary MABs concentrates around their mean exponentially in the number of steps, which is unlikely to hold as pointed out in [2], even for stationary MABs.

Asynchronous Stochastic Approximation Based Learning Algorithms for As-You-Go Deployment of Wireless Relay Networks Along a Line

We are motivated by the need, in emergency situations, for impromptu (or “as-you-go”) deployment of multihop wireless networks, by human agents or robots (e.g., unmanned aerial vehicles (UAVs)); the agent moves along a line, makes wireless link quality measurements at regular intervals, and makes on-line placement decisions using these measurements. As a first step, we have formulated such deployment along a line as a sequential decision problem. In our earlier work, reported in [1] , we proposed two possible deployment approaches: (i) the pure as-you-go approach where the deployment agent can only move forward, and (ii) the explore-forward approach where the deployment agent explores a few successive steps and then selects the best relay placement location among them. The latter was shown to provide better performance (in terms of network cost, network performance, and power expenditure), but at the expense of more measurements and deployment time, which makes explore-forward impractical for quick deployment by an energy constrained agent such as a UAV. Further, since in emergency situations the terrain would be unknown, the deployment algorithm should not require a-priori knowledge of the parameters of the wireless propagation model. In [1] , we, therefore, developed learning algorithms for the explore-forward approach. The current paper fills in an important gap by providing deploy-and-learn algorithms for the pure as-you-go approach. We formulate the sequential relay deployment problem as an average cost Markov decision process (MDP), which trades off among power consumption, link outage probabilities, and the number of relay nodes in the deployed network. While the pure as-you-go deployment problem was previously formulated as a discounted cost MDP (see [1] ), the discounted cost MDP formulation was not amenable for learning algorithms that are proposed in this paper. In this paper, first we show structural results for the optimal policy corresponding to the average cost MDP, and provide new insights into the optimal policy. Next, by exploiting the special structure of the average cost optimality equation and by using the theory of asynchronous stochastic approximation (in single and two timescale), we develop two learning algorithms that asymptotically converge to the set of optimal policies as deployment progresses. Numerical results show reasonably fast speed of convergence, and hence the model-free algorithms can be useful for practical, fast deployment of emergency wireless networks.

Actor-Critic Algorithms with Online Feature Adaptation

We develop two new online actor-critic control algorithms with adaptive feature tuning for Markov Decision Processes (MDPs). One of our algorithms is proposed for the long-run average cost objective, while the other works for discounted cost MDPs. Our actor-critic architecture incorporates parameterization both in the policy and the value function. A gradient search in the policy parameters is performed to improve the performance of the actor. The computation of the aforementioned gradient, however, requires an estimate of the value function of the policy corresponding to the current actor parameter. The value function, on the other hand, is approximated using linear function approximation and obtained from the critic. The error in approximation of the value function, however, results in suboptimal policies. In our article, we also update the features by performing a gradient descent on the Grassmannian of features to minimize a mean square Bellman error objective in order to find the best features. The aim is to obtain a good approximation of the value function and thereby ensure convergence of the actor to locally optimal policies. In order to estimate the gradient of the objective in the case of the average cost criterion, we utilize the policy gradient theorem, while in the case of the discounted cost objective, we utilize the simultaneous perturbation stochastic approximation (SPSA) scheme. We prove that our actor-critic algorithms converge to locally optimal policies. Experiments on two different settings show performance improvements resulting from our feature adaptation scheme.

Dynamic Energy Storage Control for Reducing Electricity Cost in Data Centers

As the scale of the data centers increases, electricity cost is becoming the fastest-growing element in their operation costs. In this paper, we investigate the electricity cost reduction opportunities utilizing energy storage facilities in data centers used as uninterrupted power supply units (UPS). Its basic idea is to combine the temporal diversity of electricity price and the energy storage to conceive a strategy for reducing the electricity cost. The electricity cost minimization is formulated in the framework of finite state-action discounted cost Markov decision process (MDP). We applyQ-Learning algorithm to solve the MDP optimization problem and derive a dynamic energy storage control strategy, which does not require any priori information on the Markov process. In order to address the slow-convergence problem of theQ-Learning based algorithm, we introduce a SpeedyQ-Learning algorithm. We further discuss the offline optimization problem and obtain the optimal offline solution as the lower bound on the performance of the online and learning theoretic problem. Finally, we evaluate the performance of the proposed scheme by using real workload traces and electricity price data sets. The experimental results show the effectiveness of the proposed scheme.

Q-Learning Based Energy Management Policies for a Single Sensor Node with Finite Buffer

In this paper, we consider the problem of finding optimal energy management policies in the presence of energy harvesting sources to maximize network performance. We formulate this problem in the discounted cost Markov decision process framework and apply two reinforcement learning algorithms. Prior work obtains optimal policy in the case when the conversion function mapping energy to data transmitted is linear and provides heuristic policies in the case when the same is nonlinear. Our algorithms, however, provide optimal policies regardless of the form of the conversion function. Through simulations, our policies are seen to outperform those of in the nonlinear case.

Reducing Electricity Cost of Smart Appliances via Energy Buffering Framework in Smart Grid

To reduce the long term electricity cost of smart appliances (SAs) with deferrable operation time in smart grid, we propose a novel energy buffering framework to intelligently schedule the distributed energy storage (DES) for the cost reduction of SAs in this paper. The proposed energy buffering framework determines the action policy (e.g., charging or discharging) and the power allocation policy of the DES to provide DES power to proper SAs at proper time with lower price than that of the utility grid, resulting in the reduction of the long term financial cost of SAs. Specifically, we first formulate the optimal decision problem in the energy buffering framework as a discounted cost Markov decision process (MDP) over infinite-horizon. Then, we propose an optimal scheme for the energy buffering framework to solve the discounted cost MDP based on online learning approach. Extensive simulation results show that the proposed optimal scheme for the energy buffering framework can significantly reduce the long term financial cost comparing with the baseline schemes and the myopic scheme.

Optimizing contracted resource capacity with two advance cancelation modes

Critical resources are often shared among different classes of customers. Capacity reservation allows each class of customers to better manage priorities of its customers but might lead to unused capacity. Unused capacity can be avoided or reduced by advance cancelation. This paper addresses the service capacity reservation for a given class of customers. The reservation process is characterized by: contracted time slots (CTS) reserved for the class of customers, requests for lengthy regular time slots (RTS) and two advance cancelation modes to cancel CTS one-period or two-period before. The optimal control under a given contract is formulated as an average cost Markov Decision Process (MDP) in order to minimize customer waiting times, unused CTS and CTS cancelation. Structural properties of optimal control policies are established via the corresponding discounted cost MDP problem. Numerical results show that two-period advance CTS cancelation can significantly improve the contract-based solution.

Markov decision processes with exponentially representable discounting

We generalize the geometric discount of finite discounted cost Markov Decision Processes to “exponentially representable”discount functions, prove existence of optimal policies which are stationary from some time N onward, and provide an algorithm for their computation. Outside this class, optimal “ N -stationary” policies in general do not exist.

Discounted Cost Markov Decision Processes with a Constraint

We consider a discounted cost Markov decision process with a constraint. Relating this to a vector-valued Markov decision process, we prove that there exists a constrained optimal randomized semistationary policy if there exists at least one policy satisfying a constraint. Moreover, we present an algorithm by which we can find the constrained optimal randomized semistationary policy, or we can discover that there exist no policies satisfying a given constraint.

Discounted Cost Markov Decision Processes on Borel Spaces: The Linear Programming Formulation

This paper is concerned with the linear programming formulation of Markov decision processes (or stochastic dynamic programs) with Borel state and action spaces and the discounted cost criterion. The one-stage cost function may be unbounded. A linear program and its dual are introduced, for which is shown the absence of a duality gap and that a strong duality condition holds. These results are used to determine the optimal value and the existence of an optimal policy for the discounted cost problem.

Suboptimal policy determination for large-scale Markov decision processes, Part 1: Description and bounds

This paper is the first of two papers that present and evaluate an approach for determining suboptimal policies for large-scale Markov decision processes (MDP). Part 1 is devoted to the determination of bounds that motivate the development and indicate the quality of the suboptimal design approach; Part 2 is concerned with the implementation and evaluation of the suboptimal design approach. The specific MDP considered is the infinite-horizon, expected total discounted cost MDP with finite state and action spaces. The approach can be described as follows. First, the original MDP is approximated by a specially structured MDP. The special structure suggests how to construct associated smaller, more computationally tractable MDP's. The suboptimal policy for the original MDP is then constructed from the solutions of these smaller MDP's. The key feature of this approach is that the state and action space cardinalities of the smaller MDP's are exponential reductions of the state and action space cardinalities of the original MDP.