Dynamic Programming Policy Research Articles

A simple heuristic for perishable item inventory control under non-stationary stochastic demand

In this paper, we study the single-item single-stocking location non-stationary stochastic lot sizing problem for a perishable product. We consider fixed and proportional ordering cost, holding cost and penalty cost. The item features a limited shelf life, therefore we also take into account a variable cost of disposal. We derive exact analytical expressions to determine the expected value of the inventory of different ages. We also discuss a good approximation for the case in which the shelf-life is limited. To tackle this problem, we introduce two new heuristics that extend Silver’s heuristic and compare them to an optimal Stochastic Dynamic Programming policy in the context of a numerical study. Our results demonstrate the effectiveness of our approach.

Performance Guarantee of an Approximate Dynamic Programming Policy for Robotic Surveillance

This paper is focused on the development and analysis of suboptimal decision algorithms for a collection of robots that assist a remotely located operator in perimeter surveillance. The operator is tasked with the classification of incursions across the perimeter. whenever there is an incursion into the perimeter, an unattended ground sensor (UGS) in the vicinity, signals an alert. A robot services the alert by visiting the alert location, collecting information, e.g., photo and video imagery, and transmitting it to the operator. The accuracy of operator's classification depends on the volume and freshness of information gathered and provided by the robots at locations where incursions occur. There are two competing objectives for a robot: it needs to spend adequate time at an alert location to collect evidence for aiding the operator in accurate classification but it also needs to service other alerts as soon as possible, so that the evidence collected is relevant. The decision problem is to determine the optimal amount of time a robot must spend servicing an alert. The incursions are stochastic and their statistics are assumed to be known. This problem can be posed as a Markov Decision Problem. However, even for two robots and five UGS locations, the number of states is of the order of billions rendering exact dynamic programming methods intractable. Approximate dynamic programming (ADP) via linear programming (LP) provides a way to approximate the value function and derive suboptimal strategies. The novel feature of this paper is the derivation of a tractable lower bound via LP and the construction of a suboptimal policy whose performance improves upon the lower bound. An illustrative perimeter surveillance example corroborates the results derived in this paper.

Robust stochastic optimization for reservoir operation

Optimal reservoir operation under uncertainty is a challenging engineering problem. Application of classic stochastic optimization methods to large-scale problems is limited due to computational difficulty. Moreover, classic stochastic methods assume that the estimated distribution function or the sample inflow data accurately represents the true probability distribution, which may be invalid and the performance of the algorithms may be undermined. In this study, we introduce a robust optimization (RO) approach, Iterative Linear Decision Rule (ILDR), so as to provide a tractable approximation for a multiperiod hydropower generation problem. The proposed approach extends the existing LDR method by accommodating nonlinear objective functions. It also provides users with the flexibility of choosing the accuracy of ILDR approximations by assigning a desired number of piecewise linear segments to each uncertainty. The performance of the ILDR is compared with benchmark policies including the sampling stochastic dynamic programming (SSDP) policy derived from historical data. The ILDR solves both the single and multireservoir systems efficiently. The single reservoir case study results show that the RO method is as good as SSDP when implemented on the original historical inflows and it outperforms SSDP policy when tested on generated inflows with the same mean and covariance matrix as those in history. For the multireservoir case study, which considers water supply in addition to power generation, numerical results show that the proposed approach performs as well as in the single reservoir case study in terms of optimal value and distributional robustness.

Application of ESN-based Multi Indices Dual Heuristic Dynamic Programming on Wastewater Treatment Process

摘要: 针对污水处理过程(Wastewater treatment process, WWTP)溶解氧(Dissolved oxygen, DO)及硝态氮浓度控制问题, 提出了一种多评价指标的DHP (Dual heuristic dynamic programming)控制策略. 该策略能够降低评价指标的复杂性, 提高评价网络的逼近精度. 采用回声状态网络(Echo state networks, ESNs)实现评价函数及控制策略的逼近, 研究了控制器的在线学习算法. 实验表明, 该策略在控制性能上优于单评价指标的DHP策略及常规PID控制策略. 关键词: 自适应动态规划 / 多评价指标 / 污水处理 / 回声状态网络

Tuning Approximate Dynamic Programming Policies for Ambulance Redeployment via Direct Search

In this paper we consider approximate dynamic programming methods for ambulance redeployment. We first demonstrate through simple examples how typical value function fitting techniques, such as approximate policy iteration and linear programming, may not be able to locate a high-quality policy even when the value function approximation architecture is rich enough to provide the optimal policy. To make up for this potential shortcoming, we show how to use direct search methods to tune the parameters in a value function approximation architecture so as to obtain high-quality policies. Direct search is computationally intensive. We therefore use a post-decision state dynamic programming formulation of ambulance redeployment that, together with direct search, requires far less computation with no noticeable performance loss. We provide further theoretical support for the post-decision state formulation of the ambulance-deployment problem by showing that this formulation can be obtained through a limiting argument on the original dynamic programming formulation.

Information‐Sensitive Replenishment when Inventory Records Are Inaccurate

Inspired by recent empirical work on inventory record inaccuracy, we consider a periodic review inventory system with imperfect inventory records and unobserved lost sales. Record inaccuracies are assumed to arrive via an error process that perturbs physical inventory but is unobserved by the inventory manager. The inventory manager maintains a probability distribution around the physical inventory level that he updates based on sales observations using Bayes Theorem. The focus of this study is on understanding, approximating, and evaluating optimal forward‐looking replenishment in this environment. By analyzing one‐ and two‐period versions of the problem, we demonstrate several mechanisms by which the error process and associated record inaccuracy can impact optimal replenishment. Record inaccuracy generally brings an incentive for a myopic manager to increase stock to buffer the added uncertainty. On the other hand, a forward‐looking manager will stock less than a myopic manager, in part to improve information content for future decisions. Using an approximate partially observed dynamic programming policy and associated bound, we numerically corroborate our analytical findings and measure the effectiveness of an intelligent myopic heuristic. We find that the myopic heuristic is likely sufficiently good in practical settings targeting high service levels.

Analyzing Screening Policies for Childhood Obesity.

Due to the health and economic costs of childhood obesity, coupled with studies suggesting the benefits of comprehensive (dietary, physical activity and behavioral counseling) intervention, the United States Preventive Services Task Force recently recommended childhood screening and intervention for obesity beginning at age six. Using a longitudinal data set consisting of the body mass index of 3164 children up to age 18 and another longitudinal data set containing the body mass index at ages 18 and 40 and the presence or absence of disease (hypertension and diabetes) at age 40 for 747 people, we formulate and numerically solve - separately for boys and girls - a dynamic programming problem for the optimal biennial (i.e., at ages 2, 4, …, 16) obesity screening thresholds. Unlike most screening problem formulations, we take a societal viewpoint, where the state of the system at each age is the population-wide probability density function of the body mass index. Compared to the biennial version of the task force's recommendation, the screening thresholds derived from the dynamic program achieve a relative reduction in disease prevalence of 3% at the same screening (and treatment) cost, or - due to the flatness of the disease vs. screening tradeoff curve - achieves the same disease prevalence at a 28% relative reduction in cost. Compared to the task force's policy, which uses the 95th percentile of body mass index (from cross-sectional growth charts tabulated by the Centers for Disease Control and Prevention) as the screening threshold for each age, the dynamic programming policy treats mostly 16 year olds (including many who are not obese) and very few males under 14 years old. While our results suggest that adult hypertension and diabetes are minimized by focusing childhood obesity screening and treatment on older adolescents, the shortcomings in the available data and the narrowness of the medical outcomes considered prevent us from making a recommendation about childhood obesity screening policies.

Dynamic Capacity Allocation to Customers Who Remember Past Service

We study the problem faced by a supplier deciding how to dynamically allocate limited capacity among a portfolio of customers who remember the fill rates provided to them in the past. A customer's order quantity is positively correlated with past fill rates. Customers differ from one another in their contribution margins, their sensitivities to the past, and in their demand volatilities. By analyzing and comparing policies that ignore goodwill with ones that account for it, we investigate when and how customer memory effects impact supplier profits. We develop an approximate dynamic programming policy that dynamically rationalizes the fill rates the firm provides to each customer. This policy achieves higher rewards than margin-greedy and Lagrangian policies and yields insights into how a supplier can effectively manage customer memories to its advantage. This paper was accepted by Martin Lariviere, operations management.

Persistently Optimal Policies in Stochastic Dynamic Programming with Generalized Discounting

In this paper we study a Markov decision process with a nonlinear discount function. First, we define a utility on the space of trajectories of the process in the finite and infinite time horizon and then take their expected values. It turns out that the associated optimization problem leads to a nonstationary dynamic programming and an infinite system of Bellman equations, which result in obtaining persistently optimal policies. Our theory is enriched by examples.

Tuning approximate dynamic programming policies for ambulance redeployment via direct search

Open AccessOpen Access licenseAboutSectionsView PDF ToolsAdd to favoritesDownload CitationsTrack Citations ShareShare onFacebookTwitterLinked InEmail Go to SectionOpen AccessOpen Access license HomeStochastic SystemsVol. 3, No. 2 Tuning Approximate Dynamic Programming Policies for Ambulance Redeployment via Direct SearchMatthew S. Maxwell, Shane G. Henderson, Huseyin TopalogluMatthew S. Maxwell, Shane G. Henderson, Huseyin TopalogluPublished Online:15 Nov 2013https://doi.org/10.1287/10-SSY020AbstractIn this paper we consider approximate dynamic programming methods for ambulance redeployment. We first demonstrate through simple examples how typical value function fitting techniques, such as approximate policy iteration and linear programming, may not be able to locate a high-quality policy even when the value function approximation architecture is rich enough to provide the optimal policy. To make up for this potential shortcoming, we show how to use direct search methods to tune the parameters in a value function approximation architecture so as to obtain high-quality policies. Direct search is computationally intensive. We therefore use a post-decision state dynamic programming formulation of ambulance redeployment that, together with direct search, requires far less computation with no noticeable performance loss. We provide further theoretical support for the post-decision state formulation of the ambulance-deployment problem by showing that this formulation can be obtained through a limiting argument on the original dynamic programming formulation. Back to Top Next FiguresReferencesRelatedInformationCited byInpatient Overflow: An Approximate Dynamic Programming ApproachJ. G. Dai, Pengyi Shi16 May 2019 | Manufacturing & Service Operations Management, Vol. 21, No. 4A Unified Framework for Optimization Under UncertaintyWarren B. Powell4 November 2016The Minimum Expected Penalty Relocation Problem for the Computation of Compliance Tables for Ambulance VehiclesThije van Barneveld21 April 2016 | INFORMS Journal on Computing, Vol. 28, No. 2Discrete Optimization Models for Homeland Security and Disaster ManagementLaura Albert McLay26 October 2015Simulation-Based Approximate Policy Iteration with Generalized Logistic FunctionsAntoine Sauré, Jonathan Patrick, Martin L. Puterman28 September 2015 | INFORMS Journal on Computing, Vol. 27, No. 3Technical Note—Trading Off Quick versus Slow Actions in Optimal SearchSteven M. Shechter, Farhad Ghassemi, Yasin Gocgun, Martin L. Puterman6 March 2015 | Operations Research, Vol. 63, No. 2A Bound on the Performance of an Optimal Ambulance Redeployment PolicyMatthew S. Maxwell, Eric Cao Ni, Chaoxu Tong, Shane G. Henderson, Huseyin Topaloglu, Susan R. Hunter16 July 2014 | Operations Research, Vol. 62, No. 5 Volume 3, Issue 2December 2013Pages 322-633 Article Information Metrics Information Received:November 01, 2010Published Online:November 15, 2013 Copyright © 2013, The author(s)Cite asMatthew S. Maxwell, Shane G. Henderson, Huseyin Topaloglu (2013) Tuning Approximate Dynamic Programming Policies for Ambulance Redeployment via Direct Search. Stochastic Systems 3(2):322-361. https://doi.org/10.1287/10-SSY020 PDF download

Quadratic approximate dynamic programming for input‐affine systems

SUMMARYWe consider the use of quadratic approximate value functions for stochastic control problems with input‐affine dynamics and convex stage cost and constraints. Evaluating the approximate dynamic programming policy in such cases requires the solution of an explicit convex optimization problem, such as a quadratic program, which can be carried out efficiently. We describe a simple and general method for approximate value iteration that also relies on our ability to solve convex optimization problems, in this case, typically a semidefinite program. Although we have no theoretical guarantee on the performance attained using our method, we observe that very good performance can be obtained in practice.Copyright © 2012 John Wiley & Sons, Ltd.

Dynamic policy programming

In this paper, we propose a novel policy iteration method, called dynamic policy programming (DPP), to estimate the optimal policy in the infinite-horizon Markov decision processes. DPP is an incremental algorithm that forces a gradual change in policy update. This allows us to prove finite-iteration and asymptotic l∞-norm performance-loss bounds in the presence of approximation/ estimation error which depend on the average accumulated error as opposed to the standard bounds which are expressed in terms of the supremum of the errors. The dependency on the average error is important in problems with limited number of samples per iteration, for which the average of the errors can be significantly smaller in size than the supremum of the errors. Based on these theoretical results, we prove that a sampling-based variant of DPP (DPP-RL) asymptotically converges to the optimal policy. Finally, we illustrate numerically the applicability of these results on some benchmark problems and compare the performance of the approximate variants of DPP with some existing reinforcement learning (RL) methods.

In This Issue

In This Issue

Integrated Optimization of Procurement, Processing, and Trade of Commodities

We consider the integrated optimization problem of procurement, processing, and trade of commodities in a multiperiod setting. Motivated by the operations of a prominent commodity processing firm, we model a firm that procures an input commodity and has processing capacity to convert the input into a processed commodity. The processed commodity is sold using forward contracts, while the input itself can be traded at the end of the horizon. We solve this problem optimally and derive closed-form expressions for the marginal value of input and output inventory. We find that the optimal procurement and processing decisions are governed by price-dependent inventory thresholds. We use commodity markets data for the soybean complex to conduct numerical studies and find that approximating the joint price processes of multiple output commodities using a single, composite output product and using the approximate price process to determine procurement and processing decisions is near optimal. Compared to a myopic spread-option-based heuristic, the optimization-based dynamic programming policy provides significant benefits under conditions of tight processing capacities and high price volatilities. Finally, we propose an approximation procedure to compute heuristic policies and an upper bound to compare the heuristic against, when commodity prices follow multifactor processes.

A Dynamic Programming Policy Improvement Approach to the Development of Maintenance Policies for 2-Phase Systems With Aging

The deterioration observed in many industrial systems may be modeled in two phases. In the first phase, a period during which the system operates fault free ends with entry into a worn state. In the second phase, the system spends time in the worn state prior to failure. Should the system be found to be in the worn state upon inspection, failure can be pre-empted by preventive maintenance. Transitions into the worn state occur more frequently as the system ages, as does the cost of maintaining the system. The goal of analysis is the design of cost-effective policies for the inspection, maintenance, and renewal of such systems. The paper extends previous work by offering a choice between a (cheap) repair and a (more expensive) renewal of the system, should it be found to be in the worn state upon inspection. The decision-maker may also renew the system at any time without inspection. We propose simple, cost effective heuristic policies, whose design avoids the computational complexities of a full dynamic programming (DP) solution. The closeness to optimality of these policies is investigated, as is their sensitivity to parameter misspecification. A numerical investigation identifies cases for which the inclusion of the repair option is most advantageous.

Compactness of the space of non-randomized policies in countable-state sequential decision processes

For sequential decision processes with countable state spaces, we prove compactness of the set of strategic measures corresponding to nonrandomized policies. For the Borel state case, this set may not be compact (Piunovskiy, Optimal control of random sequences in problems with constraints. Kluwer, Boston, p. 170, 1997) in spite of compactness of the set of strategic measures corresponding to all policies (Schäl, On dynamic programming: compactness of the space of policies. Stoch Processes Appl 3(4):345–364, 1975b; Balder, On compactness of the space of policies in stochastic dynamic programming. Stoch Processes Appl 32(1):141–150, 1989). We use the compactness result from this paper to show the existence of optimal policies for countable-state constrained optimization of expected discounted and nonpositive rewards, when the optimality is considered within the class of nonrandomized policies. This paper also studies the convergence of a value-iteration algorithm for such constrained problems.

DYNAMIC ROUTING OF CUSTOMERS WITH GENERAL DELAY COSTS IN A MULTISERVER QUEUING SYSTEM

We consider a network of parallel service stations each modeled as a single-server queue. Each station serves its own dedicated customers as well as generic customers who are routed from a central controller. We suppose that the cost incurred by a customer is an increasing function of her time spent in the system. In a significant advance on most previous work, we do not require waiting costs to be convex, still less linear. With the objective of minimizing the long-run average waiting cost, we develop two heuristic routing policies, one of which is based on dynamic programming policy improvement and the other on Lagrangian relaxation. In developing the latter policy, we show that each station is “indexable” under mild conditions for customers’ waiting costs and also prove some structural results on the admission control problem that naturally arises as a result of the Lagrangian relaxation. We then test the performance of our heuristics in an extensive numerical study and show that the Lagrangian heuristic demonstrates a strong level of performance in a range of traffic conditions. In particular, it clearly outperforms both a greedy heuristic, which is a standard proposal in complex routing problems, and a recent proposal from the heavy traffic literature.

Allocation Models and Heuristics for the Outsourcing of Repairs for a Dynamic Warranty Population

We consider a scenario in which a large equipment manufacturer wishes to outsource the work involved in repairing purchased goods while under warranty. Several external service vendors are available for this work. We develop models and analyses to support decisions concerning how responsibility for the warranty population should be divided between them. These also allow the manufacturer to resolve related questions concerning, for example, whether the service capacities of the contracted vendors are sufficient to deliver an effective post-sales service. Static allocation models yield information concerning the proportions of the warranty population for which the vendors should be responsible overall. Dynamic allocation models enable consideration of how such overall workloads might be delivered to the vendors over time in a way which avoids excessive variability in the repair burden. We apply dynamic programming policy improvement to develop an effective dynamic allocation heuristic. This is evaluated numerically and is also used as a yardstick to assess two simple allocation heuristics suggested by static models. A dynamic greedy allocation heuristic is found to perform well. Dividing the workload equally among vendors with different service capacities can lead to serious losses.

Dynamic policies for uncertain time‐critical tasking problems

AbstractA recent paper by Gaver et al. 6 argued the importance of studying service control problems in which the usual assumptions (i) that tasks will wait indefinitely for service and (ii) that successful service completions can be observed instantaneously are relaxed. Military and other applications were cited. They proposed a model in which arriving tasks are available for service for a period whose duration is unknown to the system's controller. The allocation of a large amount of processing to a task may make more likely its own successful completion but may also result in the loss of many unserved tasks from the system. Gaver et al. 6 called for the design of dynamic policies for the allocation of service which maximizes the rate of successful task completions achieved, or which come close to doing so. This is the theme of the paper. We utilize dynamic programming policy improvement approaches to design heuristic dynamic policies for service allocation which may be easily computed. In all cases studied, these policies achieve throughputs close to optimal. © 2008 Wiley Periodicals, Inc. Naval Research Logistics, 2008

Remarks on sensitive equilibria in stochastic games with additive reward and transition structure

A class of stochastic games with additive reward and transition structure is studied. For zero-sum games under some ergodicity assumptions 1-equilibria are shown to exist. They correspond to so-called sensitive optimal policies in dynamic programming. For a class of nonzero-sum stochastic games with nonatomic transitions nonrandomized Nash equilibrium points with respect to the average payoff criterion are also obtained. Included examples show that the results of this paper can not be extented to more general payoff or transition structure.