Finite Horizon Stochastic Dynamic Programming Research Articles

Stochastic inventory routing with dynamic demands and intra-day depletion

Businesses such as supermarkets face an inventory routing problem with distinct stochastic dynamic demand distributions for various weekdays and times of the day. Most research, however, considers aggregated per-period demand and inventory depletion, thereby oversimplifying actual events. We formulate the problem as a finite-horizon stochastic dynamic program that accounts for the dynamic demand throughout the planning horizon and within each planning period. The models include different levels of information aggregation to decide the replenishment quantities and times by dividing periods into sub-periods with distinct demand distributions. This division helps the supplier create efficient replenishment plans and account for intra-day inventory levels. The modeled characteristics include routing, holding and stock-out costs, delivery time windows, inventory and vehicle capacities, and delivery time windows. The problem is solved using an iterative lookahead algorithm with an adaptive large neighborhood search for the routing and policy learning for the replenishment decisions. Numerical evaluations show the benefit of intra-day planning, effects of instance sizes, different coefficients of variation, and holding costs compared to benchmark replenishment policies. Considering intra-day depletion and demand data for each sub-period helps achieve savings of more than 20% compared to planning for full periods. The savings are obtained by adjusting delivery quantities, incorporating intra-day consumption, creating more efficient delivery routes, and correctly timing replenishments.

Dynamic ordering decisions with approximate learning of supply yield uncertainty

We consider the real-life problem of a coach bus manufacturer located in Turkey, facing the problem of setting ordering quantities for a part procured from an unreliable supplier, where the number of items delivered is binomially distributed with an unknown yield parameter, p. We use the well-defined finite-horizon planning context with deterministic demand per period, purchasing, holding, and shortage costs to investigate the effectiveness of a fill-rate based approximate learning scheme in comparison to an exact Bayesian learning scheme, where observations on the supplier's delivery performance are used to update the assumed distribution of p. We formulate the exact optimal learning problem as a Bayes-adaptive Markov decision process and solve the corresponding finite horizon stochastic dynamic program to provide insights on the value of online learning in comparison to the unrealistic perfect information (PI) and no information (NT) benchmarks. We contrast the performance of the so-called Bayesian Updating (BU) policy to other practical approaches such as using an assumed/guessed value of p and implementing a constant safety stock. Noting the significant value of learning, we finally study the effectiveness of an approximate learning formulation that does not enjoy the asymptotic consistency and convergence properties but involves much lower computational burden, and demonstrate its confounding performance, at times beating the BU policy with exact Bayesian updates.

Improved Crop Productivity Through Optimized Planting Schedules

Problem definition: We study the optimal seeding policy under rainfall uncertainty in rain-fed agriculture and explore its advantage over commonly used heuristics in practice. Academic/practical relevance: Our work is in the area of agriculture operations, and we focus on the improvement of farmer’s expected total profit by optimizing planting schedules. Methodology: We model a farmer’s planting problem under limited planting capacity in a finite-horizon stochastic dynamic program. Results: We show that the optimal planting policy is a time-dependent, threshold-type policy, and the optimal threshold is dependent on the soil water content and planting capacity. In our computational study, we use the well-known Decision Support System for Agrotechnology Transfer simulator used in agriculture to obtain the expected yield when planting in any given period. Utilizing field weather data from Southern Africa, in a real-size, large-scale problem, we demonstrate a significant relative profit advantage of the optimal planting schedule over commonly used heuristics in practice. We show that the relative advantage of the optimal policy increases as the climate condition becomes more severe for planting. We also develop a heuristic based on the secretary problem and demonstrate the increased efficacy of the secretary heuristic. Managerial implications: We show that the farmer should plant down to the optimal threshold of seed amount. However, in practice, farmers start to plant each year after observing enough cumulative rainfall. Utilizing field weather data, in a real-size, large-scale problem, we show significant improvement of the expected total profit if the farmer could adopt the optimal policy.

Joint maintenance and spare parts inventory optimization for multi-unit systems considering imperfect maintenance actions

Joint maintenance and spare parts inventory optimization has attracted increasing attention in recent years because of its capability in addressing the maintenance planning and the spare parts provisioning of industrial systems simultaneously. However, imperfect maintenance (IM) actions are either neglected or over-simplified as constant improvements in existing studies, which reduces their practicality in industrial applications. To tackle this limitation, this paper investigates the joint maintenance and spare parts inventory optimization for multi-unit systems with the consideration of IM actions as random improvement factors. First, a two-step approximate derivation method is proposed, which overcomes the derivation difficulties of replacement numbers due to the introduction of random improvement factors and enables the construction of the inventory level transition relationship. Then based on the inventory level transition relationship, an expected total cost model is formulated via the finite horizon stochastic dynamic programming (FHSDP). The decision variables are optimized by the joint use of enumeration and the FHSDP. Finally, a numerical simulation of a wind farm is carried out for illustration. Sensitivity analyses are further conducted to study the influences of critical parameters.

Power management optimization in plug-in hybrid electric vehicles subject to uncertain driving cycles

Abstract Optimization of power management in plug-in hybrid electric vehicles (PHEVs) with dual-power-source plays a critical role in achieving higher fuel economy and less pollutant emissions. In this study, power management and optimal control strategies in PHEVs have been investigated subject to uncertain driving cycles of individual drivers for particular trips. First, a stochastic driving cycle is constructed to more accurately model the dynamic characteristics of the uncertain driving cycles, derived from the historic record of individual drivers. Finite-horizon stochastic dynamic programming is adapted to globally optimize the vehicle performance in stochastic sense. Simulation results show that the proposed strategy significantly improves fuel economy, indicating the present optimization approach is very effective in exploring the potential of the hybridization of power train. A higher discretization of (that is, with smaller step sizes in) vehicle dynamics state variables (vehicle velocity, power demand and battery state of charge) has a positive impact on the fuel economy while the limitation of driving operability actually degrades the fuel economy. The commuting time with doubly truncated normal distribution slightly enhances the fuel economy in comparison with uniform distribution. In addition, there exists a tradeoff between the fuel economy and the pollutant emissions. These results could be utilized as a guideline for the design of PHEVs with different objectives.

Managing virtual appointments in chronic care

Virtual appointments between patients and healthcare providers can offer a cost-effective alternative to traditional office appointments for managing chronic conditions. Virtual appointments increase contact with the physician by either substituting or complementing office appointments, leading to improved health outcomes. The true value of virtual appointments cannot be realized until they are truly integrated with the office appointment systems. In this study, we introduce a capacity allocation model to study the use of virtual appointments in a chronic care setting. Specifically, we develop a finite horizon stochastic dynamic program to determine which patients to schedule for office and virtual appointments that maximizes aggregate health benefits across a cohort of patients. Optimal policy characterization for this problem is challenging. We find that, under certain conditions, a myopic heuristic, where the sickest patients are scheduled for office appointments and the next sickest patients are scheduled for virtual appointments, is optimal. We show that the myopic heuristic performs well even in more general settings. Our findings further show that virtual appointments serve a dual purpose: they may reduce the number of office appointments and may trigger follow-up office appointments.

Staff assignment policies for a mass casualty event queuing network

We study parallel queuing systems in which heterogeneous teams collaborate to serve queues with three different prioritization levels in the context of a mass casualty event. We assume that the health condition of casualties deteriorate as time passes and aim to minimize total deprivation cost in the system. Servers (i.e. doctors and nurses) have random arrival rates and they are assigned to a queue as soon as they arrive. While nurses and doctors serve their dedicated queues, collaborative teams of doctors and nurses serve a third type of customer, the patients in critical condition. We model this queueing network with flexible resources as a discrete-time finite horizon stochastic dynamic programming problem and develop heuristic policies for it. Our results indicate that the standard $$c \mu $$ rule is not an optimal policy, and that the most effective heuristic policy found in our simulation study is intuitive and has a simple structure: assign doctor/nurse teams to clear the critical patient queue with a buffer of extra teams to anticipate future critical patients, and allocate the remaining servers among the other two queues.

A Markov chain model of military personnel dynamics

Personnel retention is one of the most significant challenges faced by the US Army. Central to the problem is understanding the incentives of the stay-or-leave decision for military personnel. Using three years of data from the US Department of Defense, we construct and estimate a Markov chain model of military personnel. Unlike traditional classification approaches, such as logistic regression models, the Markov chain model allows us to describe military personnel dynamics over time and answer a number of managerially relevant questions. Building on the Markov chain model, we construct a finite-horizon stochastic dynamic programming model to study the monetary incentives of stay-or-leave decisions. The dynamic programming model computes the expected pay-off of staying versus leaving at different stages of the career of military personnel, depending on employment opportunities in the civilian sector. We show that the stay-or-leave decisions from the dynamic programming model possess surprisingly strong predictive power, without requiring personal characteristics that are typically employed in classification approaches. Furthermore, the results of the dynamic programming model can be used as an input in classification methods and lead to more accurate predictions. Overall, our work presents an interesting alternative to classification methods and paves the way for further investigations on personnel retention incentives.

Customer differentiated end-of-life inventory problem

This paper deals with the service parts end-of-life inventory problem in a circumstance that demands for service parts are differentiated. Customer differentiation might be due to criticality of the demand or based on various service contracts. In both cases, we model the problem as a finite horizon stochastic dynamic program and characterize the structure of the optimal inventory policy. We show that when customers are differentiated based on the demand criticality then the optimal structure consists of time and state dependent threshold levels for inventory rationing. In case of differentiation based on service contracts, we show that in addition to rationing thresholds we also need contract extension thresholds by which the system decides whether to offer an extension to an expiring contract or not. By numerical experiments in both cases, we identify the value of incorporating such decisions in service parts end-of-life inventory management with customer differentiation. Moreover, we show that these decisions not only result in cost efficiency but also decrease the risk of part obsolescence drastically.

Improving Access to Community-Based Chronic Care Through Improved Capacity Allocation

This paper studies a model of community-based healthcare delivery for a chronic disease. In this setting, patients periodically visit the healthcare delivery system, which influences their disease progression and consequently their health outcomes. We investigate how the provider can maximize community-level health outcomes through better operational decisions pertaining to capacity allocation across different patients. To do so, we develop an integrated capacity allocation model that incorporates clinical (disease progression) and operational (capacity constraint) aspects. Specifically, we model the provider's problem as a finite horizon stochastic dynamic program, where the provider decides which patients to schedule at the beginning of each period. Therapy is provided to scheduled patients, which may improve their health states. Patients that are not seen follow their natural disease progression. We derive a quantitative measure for comparison of patients' health states and use it to design an easy-to-implement myopic heuristic that is provably optimal in special cases of the problem. We employ the myopic heuristic in a more general setting and test its performance using operational and clinical data obtained from Mobile C.A.R.E. Foundation, a community-based provider of pediatric asthma care in Chicago. Our extensive computational experiments suggest that the myopic heuristic can improve the health gains at the community level by up to 15% over the current policy. The benefit is driven by the ability of our myopic heuristic to alter the duration between visits for patients with different health states depending on the tightness of the capacity and the health states of the entire patient population.

Improving Health Outcomes Through Better Capacity Allocation in a Community-Based Chronic Care Model

This paper studies a model of community-based healthcare delivery for a chronic disease. In this setting, patients periodically visit the healthcare delivery system, which influences their disease progression and consequently their health outcomes. We investigate how the provider can maximize community-level health outcomes through better operational decisions pertaining to capacity allocation across different patients. To do so, we develop an integrated capacity allocation model that incorporates clinical (disease progression) and operational (capacity constraint) aspects. Specifically, we model the provider's problem as a finite horizon stochastic dynamic program, where the provider decides which patients to schedule at the beginning of each period. Therapy is provided to scheduled patients, which may improve their health states. Patients that are not seen follow their natural disease progression. We derive a quantitative measure for comparison of patients' health states and use it to design an easy-to-implement myopic heuristic that is provably optimal in special cases of the problem. We employ the myopic heuristic in a more general setting and test its performance using operational and clinical data obtained from Mobile C.A.R.E. Foundation, a community-based provider of pediatric asthma care in Chicago. Our extensive computational experiments suggest that the myopic heuristic can improve the health gains at the community level by up to 15% over the current policy. The benefit is driven by the ability of our myopic heuristic to alter the duration between visits for patients with different health states depending on the tightness of the capacity and the health states of the entire patient population.

An Asymptotically Efficient Simulation-Based Algorithm for Finite Horizon Stochastic Dynamic Programming

We present a simulation-based algorithm called Simulated Annealing Multiplicative Weights (SAMW) for solving large finite-horizon stochastic dynamic programming problems. At each iteration of the algorithm, a probability distribution over candidate policies is updated by a simple multiplicative weight rule, and with proper annealing of a control parameter, the generated sequence of distributions converges to a distribution concentrated only on the best policies. The algorithm is asymptotically efficient, in the sense that for the goal of estimating the value of an optimal policy, a provably convergent finite-time upper bound for the sample mean is obtained