Stochastic Dynamic Programming Framework Research Articles

Economic optimization of Wolbachia-infected Aedes aegypti release to prevent dengue.

Dengue virus, primarily transmitted by the Aedes aegypti mosquito, is a major public health concern affecting ≈3.83 billion people worldwide. Recent releases of Wolbachia-transinfected Ae. aegypti in several cities worldwide have shown that it can reduce dengue transmission. However, these releases are costly, and, to date, no framework has been proposed for determining economically optimal release strategies that account for both costs associated with disease risk and releases. We present a flexible stochastic dynamic programming framework for determining optimal release schedules for Wolbachia-transinfected mosquitoes that balances the cost of dengue infection with the costs of rearing and releasing transinfected mosquitoes. Using an ordinary differential equation model of Wolbachia and dengue in a hypothetical city loosely describing areas at risk of new dengue epidemics, we determined that an all-or-nothing release strategy that quickly brings Wolbachia to fixation is often the optimal solution. Based on this, we examined the optimal facility size, finding that it was inelastic with respect to the mosquito population size, with a 100% increase in population size resulting in a 50-67% increase in optimal facility size. Furthermore, we found that these results are robust to mosquito life-history parameters and are mostly determined by the mosquito population size and the fitness costs associated with Wolbachia. These results reinforce that Wolbachia-transinfected mosquitoes can reduce the cost of dengue epidemics. Furthermore, they emphasize the importance of determining the size of the target population and fitness costs associated with Wolbachia before releases occur. © 2024 The Authors. Pest Management Science published by John Wiley & Sons Ltd on behalf of Society of Chemical Industry.

Multilayer Iterative Stochastic Dynamic Programming for Optimal Energy Management of Residential Loads with Electric Vehicles

This work introduces a multilayer iterative stochastic dynamic programming (MISDP) framework for optimizing energy management in smart residential settings, incorporating electric vehicles to reduce energy costs while enhancing operational efficiency. The study investigates the complexities of managing residential loads with integrated EV batteries, set against the backdrop of unpredictable charging demands and fluctuating energy prices. The proposed method is designed to optimize charging and discharging schedules, ensuring cost‐effective energy consumption without compromising the longevity of EV’s battery operations. The proposed MISDP strategy encompasses multi‐iteration processes, both at internal and external levels, that not only highlight the method’s capacity for precise, real‐time decision‐making but also underscore its adaptability to the dynamic nature of energy systems. The external iteration primarily focuses on adapting to broader operational variables, such as fluctuating prices and demand patterns, setting a framework for optimization. Concurrently, the internal iteration updates the details of EV battery operation, fine‐tuning charging and discharging strategies to refine the control law sequence for each operational period, ensuring optimal energy management. Throughout the iteration process, the framework ensures the performance index function remains bounded, adhering strictly to the evolving control law sequence. Through comparative analysis, the MISDP framework is evaluated against different optimization techniques, demonstrating its superior capability in achieving significant energy cost savings and operational effectiveness while ensuring convergence under stochastic conditions.

The small-bat-in-summer paradigm: Energetics and adaptive behavioural routines of bats investigated through a stochastic dynamic model.

Strong seasonality at high latitudes represents a major challenge for many endotherms as they must balance survival and reproduction in an environment that varies widely in food availability and temperature. To avoid energetic mismatches caused by limited foraging time and stochastic weather conditions, bats employ the energy-saving state of torpor during summer to save accumulated energy reserves. However, at high-latitude small-bats-in-summer face a particular challenge: as nocturnal foragers, they rely on the darkness at night to avoid predators and/or interspecific competition, but live in an environment with short, light summer nights, and even a lack of true night at the northernmost distributions of some bat species. To predict optimal behaviour in relation to latitudinal variation in diurnal cycles, we constructed a stochastic dynamic programming model of bats living at high latitudes. Using a stochastic dynamic programming framework with values that are representative for our study system, we show that individual energetic reserves are a strong driver of daytime use of torpor and night-time foraging behaviour alike, with these linked effects being both temperature- and photoperiod-dependent. We further used the model to predict survival probabilities at five locations across a latitudinal gradient (60.1° N to 70.9° N), finding that combinations of photoperiod and temperature conditions limited population distributions in the model. To verify our model results, we compared predictions for optimal decisions with our own empirical data collected on northern bats (Eptesicus nilssonii) from two latitudes in Norway. The similarities between our predictions and observations provide strong evidence that this model framework incorporates the most important drivers of diurnal decision-making in bat physiology and behaviour. Comparing empirical data and model predictions also revealed that bats facing lighter night conditions further north restrict their mass gain, which strengthens the hypothesis that predation threat is a main driver of bat nocturnality. Our model findings regarding state-dependent decisions in bats should contribute to the understanding of how bats cope with the summer challenges at high latitudes.

Large-scale financial planning via a partially observable stochastic dual dynamic programming framework

The multi-stage stochastic programming (MSP) approach is widely used to solve financial planning problems owing to its flexibility. However, the size of an MSP problem grows exponentially with the number of stages, and such problem can easily become computationally intractable. Financial planning problems often consider planning horizons of several decades, and thus, the curse of dimensionality can become a critical issue. Stochastic dual dynamic programming (SDDP), a sampling-based decomposition algorithm, has emerged to resolve this issue. While SDDP has been successfully implemented in the energy domain, few applications of SDDP are found in the finance domain. In this study, we identify the major obstacle in using SDDP to solve financial planning problems to be the stagewise independence assumption and propose a partially observable SDDP (PO-SDDP) framework to overcome such limitations. We argue that the PO-SDDP framework, which models uncertainties using discrete-valued partially observable Markov states and introduces feasibility cuts, can properly address large-scale financial planning problems.

Two-timescale coordinated operation of wind-advanced adiabatic compressed air energy storage system: A bilevel stochastic dynamic programming method

Renewable energy is a promising solution to address the energy crisis and environmental issues, but it comes with challenges due to its inherent volatility and limited dispatchability. Advanced adiabatic compressed air energy storage (AA-CAES) is a favorable partner for centralized renewable integration, due to its numerous benefits, such as large capacity, long lifetime, fast response capability, and zero carbon emissions. For the economic management of a wind-AACAES system, a battery-like AA-CAES dispatch model is proposed, where the charging/discharging efficiencies and capacities are dependent on the operating power and state-of-charge (SoC) to capture the part-load characteristics. A hybrid control strategy is proposed to enhance the flexibility and efficiency of the compression side under off-design conditions caused by simultaneous changes in back pressure and load. To provide dispatch and control strategies of AA-CAES, a multi-timescale optimization problem is established. The slow timescale determines the baseline output scheduling on an hourly basis to maximize total revenue, while the fast timescale updates the real-time adjustment every five minutes to minimize the penalty of tracking error. A bi-level stochastic dynamic programming (SDP) framework is formulated to incorporate the wind uncertainties and multi timescale coordination, where terminal SoCs and value functions are essential to integrate decisions across slow and fast timescales. A non-iterative parametric programming-based method is proposed to solve the slow-timescale SDP, and a simulation-based rollout algorithm is applied to extract the fast-timescale actions, which yields evident improvement over any heuristic base policy that is sequentially consistent. The numerical results demonstrate that the proposed method has a performance gap of approximately 7.1%, which outperforms the rule-based method by 32.6%. The installation of AA-CAES improves the total revenue by 11.9%. Additionally, the hybrid control strategy enhances the charging flexibility and efficiency by 30.7% and 4.3%, respectively, resulting in a 3.9% reduction in tracking deviation penalty.

A solution approach for sponsored search advertising and dynamic pricing for a perishable product and an online retailer with budget constraint

This paper presents an algorithm based on stochastic dynamic programming framework for an online retailer who is to maximize his revenue by dual use of sponsored search advertising and dynamic pricing for a fixed inventory of a perishable product over a finite horizon. The available budget for advertising is limited over the planning horizon. As there is an interaction between bidding and pricing decisions, we provide an integrated advertising and pricing solution approach. First, we prove some properties of optimal policy. Second, according to the properties, we develop an efficient algorithm to solve large-scale instances. Finally, we conduct several numerical experiments to compare the proposed algorithm with an optimal approach which is presented in the form of a non-linear mathematical model. We observe that in all provided instances, the proposed algorithm reaches the optimal solution in far less time than the optimal approach. Additionally, we indicate the relationship between the run time of the presented approach and inventory level. In contrast with expectations, we find out that for a particular parameter setting, an increase in inventory level does not necessarily lead to an increase in the run time of the presented approach.

Optimal specialty crop planning policies with yield learning and forward contract

Hops are the flowers of a specialty crop that provide unique flavors to craft beer. We study a multi‐year crop planning problem for a farmer who seeks to add hops production to the current production of a conventional crop. The farmer must dynamically determine the number of acres to allocate to each crop and design the terms of the forward contract under which brewers purchase the hops, considering the uncertainty of weather conditions, hops yield, hops spot market price, and conventional crop price. We formulate a multi‐period stochastic dynamic programming framework that incorporates statistical learning methods that depend on exogenous factors. We also develop an easy‐to‐implement learning‐based marginal total profit heuristic which can potentially be used as a decision support tool. Our numerical analyses suggest that yield learning is particularly important for a farmer who is considering investing in a high margin, but potentially risky, new crop such as hops. We also characterize the conditions under which yield learning is most beneficial for farmers. This paper contributes to the literature on specialty crop planning by introducing a multi‐year planning framework that incorporates unique characteristics of specialty crop production. We fill a gap in the crop planning literature by considering how farmers can learn about crop yields based on realized yields and exogenous factors such as weather conditions. Our paper is also of practical importance for farmers who seek to diversify their crop portfolio to hedge against risks associated with trade tensions and potential price drops for conventional crops.

Dual Discounting in Renewable Resource Planning under Risk.

Renewable resource planning and management projects entail evaluating economic and ecological criteria in the long term. During the past decade or so, dual discounting--ecological criteria discounted at a smaller rate than that for economic criteria--has been proposed for such projects as an alternative to the prevailing single-discounting scheme, out of theoretical and empirical considerations. We focus on how to apply this principle in planning problems that involve a multitude of risk in the attendant biological-economic system. A stochastic dynamic programming framework is introduced which allows dual discounting rates and finds the optimal decisions (passively) adapting to changes in the system. Furthermore, we show how to evaluate the variances of the criteria in this framework. With a case study of managing public forestlands in the US Pacific Northwestern region for both timber return and habitat preservation for the northern spotted owl (Strix occidentalis caurina), we illustrate the impacts that the dual-discounting scheme has on the trade-off between conflicting management objectives, the optimal planning strategy, the temporal development of the portion of the forestlands suitable for owl habitats, as well as its steady-state expected value and standard deviation.

Dynamic hedging in incomplete markets using risk measures

Abstract In this paper, we consider the pricing of financial derivatives that involve dynamic hedging strategies and payments within the planning horizon. Equity-indexed annuities (EIAs), guaranteed investment certificates (GICs) and American and barrier options are typical examples of these products. Our exploration involves the use and comparison of alternative models that use risk measures. Although the hedging is done for each observation of the input stochastic process, the appropriate mix of risk measures and state dynamic equations helps the issuer to appropriately tailor the overall risk exercise. These different models are solved by a unified backward stochastic dynamic programming framework that we imbed with parametric techniques to shorten the running time and manage the curse of dimensionality in dynamic programming. To demonstrate the flexibility of this framework we present numerical examples featuring GICs and point-to-point EIAs. Finally, by using sampling techniques, optimal hedging strategies and specific indicators of the hedging performance, we make recommendations on how to fine tune the risk measure parameters.

A multi-state Markov chain model for rainfall to be used in optimal operation of rainwater harvesting systems

Consideration of rainfall dynamics can rationalize the operation of rainwater harvesting systems. This study aims at establishing a methodology to construct a Markov chain model for time series of rainfall in temperate climates that optimal operation of rainwater harvesting systems is cast in the framework of stochastic dynamic programming. Due to the seasonal climate, the model is time-varying but stationary in each month. The states of the Markov chain are ranges of rainfall depths in 10 min. Transition probabilities determining the dynamics of the Markov chain are to be estimated for each month. The occurrence of dry states is frequent enough to estimate the transition probabilities from a dry state empirically. While, on the condition that a wet state is observed, the rainfall depth in the next 10 min is assumed to obey to the gamma distribution with two parameters. New formulae, including two exponent parameters, are proposed to relate the conditional mean and variance with the parameters of the gamma distribution. The values of the exponent parameters are identified from sequential searches so that the monthly average rainfall depths in 10 min become consistent with the observed ones. Then, a complete set of transition probabilities achieving the mean-reverting property is obtained to establish the Markov chain model. Examples of operating a hypothetical rainwater harvesting system are presented to demonstrate the utility of the Markov chain model in application to optimal management of water resources and stormwater retention in the framework of stochastic dynamic programming. The mean-reverting smooth transition probabilities also contribute to stabilizing the optimal policy.

Specialist care in rural hospitals: From Emergency Department consultation to hospital discharge

In urban and rural hospitals, congested Emergency Departments (EDs) are filled with patients boarding in the ED awaiting admission to inpatient wards. We study this problem beyond the walls of the ED, examining the multi-departmental process managed by specialists. In rural hospitals, an Internal Medicine Specialist (Internist) commonly serves simultaneously as both the Intensive Care Unit (ICU) physician and Internist on call. We develop a stochastic dynamic programming framework for specialists’ workflow decisions and apply it to data sets developed from two rural hospitals. One uses the dual role approach and the other, similar to urban hospitals, staffs the ICU with another physician, each with a single role. Our empirical results show that, excluding an overnight batch, arrivals of ED consultation requests for rural specialists follow a homogeneous Poisson process. Our models help identify better policies and determine how much better off a hospital is with two rather than one Internist on call. Although current guidelines suggest an early inpatient discharge strategy, we find that specialists should give higher priority to ED consultations unless a threshold number of patients are boarding in the ED or until a threshold time of day when specialists should give higher priority to inpatient discharges.

Dynamic mission abort policy for systems operating in a controllable environment with self-healing mechanism

Catastrophic failures of safety-critical systems could result in significant economic losses and damage. To improve the survival probability of safety-critical systems, self-healing mechanism is usually considered in the design stage and mission abort may be conducted if the failure risk becomes too high. We investigate the optimal mission abort policies for systems subject to a controllable shock process with self-healing mechanism. Minor failure and catastrophic failures are two competing failure modes of the system. Imperfect repair is carried out when a minor failure occurs and the external shock process is renewed after the imperfect repair. Mission abort decisions are considered based on the time in mission and the number of experienced minor failures. The optimal mission abort problem is formulated within the framework of stochastic dynamic programming to minimize the expected total cost of mission failure, system failure and imperfect repair. The structural properties of the optimal policy are investigated using the optimal stopping theory and a case study is presented to illustrate the obtained results.

Lower cost departures for airlines: Optimal policies under departure metering

Departure metering is an airport surface management procedure that limits the number of aircraft in the runway queue by holding aircraft either at a predesigned metering area or at gates. Field tests of the procedure have shown significant fuel savings, implying that the procedure can play an important role in the Next Generation Air Transportation System being implemented in the U.S. In this paper we study optimal departure operations at airports in the context of departure metering. More specifically, we develop a stochastic dynamic programming framework for tactical management of pushback operations at gates and for determining the optimal number of aircraft to be directed to the runway queue from the metering areas. We introduce four easy-to-implement departure metering policies, and perform comparative analyses between these practical policies and the numerical-optimization based policies. In addition, from a strategic perspective, we identify optimal capacities for metering areas to be used as part of departure metering implementations. Overall, we find that the annual fuel and operating savings for airlines could be around $1.7 million if our proposed policies are implemented at the Detroit Metropolitan Wayne County Airport. Such policies can also be adapted by other airports to improve the overall efficiency of surface traffic management and departure operations.

An Approximate Dynamic Programming Approach for Dual Stochastic Model Predictive Control

Abstract Dual control explicitly addresses the problem of trading off active exploration and exploitation in the optimal control of partially unknown systems. While the problem can be cast in the framework of stochastic dynamic programming, exact solutions are only tractable for discrete state and action spaces of very small dimension due to a series of nested minimization and expectation operations. We propose an approximate dual control method for systems with continuous state and input domain based on a rollout dynamic programming approach, splitting the control horizon into a dual and an exploitation part. The dual part is approximated using a scenario tree generated by sampling the process noise and the unknown system parameters, for which the underlying distribution is updated via Bayesian estimation along the horizon. In the exploitation part, we fix the resulting parameter estimate of each scenario branch and compute an open-loop control sequence for the remainder of the horizon. The key benefit of the proposed sampling-based approximation is that it enables the formulation as one optimization problem that computes a collection of control sequences over the scenario tree, leading to a dual model predictive control formulation.

Revenue Management of a Professional Services Firm Under a Quality-Revelation Model

Professional service firms (PSFs) such as management consulting, law, accounting, investment banking, architecture, advertising and home-repair companies provide services for turnkey complicated projects. The operational tasks of such firms consists of first bidding for the project and, if successful in the bid, assigning employee resources to the project. In this paper we model this as a revenue management problem and develop a stochastic dynamic programming framework to aid the firm in their bidding and assignment process. We study this primarily under a quality-revelation model where the employees that would be assigned to the project are committed ex ante, as part of the bid. The win probability depends on the quoted price as well as the quality of the employees who would work on the project. We also consider a quality-reputation model where the bid’s win probability depends on past performance, say an average of the quality of past jobs. The problem is computationally challenging and we provide a series of bounds and solution methods to approximate the stochastic dynamic program. Based on our computational techniques we are able to address a number of interesting questions on utilization and the value of each employee type for a PSF. Our methodology provides management a toolkit for bidding on projects as well as to perform workforce analytics to determine optimal utilization levels and staffing decisions.

Risk-averse stochastic dual dynamic programming approach for the operation of a hydro-dominated power system in the presence of wind uncertainty

Operation planning models for hydro-dominated power systems usually use low temporal resolutions due to the excessive computational burden, thus ignoring short-term characteristics of such systems. As a result, in systems coupled with wind energy, such models may fail to accurately capture wind variability, and may not appropriately take into account potential consequences of uncertainty on the system operation. This paper addresses this drawback by (i) “controlling” the cost associated with the operation of a hydro-dominated power system equipped with wind power and batteries via a risk-measure and (ii) formulating the short-term load balance as probabilistic constraints in order to hedge against potential extreme wind power scenarios. The risk-averse scheme is embedded in the stochastic dual dynamic programming framework. Simulation results for a case study on a real industrial setting show that hedging the system against the short-term volatility of wind power contributes to mitigating the risk of excessive operations costs or load curtailments, and that the consideration of the decision maker risk profile contributes to decreasing the variability of the solutions. In addition, the results of the application also illustrate the potential of the scheme to assess the energy situation of a country or a region under the penetration of wind energy and batteries deployment.

Multistage Stochastic Unit Commitment Using Stochastic Dual Dynamic Integer Programming

Unit commitment (UC) is a key operational problem in power systems for the optimal schedule of daily generation commitment. Incorporating uncertainty in this already difficult mixed-integer optimization problem introduces significant computational challenges. Most existing stochastic UC models consider either a two-stage decision structure, where the commitment schedule for the entire planning horizon is decided before the uncertainty is realized, or a multistage stochastic programming model with relatively small scenario trees to ensure tractability. We propose a new type of decomposition algorithm, based on the recently proposed framework of stochastic dual dynamic integer programming (SDDiP), to solve the multistage stochastic unit commitment (MSUC) problem. We propose a variety of computational enhancements to SDDiP, and conduct systematic and extensive computational experiments to demonstrate that the proposed method is able to handle elaborate stochastic processes and can solve MSUCs with a huge number of scenarios that are impossible to handle by existing methods.

Sponsored search advertising and dynamic pricing for perishable products under inventory-linked customer willingness to pay

Several online retailers provide inventory availability information on their websites in addition to leveraging sponsored search advertising to drive customer traffic to their retail websites. The increased ability of users to interact over the internet encourages retailers to shift to sponsored search advertising. In this paper, we design a decision support model to provide strategic bid and pricing decisions to a retailer selling a perishable product over a short horizon using sponsored search advertising to attract customers to his website. The retailer complements sponsored search bidding with dynamic pricing in a multi-period stochastic dynamic programming framework. Our analyses show that it is optimal for the retailer to invest heavily in bidding at low inventory levels, whereas at high levels of inventory he should use price promotions to enhance profits. We also find that the optimal bid and price increase with the increase in mean and variability of the customer's reservation price.

POMDP and MOMDP solutions for structural life-cycle cost minimization under partial and mixed observability

Scheduling of inspection and maintenance policies during the life-cycle of operating infrastructure necessitates optimization of long-term objectives in stochastic environments. Modern answers to the problem should focus on quantitative decision-making techniques, taking advantage of informative but uncertain data that become available in time. As such, the problem is efficiently addressed within the framework of stochastic dynamic programming by means of Partially Observable Markov Decision Processes (POMDPs) and Mixed Observability Markov Decision Processes (MOMDPs). Although these methodologies can provide very sophisticated solutions with optimality guarantees, important computational challenges often emerge, mainly due to the continuity of the multidimensional belief space on the probability simplex. In response, recent value iteration algorithms based on point-based approaches have been suggested, focusing on reachable belief points that can support an accurate value function. In this work, several POMDP and MOMDP point-based algorithms, with various characteristics regarding the exploration of the belief space and the value function update procedures, are rigorously analyzed. The algorithms are compared and evaluated in terms of accuracy and performance in stationary and nonstationary problems of structural inspection and maintenance for life-cycle cost minimization. Results are thoroughly discussed and several insights along with practical suggestions for similar problems are provided.

Stochastic Optimal Energy Management of Smart Home With PEV Energy Storage

This paper proposes a stochastic dynamic programming framework for the optimal energy management of a smart home with plug-in electric vehicle (PEV) energy storage. This paper is motivated by the challenges associated with intermittent renewable energy supplies and the local energy storage opportunity presented by vehicle electrification. This paper seeks to minimize electricity ratepayer cost, while satisfying home power demand and PEV charging requirements. First, various operating modes are defined, including vehicle-to-grid, vehicle-to-home, and grid-to-vehicle. Second, we use equivalent circuit PEV battery models and probabilistic models of trip time and trip length to formulate the PEV to smart home energy management stochastic optimization problem. Finally, based on time-varying electricity price and time-varying home power demand, we examine the performance of the three operating modes for typical weekdays.