Stochastic Dynamic Programming Problem Research Articles

Information Relaxation and a Duality-Driven Algorithm for Stochastic Dynamic Programs

The curse of dimensionality significantly restricts the use of dynamic programming methods in solving complex problems. Consequently, researchers and practitioners often resort to approximate (suboptimal) control policies that strike a balance between ease of implementation and satisfactory performance. Information relaxation-based duality techniques generate both upper and lower bounds for the true values of stochastic dynamic programming problems, allowing us to evaluate the optimality of approximate policies through the dual gap. However, the literature still lacks guidance on handling cases where the gaps are excessively loose. In “Information Relaxation and a Duality-Driven Algorithm for Stochastic Dynamic Programs,” Chen, Ma, Liu, and Yu develop a novel DDP framework to obtain and tighten confidence interval estimates for the true value functions of SDP problems. Leveraging a new finding that the dual operation yields subsolutions, they establish convergence guarantees for DDP. Additionally, a regression-based Monte Carlo method is introduced, aimed at high-dimensional applications. Numerical examples demonstrate that DDP effectively improves dual gaps for various heuristics that are commonly used in the literature.

Reinforcement Learning Powered Station Parameter Optimized Radiation Therapy (SPORT): A Novel Treatment Planning and Beam Delivery Technique

Conventional intensity modulated radiation therapy (IMRT) with a typical 5-20 fixed beams often does not provide sufficient angular sampling required for conformal dose shaping, whereas current volumetric modulated arc therapy (VMAT) discretizes the angular space into equally spaced control points without considering the differential need for intensity modulation of different angles, leading to undersampling at some angles while oversampling at some other angles. Our goal is to develop a node or station parameter optimized radiation therapy (SPORT) strategy with simultaneously optimized angular sampling and beam modulation by leveraging state-of-the-art reinforcement learning and the unique capability of modern digital LINACs in dose delivery through programmable nodal points. We developed a SPORT optimization framework, in which, the process of programming control points (or station parameters) was formulated as a stochastic dynamic programming problem, which was solved by a reinforcement learning-based algorithm. On-policy reinforcement learning method, namely, state-action-reward-state-action (SARSA) was integrated with deep convolutional neural network to predict station parameters by utilizing the patient's anatomical structures meanwhile considering the delivery capability of a typical digital LINAC machine. Here, the deep convolutional neural network estimated the state-action value by using the quality of the plan with current station parameters when a next potential station parameter was selected. The state-action value was then updated by SARSA learning. The quality of the plan was quantified by dosimetry constraints. The model was assessed by a retrospective study on a cohort of patients underwent head-and-neck radiation therapy. Dosimetric analysis and delivery efficiency comparisons were used to evaluate the performance of the proposed framework. Our model was used to generate 16 plans unseen in the original training set. All the plans predicted by our model achieved better dose distributions without violating clinical planning constraints. Moreover, instead of using 4 full standard arcs in the original clinically used plans obtained via manual optimization, the predicted plans only used one full standard arc (about 178 control points) plus boost from a few sub-arcs (less than 30 degrees of gantry angles), which significantly improved the efficiency of the beam delivery. We are in the process of integrating the sub-arcs into the full arc by considering the programmable capability of modern LINACs. We demonstrated that a machine learning-based SPORT framework capable of optimizing the spatial sampling and beam modulation simultaneously for modern radiation therapy. The framework not only significantly improves the quality and efficiency of beam delivery, but also has the potential to be incorporated into current clinical workflow to improve the efficiency of dose planning and delivery.

Software for "Asymptotically Optimal Sampling Policy for Selecting Top-m Alternatives"

We consider selecting the top-m alternatives from a finite number of alternatives via Monte Carlo simulation. Under a Bayesian framework, we formulate the sampling decision as a stochastic dynamic programming problem and develop a sequential sam-pling policy that maximizes a value function approximation one-step look ahead. To show the asymptotic optimality of the proposed procedure, the asymptotically optimal sampling ratios that optimize the large deviations rate of the probability of false selection for select-ing the top-m alternatives have been rigorously defined. The proposed sampling policy is not only proved to be consistent but also achieve the asymptotically optimal sampling ratios. Numerical experiments demonstrate superiority of the proposed allocation proce-dure over existing ones.

Asymptotically Optimal Sampling Policy for Selecting Top-m Alternatives

We consider selecting the top-m alternatives from a finite number of alternatives via Monte Carlo simulation. Under a Bayesian framework, we formulate the sampling decision as a stochastic dynamic programming problem and develop a sequential sampling policy that maximizes a value function approximation one-step look ahead. To show the asymptotic optimality of the proposed procedure, the asymptotically optimal sampling ratios that optimize the large deviations rate of the probability of false selection for selecting the top-m alternatives have been rigorously defined. The proposed sampling policy is not only proved to be consistent but also achieve the asymptotically optimal sampling ratios. Numerical experiments demonstrate superiority of the proposed allocation procedure over existing ones. History: Accepted by Bruno Tuffin, Area Editor for Simulation. Funding: This work was supported by the National Natural Science Foundation of China [Grants 72250065, 72293582, 72022001, and 71901003], and the National Science Foundation [Grant DMS-2053489], the major project of the National Natural Science Foundation of China [Grant 72293582], and the China Scholarship Council [Grant CSC202206010152]. Supplemental Material: The software that supports the findings of this study is available within the paper and its Supplemental Information ( https://pubsonline.informs.org/doi/suppl/10.1287/ijoc.2021.0333 ) as well as from the IJOC GitHub software repository ( https://github.com/INFORMSJoC/2021.0333 ). The complete IJOC Software and Data Repository is available at https://informsjoc.github.io/ .

An approximate dynamic programming approach for the maintenance optimisation of networked critical infrastructures

The continuous performance of networked infrastructures, such as transportation, telecommunications, and power transmission, is critical to the economic development and social well-being of society. To ensure their reliability in continuously meeting prescribed demand, cost-effective maintenance is essential. To this end, the influence of maintenance actions measures on the reliability of an infrastructure need to be considered both during their implementation and after their completion. To address this problem and to consider the uncertainty of future states, we model the multi-period optimisation of an infrastructure’s maintenance planning as a stochastic dynamic programming problem. An approximate dynamic programming approach is developed to deal with the computational complexity and solve the described problem. The proposed approach can help deal with the uncertainty of future infrastructure states by determining the optimal maintenance plan for each possible state in advance.

Valuation of an unexplored oilfield under uncertain oil price and reservoir condition: A stochastic dynamic programming approach with simulation-based reward function

We develop an unexplored oilfield valuation model under uncertain exploration outcomes, reservoir conditions, and oil prices. The exploration outcomes follow a binomial process, while oil prices are represented using the Schwartz-Smith model. The reservoir conditions are characterized by the joint probability distribution of post-discovery parameters estimated using a machine learning approach. The corresponding valuation model is a typical stochastic dynamic programming problem with a simulation-based reward function. The net cash flow lattice takes the form of a recombining quadrinomial derived from the discrete representation of the Schwartz-Smith oil price model. We apply backward induction with embedded real-coded genetic algorithms to the net cash flow lattice to calculate the oilfield value. The model allows for field abandonment before lease expiration if the remaining reserve is uneconomical. To improve computational efficiency, we combine the Latin hypercube sampling and antithetic variates to reduce the variances. The model is implemented in an unexplored oilfield under two scenarios of oil presence probability: 100% and 75%. In the first scenario, we obtained a mean oilfield value of US$5.5 million with a CVaR of US$0.79 million, while in the second, we came up with a mean of -US$0.77 million and a CVaR of US$31.74 million.

Delay-Tolerant UAV-Assisted Communication: Online Trajectory Design and User Association

UAV-assisted communication is a promising paradigm to enhance the performance of wireless networks. In this paper, we consider the network randomness in the delay-tolerant UAV-assisted communication systems. We aim to optimize both the UAV trajectory and user association in an online manner to achieve a finite queueing delay at the users' buffers with the minimum energy consumption of the UAVs. We first formulate our problem as a stochastic dynamic programming problem that integrates the future expected energy consumption and delay into its objective function and constraints, respectively. To deal with the inaccurate knowledge of future user demand and mobility, we apply the Lyapunov optimization technique and convert it into a series of online joint UAV trajectory and user association optimization problems. However, each online problem is a non-convex non-linear integer programming problem, which is very challenging to solve in general. As such, we first eliminate the coupling relationship of user association variables in each online problem. By exploiting the special graph structure of the problem, we can decompose the joint problem into inner optimization and outer optimization problems. Then, we propose the online UAV mobility and user association selection (UMUAS) scheme, which includes two steps to solve the inner and outer problems. Simulation results show that our online UMUAS scheme achieves the near-optimal performance of the energy-delay tradeoff, which also has a 44.2% improvement over the greedy scheme and a 51.3% improvement over the interference-free scheme. Moreover, it achieves the highest energy efficiency among the three benchmark schemes.

An envelope method for solving continuous-time stochastic models with occasionally binding constraints

I introduce a finite-difference solution method based on the envelope condition in continuous-time stochastic dynamic programming problems. The envelope method is easier to code and, in the presence of occasionally binding constraints, faster and more stable than popular methods based on the Hamilton–Jacobi–Bellman equation. As an illustration, I solve a stochastic growth model with irreversible investment.

Evaluation of photovoltaic storage systems on energy markets under uncertainty using stochastic dynamic programming

The rising share of intermittent renewable energy production in energy systems increasingly poses a threat to system stability and the price level in energy markets. However, the effects of renewable energy production onto electricity markets also give rise to new business opportunities. The expected increase in price differences increases the market potential for storage applications and combinations with renewable energy production. The value of storage depends critically on the operation of the storage system.In this study, we evaluate large-scale photovoltaic (PV) storage systems under uncertainty, as renewable energy production and electricity prices are fundamentally uncertain. In comparison to households who largely consume the stored energy themselves, the major business case for large-scale PV and storage systems is arbitrage trading on the electricity markets. The operation problem is formulated as a Markov decision process (MDP). Uncertainties of renewable energy production are integrated into an electricity price model using ARIMA-type approaches and regime switching. Due to non-stationarity and heteroskedasticity of the underlying processes, an appropriate stochastic modeling procedure is developed. The MDP is solved using stochastic dynamic programming (SDP) and recombining trees (RT) to reduce complexity taking into account the different time scales in which decisions have to be taken. We evaluate the solution of the SDP problem against Monte Carlo simulations with perfect foresight and against a storage dispatch heuristic. The program is applied to the German electricity and reserve power market to show the potential increase in storage value with higher price spreads, and evaluate a possible imposition of the feed-in levy onto energy directly stored from the common grid.

Gradient-bounded dynamic programming for submodular and concave extensible value functions with probabilistic performance guarantees

We consider stochastic dynamic programming problems with high-dimensional, discrete state-spaces and finite, discrete-time horizons that prohibit direct computation of the value function from a given Bellman equation for all states and time steps due to the “curse of dimensionality”. For the case where the value function of the dynamic program is concave extensible and submodular in its state-space, we present a new algorithm that computes deterministic upper and stochastic lower bounds of the value function in the realm of dual dynamic programming. We show that the proposed algorithm terminates after a finite number of iterations. Furthermore, we derive probabilistic guarantees on the value accumulated under the associated policy for a single realisation of the dynamic program and for the expectation of this value. Finally, we demonstrate the efficacy of our approach on a high-dimensional numerical example from delivery slot pricing in attended home delivery.

Mechanism Design for Stochastic Dynamic Parking Resource Allocation

In this paper, we study a parking management problem where an operator manages a publicly owned parking service system with unknown parking demand. Assuming that the operator has perfect information, we first formulate the operator's problem as a stochastic dynamic programming problem, and to overcome the curse of dimensionality, we resort to approximate dynamic programming for solving it. However, in practice, some information that is essential for centralized management is usually privately known, which provides incentives for strategic behaviors of drivers and could lead to suboptimal system performance. We design a two‐step mechanism and prove that, in step 1, drivers’ choices of whether or not to enter the managed system following the approximate optimal solution satisfy Bayesian‐Nash equilibrium (BNE), and in step 2, that truthful reporting is a dominant strategy for all drivers under any circumstance. We investigate the properties of the resulting equilibria, and further modify the mechanism to ensure that the desired approximate system optimum solution is the only resulting BNE. Numerical examples show that the mechanism design not only enhances the average system performance but also increases the system robustness.

Dynamic expansion of flexible capacity with and without pricing coordination

This research presents an optimal multi-period dynamic capacity expansion framework for a shared flexible capacity producing two products. The framework allows modeling coordinated and non-coordinated capacity and marketing decisions under demand uncertainties. In our two-level decision process, the upper level considers the strategic capacity expansion and formulates multi-period uncertainty as a stochastic dynamic programming (DP) problem, and the lower level considers each period’s pricing and capacity allocation decisions in response to available capacity and market demand. We compare capacity expansion decisions under centralized pricing schemes that coordinate the pricing decisions of different products based on the overall interest of the firm and the decentralized pricing schemes based on the interests of individual product managers. Numerical results show that adopting a decision coordination strategy may help firms to maintain flexibility and reduce capacity expansion investment while enhancing overall profitability.

Addressing state space multicollinearity in solving an ozone pollution dynamic control problem

High ground-level ozone concentrations constitute a serious air quality problem in many metropolitan regions. In this paper, we study a stochastic dynamic programming (SDP) formulation of the Atlanta metropolitan ozone pollution problem that seeks to reduce ozone via reductions of nitrogen oxides. The initial SDP formulation involves a 524-dimensional continuous state space, including ozone concentrations that are highly correlated. In prior work, a design and analysis of computer experiments (DACE) based approximate dynamic programming (ADP) solution method was able to conduct dimensionality reduction and value function approximation to enable a computationally-tractable numerical solution. However, this prior work did not address state space multicollinearity. In statistical modeling, high multicollinearity is well-known to adversely affect the generalizability of the constructed model. This issue is relevant whenever an empirical model is trained on data, but is largely ignored in the ADP literature. We propose approaches for addressing the multicollinearity in the Atlanta case study and demonstrate that if high multicollinearity is ignored, the resulting empirical models provide misleading information within the ADP algorithm. Because many SDP applications involve multicollinear continuous state spaces, the lessons learned in our research can guide the development of ADP approaches for a wide variety of SDP problems.

Information theory for ranking and selection

AbstractWe study the classical ranking and selection problem, where the ultimate goal is to find the unknown best alternative in terms of the probability of correct selection or expected opportunity cost. However, this paper adopts an alternative sampling approach to achieve this goal, where sampling decisions are made with the objective of maximizing information about the unknown best alternative, or equivalently, minimizing its Shannon entropy. This adaptive learning is formulated via a Bayesian stochastic dynamic programming problem, by which several properties of the learning problem are presented, including the monotonicity of the optimal value function in an information‐seeking setting. Since the state space of the stochastic dynamic program is unbounded in the Gaussian setting, a one‐step look‐ahead approach is used to develop a policy. The proposed policy seeks to maximize the one‐step information gain about the unknown best alternative, and therefore, it is called information gradient (IG). It is also proved that the IG policy is consistent, that is, as the sampling budget grows to infinity, the IG policy finds the true best alternative almost surely. Later, a computationally efficient estimate of the proposed policy, called approximated information gradient (AIG), is introduced and in the numerical experiments its performance is tested against recent benchmarks alongside several sensitivity analyses. Results show that AIG performs competitively against other algorithms from the literature.

Warranty Matching in a Consumer Electronics Closed-Loop Supply Chain

Problem definition : We examine a dynamic assignment problem faced by a large wireless service provider (WSP) that is a Fortune 100 company. This company manages two warranties: (i) a customer warranty that the WSP offers its customers and (ii) an original equipment manufacturer (OEM) warranty that OEMs offer the WSP. The WSP uses devices refurbished by the OEM as replacement devices, and hence their warranty operation is a closed-loop supply chain. Depending on the assignment the WSP uses, the customer and OEM warranties might become misaligned for customer-device pairs, potentially incurring a cost for the WSP. Academic/practical relevance : We identify, model, and analyze a new dynamic assignment problem that emerges in this setting called the warranty matching problem. We introduce a new class of policies, called farsighted policies, which can perform better than myopic policies. We also propose a new heuristic assignment policy, the sampling policy, which leads to a near-optimal assignment. Our model and results are motivated by a real-world problem, and our theory-guided assignment policies can be used in practice; we validate our results using data from our industrial partner. Methodology : We formulate the problem of dynamically assigning devices to customers as a discrete-time stochastic dynamic programming problem. Because this problem suffers from the curse of dimensionality, we propose and analyze a set of reasonable classes of assignment policies. Results : The performance metric that we use for a given assignment policy is the average time that a replacement device under a customer warranty is uncovered by an OEM warranty. We show that our assignment policies reduce the average uncovered time and the expected number of out-of-OEM-warranty returns by more than 75% in comparison with our industrial partner’s current assignment policy. We also provide distribution-free bounds for the performance of a myopic assignment policy and of random assignment, which is a proxy for the WSP’s current policy. Managerial implications : Our results indicate that, in closed-loop supply chains, being completely farsighted might be better than being completely myopic. Also, policies that are effective in balancing short-term and long-term costs can be simple and effective, as illustrated by our sampling policy. We describe how the performance of myopic and farsighted policies depend on the size and length of inventory buildup.

Two-level task scheduling with multi-objectives in geo-distributed and large-scale SaaS cloud

With the exploding of data-intensive web applications and requests (tasks), geo-distributed and large-scale data centers (DCs) are widely deployed in Software as a Service (SaaS) cloud, but server failures continue to grow at the same time. In this context, task scheduling problems become more intricate and both scheduling quality and scheduling speed raise further concerns. In this paper, we first propose a virtualized & monitoring SaaS model with predictive maintenance to minimize the costs of fault tolerance. Then with the monitored and predicted available states of servers, we focus on dynamic real-time task scheduling in geo-distributed and large-scale DCs with heterogeneous servers. Multiple objectives, including the long-term performance benefits, energy and communication costs, are taken into consideration in order to improve scheduling quality. For inter-DC and intra-DC task scheduling, two dynamic programming problems are formulated respectively, but there exists the problem that both state and action spaces are too large to be solved by simple iterations. To address this issue, we introduce the idea of reinforcement learning theory into solving traditional stochastic dynamic programming problems in the large-scale SaaS cloud, and put forward a cascaded two-level (inter-DC and intra-DC level) approximate dynamic programming (ADP) task-scheduling algorithm. The computation complexity can be significantly reduced and scheduling speed can be greatly improved. Finally, we conduct experiments with both random simulation data and Google cloud trace-logs. QoS evaluations and comparisons demonstrate that two ADP algorithms can work cooperatively, and our two-level ADP algorithm is more effective under large quantity of bursty requests.

From Probabilistic Seasonal Streamflow Forecasts to Optimal Reservoir Operations: A Stochastic Programming Approach

Abstract We investigate the potential use of seasonal streamflow forecasts for the realtime operation of Angat reservoir (Philippines). The system is characterized by a strong intra- and inter-annual variability in the inflow process, which is further amplified by the El Nino Southern Oscillation (ENSO). We bank on the relationship between ENSO indices and local hydro-climatological processes to issue probabilistic streamflow forecasts (with a 3-month forecast horizon) and then integrate them within a Multistage Stochastic Programming (MSP) approach. The rolling-horizon, forecast-informed scheme is adopted for the period 1968–2014 and benchmarked against deterministic optimization solutions with perfect forecasts, climatology, and mean forecasts. We also compare its performance with the current operating rules, and the operating rules obtained by solving a Stochastic Dynamic Programming problem. Results show that the MSP approach can help reduce the severity of failures during prolonged droughts caused by ENSO.

Warranty Matching in a Consumer Electronics Closed-Loop Supply Chain

Problem definition: We examine a dynamic assignment problem faced by a large Wireless Service Provider (WSP) that is a Fortune 100 company. This company manages two warranties: (i) a customer warranty that WSP offers its customers, and (ii) an Original Equipment Manufacturer (OEM) warranty that OEMs offer the WSP. The WSP uses devices refurbished by the OEM as replacement devices and hence their warranty operation is a closed-loop supply chain. Depending on the assignment policy WSP uses, the customer and OEM warranties might become misaligned for customer-device pairs, potentially incurring a cost for WSP. Academic/practical relevance: We identify, model, and analyze a new dynamic assignment problem that emerges in this setting called the warranty matching problem. We introduce a new class of policies, called farsighted policies, which can perform better than myopic policies. We also propose a new heuristic assignment policy, the sampling policy, which leads to a near-optimal assignment. Our model and results are motivated by a real-world problem, and our theory-guided assignment policies can be used in practice; we validate our results using data from our industrial partner. Methodology: We formulate the problem of dynamically assigning devices to customers as a discrete-time stochastic dynamic programming problem. Because this problem suffers from the curse of dimensionality, we propose and analyze a set of reasonable classes of assignment policies. Results: The performance metric that we use for a given assignment policy is the average time that a replacement device under a customer warranty is uncovered by an OEM warranty. We show that our assignment policies reduce the average uncovered time and the expected number of out-of-OEM-warranty returns by more than 75% in comparison to our industrial partner's current assignment policy. We also provide distribution-free bounds for the performance of a myopic assignment policy, and of random assignment, which is a proxy for the current WSP policy. Managerial implications: Our results indicate that, in closed-loop supply chains, being completely far-sighted might be better than being completely myopic. Also, policies that are effective in balancing short-term and long-term costs can be simple and effective, as illustrated by our sampling policy. We describe how the performance of myopic and farsighted policies depend on the size and length of inventory build-up.

Accuracy of policy function approximations for strongly concave recursive problems

AbstractUnder the hypotheses of strong concavity of the aggregator function and concavity of the stochastic operator which define the objective function of the stochastic dynamic programming problem (SDPP), we prove that the policy function approximation of the problem is a Hölder continuous function with respect to the value function approximation. From this, explicit error bounds for computation of the solution of such problems are provided. To illustrate the results we apply the error control formula to the solution of two SDPPs with aggregator functions: the neoclassical Ramsey economic growth model and the Lucas asset pricing model.

Exit dynamics of start-up firms: Structural estimation using indirect inference

We estimate by means of indirect inference a structural economic model where firms’ exit and investment decisions are the solution to a discrete–continuous stochastic dynamic programming problem. Our method solves the main difficulty of simulation-based inference in structural discrete–continuous choice models, namely that the simulated trajectories are discontinuous functions of the structural parameters. Estimating the model on all start-up firms in the Norwegian manufacturing sector, we find that if the expected value of continuing production is persistently low relative to the expected value of exit, the firm has a high probability to exit.