Stochastic Dual Dynamic Programming Algorithm Research Articles

Uncertainty dynamics in energy planning models: An autoregressive and Markov chain modeling approach

This paper deals with the modeling of stochastic processes in long-term multi-stage energy planning problems when limited information is available about the distributions of such processes. Starting from simple estimates of variation intervals for uncertain parameters, such as energy demands and costs, we model the temporal correlation of these parameters through carefully constructed autoregressive (AR) models that respect the intervals defined in each period. We introduce a coefficient for “zigzag” effects in the evolution of uncertain processes, which controls the correlation across periods and also the likelihood of extreme scenarios. To preserve the convexity of the stochastic problem, we discretize the AR models associated with the cost parameters involved in the objective function by Markov chains. The resulting formulation is then solved using a Stochastic Dual Dynamic Programming (SDDP) algorithm available in the literature that handles finite-state Markov chains. Our numerical experiments, carried out on the Swiss energy system, show a very desirable adaptation strategy of investment decisions to uncertainty scenarios, a behavior that is not observed when the temporal correlation is ignored. Moreover, the solutions lead to better out-of-sample cost performances, especially on extreme scenario realizations, than the non-correlated ones which usually result in overcapacities to protect against high, but unlikely, parameter variations over time.

Medium-term stochastic hydrothermal scheduling with short-term operational effects for large-scale power and water networks

The high integration of variable renewable sources in electric power systems entails a series of challenges inherent to their intrinsic variability. A critical challenge is to correctly value the water available in reservoirs in hydrothermal systems, considering the flexibility that it provides. In this context, this paper proposes a medium-term multistage stochastic optimization model for the hydrothermal scheduling problem solved with the stochastic dual dynamic programming algorithm. The proposed model includes operational constraints and simplified mathematical expressions of relevant operational effects that allow more informed assessment of the water value by considering, among others, the flexibility necessary for the operation of the system. In addition, the hydrological uncertainty in the model is represented by a vector autoregressive process, which allows capturing spatio-temporal correlations between the different hydro inflows. A calibration method for the simplified mathematical expressions of operational effects is also proposed, which allows a detailed short-term operational model to be correctly linked to the proposed medium-term linear model. Through extensive experiments for the Chilean power system, the results show that the difference between the expected operating costs of the proposed medium-term model, and the costs obtained through a detailed short-term operational model was only 0.1%, in contrast to the 9.3% difference obtained when a simpler base model is employed. This shows the effectiveness of the proposed approach. Further, this difference is also reflected in the estimation of the water value, which is critical in water shortage situations.

Batch Learning SDDP for Long-Term Hydrothermal Planning

We consider the stochastic dual dynamic programming (SDDP) algorithm - a widely employed algorithm applied to multistage stochastic programming - and propose a variant using experience replay - a batch learning technique from reinforcement learning. To connect SDDP with reinforcement learning, we cast SDDP as a Q-learning algorithm and describe its application in both risk-neutral and risk-averse settings. We demonstrate the superiority of the algorithm over conventional SDDP by benchmarking it against PSR's SDDP software using a large-scale instance of the long-term planning problem of inter-connected hydropower plants in Colombia. We find that SDDP with batch learning is able to produce tighter optimality gaps in a shorter amount of time than conventional SDDP. We also find that batch learning improves the parallel efficiency of SDDP backward passes.

A stochastic policy algorithm for seasonal hydropower planning

AbstractHydropower producers need to plan several months or years ahead to estimate the opportunity value of water stored in their reservoirs. The resulting large-scale optimization problem is computationally intensive, and model simplifications are often needed to allow for efficient solving. Alternatively, one can look for near-optimal policies using heuristics that can tackle non-convexities in the production function and a wide range of modelling approaches for the price- and inflow dynamics. We undertake an extensive numerical comparison between the state-of-the-art algorithm stochastic dual dynamic programming (SDDP) and rolling forecast-based algorithms, including a novel algorithm that we develop in this paper. We name it Scenario-based Two-stage ReOptimization abbreviated as STRO. The numerical experiments are based on convex stochastic dynamic programs with discretized exogenous state space, which makes the SDDP algorithm applicable for comparisons. We demonstrate that our algorithm can handle inflow risk better than traditional forecast-based algorithms, by reducing the optimality gap from 2.5 to 1.3% compared to the SDDP bound.

Medium-term multi-stage distributionally robust scheduling of hydro–wind–solar complementary systems in electricity markets considering multiple time-scale uncertainties

Joint trading of hydro–wind–solar complementary systems (HWSCSs) in the electricity market (EM) helps to reduce the imbalance cost and increase profits. However, multiple energy resources and market price uncertainties affect the trading strategies. Existing medium-term (MT) scheduling approaches assume that the probability distribution of the random variable is perfectly known. Short-term variations were also ignored, which led to revenue loss and trading risk. To address the above issues, this paper proposes an MT multi-stage distributionally robust optimization (MDRO) scheduling approach for a price-taking HWSCS in the EM. Firstly, hourly unit commitment (HUC) constraints are incorporated into the MT scheduling model to accurately capture short-term variations. A novel ambiguity set is designed based on the modified chi-square distance to address probability distribution uncertainties at two different time scales. Subsequently, an MDRO scheduling model is proposed to optimize the trading strategy. Finally, the proposed MDRO model is converted to a large-scale multi-stage integer programming problem based on linearization and reformation. The stochastic dual dynamic integer programming algorithm is modified to ensure computational tractability. Xiluodu-Xiangjiaba HWSCS, located in the Jinsha River in China, was selected as a case study. The results show that: 1) the MDRO model is more robust to distributional uncertainties than the multi-stage stochastic programming (MSSP) model. When the probability distribution of the random variable changes, the MDRO model yields a higher expected revenue (+2.43%) and a lower standard deviation (-60.8%) of revenue, which illustrates lower trading risk. 2) Compared with MSSP, deterministic, two-stage stochastic programming, and distributionally robust optimization models, the MDRO model exhibits the best out-of-sample performance in terms of the highest expected revenue and lowest trading risk. 3) Incorporating HUC constraints into the MDRO model helps to increase the total revenue (+3.53%) and energy generation (+3.31%) at the expense of increasing the computational burden.

A Nonconvex Regularization Scheme for the Stochastic Dual Dynamic Programming Algorithm

We propose a new nonconvex regularization scheme to improve the performance of the stochastic dual dynamic programming (SDDP) algorithm for solving large-scale multistage stochastic programs. Specifically, we use a class of nonconvex regularization functions, namely folded concave penalty functions, to improve solution quality and the convergence rate of the SDDP procedure. We develop a strategy based on mixed-integer programming to guarantee global optimality of the nonconvex regularization problem. Moreover, we establish provable convergence guarantees for our customized SDDP algorithm. The benefits of our regularization scheme are demonstrated by solving large-scale instances of two multistage stochastic optimization problems. History: Pascal Van Hentenryck, Area Editor for Computational Modeling: Methods & Analysis. Supplemental Material: The software that supports the findings of this study is available within the paper and its Supplemental Information ( https://pubsonline.informs.org/doi/suppl/10.1287/ijoc.2021.0255 ) as well as from the IJOC GitHub software repository ( https://github.com/INFORMSJoC/2021.0255 ). The complete IJOC Software and Data Repository is available at https://informsjoc.github.io/ .

Multi-stage distributionally robust optimization for hybrid energy storage in regional integrated energy system considering robustness and nonanticipativity

Hybrid energy storage system (HESS) has advantages in coping with the uncertainty of renewable energy and improving the stability of regional integrated energy systems (RIES). Few of the current HESS uncertainty scheduling methods consider both robustness and nonanticipativity, which may result in scheduling strategies that are not feasible in practice. Therefore, this research constructs a HESS (containing electric, thermal, hydrogen and natural gas storage devices) and forms a hybrid energy storage operator (IHESO) with an independent market position based on the HESS to provide energy services. Subsequently, a novel multi-stage distributionally robust optimization (MSDRO) method is established to maximize HESS revenue. Robustness and nonanticipativity can be satisfied in the uncertainty scheduling process of this method. After that, the stochastic dual dynamic integer programming (SDDiP) algorithm is introduced to solve multi-stage nested problems of complex systems. Finally, simulations are conducted in the actual system of three seasons and five scenarios. The conclusions are as follows: 1) The scheduling strategy provided by the MSDRO model and SDDiP algorithm can ensure effective operation based on satisfying robustness and nonanticipativity. 2) Compared with MSO, TSDRO and MSRO, the method proposed can improve the HESS economy and stability, increase renewable energy consumption and reduce carbon emissions.

Dual SDDP for risk-averse multistage stochastic programs

Risk-averse multistage stochastic programs appear in multiple areas and are challenging to solve. Stochastic Dual Dynamic Programming (SDDP) is a well-known tool to address such problems under time-independence assumptions. We show how to derive a dual formulation for these problems and apply an SDDP algorithm, leading to converging and deterministic upper bounds for risk-averse problems.

A Nonconvex Regularization Scheme for the Stochastic Dual Dynamic Programming Algorithm

A Nonconvex Regularization Scheme for the Stochastic Dual Dynamic Programming Algorithm

Hydropower Aggregation by Spatial Decomposition—An SDDP Approach

The balance between detailed technical description, representation of uncertainty and computational complexity is central in long-term scheduling models applied to hydro-dominated power system. The aggregation of complex hydropower systems into equivalent energy representations (EER) is a commonly used technique to reduce dimensionality and computation time in scheduling models. This work presents a method for coordinating the EERs with their detailed hydropower system representation within a model based on stochastic dual dynamic programming (SDDP). SDDP is applied to an EER representation of the hydropower system, where feasibility cuts derived from optimization of the detailed hydropower are used to constrain the flexibility of the EERs. These cuts can be computed either before or during the execution of the SDDP algorithm and allow system details to be captured within the SDDP strategies without compromising the convergence rate and state-space dimensionality. Results in terms of computational performance and system operation are reported from a test system comprising realistic hydropower watercourses.

A stochastic dual dynamic integer programming based approach for remanufacturing planning under uncertainty

ABSTRACT We seek to optimize the production planning of a three-echelon remanufacturing system under uncertain input data. We consider a multi-stage stochastic integer programming approach and use scenario trees to represent the uncertain information structure. We introduce a new dynamic programming formulation that relies on a partial nested decomposition of the scenario tree. We then propose a new approximate stochastic dual dynamic integer programming algorithm based on this partial decomposition. Our numerical results show that the proposed solution approach is able to provide near-optimal solutions for large-size instances with a reasonable computational effort.

A variant SDDP approach for periodic-review approximately optimal pricing of a slow-moving a item in a duopoly under price protection with end-of-life return and retail fixed markdown policy

In this paper, we examine a selling environment where a manufacturer-controlled retailer and an independent retailer sell a slow-moving A item. The manufacturer offers the independent retailer a price protection contract stipulating that the manufacturer reimburses the independent retailer in case of a reduction in the wholesale price. The price set by the independent retailer is assumed to be determined by Retail Fixed Markdown (RFM) policy. The manufacturer also offers the independent retailer a special discount rate for the replenishment orders and the retailers are assumed to follow (R, S) inventory replenishment policy. The manufacturer adopts a periodic-review pricing strategy and the mean demand observed by each retailer in a given period depends on the prices. We also take the customers choosing no-purchase option into account. We employ multinomial logit (MNL) models to forecast customers’ preferences based on retail prices. The retailers’ market shares are estimated by customized choice probability functions. We propose stochastic programming models to determine the manufacturer’s pricing strategy. Then, we propose a variant Stochastic Dual Dynamic Programming (SDDP) algorithm to determine the manufacturer’s approximately optimal pricing strategy by getting around three curses of dimensionality. Then, we move on to the observations on the impact of four critically important contractual parameters on the price, the market shares and the expected total net profits and finally discuss some possible approaches for the selection of the best compromise values of those contractual parameters.

Robust Portfolio Optimization with Respect to Spectral Risk Measures Under Correlation Uncertainty

This paper proposes a distributionally robust multi-period portfolio model with ambiguity on asset correlations with fixed individual asset return mean and variance. The correlation matrix bounds can be quantified via corresponding confidence intervals based on historical data. We employ a general class of coherent risk measures namely the spectral risk measure, which includes the popular measure conditional value-at-risk (CVaR) as a particular case, as our objective function. Specific choices of spectral risk measure permit flexibility for capturing risk preferences of different investors. A semi-analytical solution is derived for our model. The prominent stochastic dual dynamic programming (SDDP) algorithm adapted with intricate modifications is developed as a numerical method under the discrete distribution setting. In particular, our new formulation accounts for the unknown worst-case distribution in each iteration. We verify the convergence property of this algorithm under the setting of finite scenarios. Our results show that the optimal solution favours a certain degree of anti-diversification due to dependence ambiguity and exhibits its protection ability during the financial crisis period.

Optimizing the Management of Multireservoir Systems Under Shifting Flow Regimes

AbstractOver the past few decades, significant research efforts have been devoted to the development of tools and techniques to improve the operational effectiveness of multireservoir systems. One of those efforts focuses on the incorporation of relevant hydrologic information into reservoir operation models. This effort is particularly relevant in regions characterized by low‐frequency climate signals, where time series of river discharges exhibit regime‐like behavior. Failure to properly capture such regime‐like behavior yields suboptimal operating policies, especially in systems characterized by large storage capacity such as large multireservoir systems. Hidden Markov Modeling is a class of hydrological models that can accommodate both overdispersion and serial dependence in time series, two essential hydrological properties that must be captured when modeling a system where the climate is switching between different states (e.g., dry, normal, and wet). In terms of reservoir operation, Stochastic Dual Dynamic Programming (SDDP) is one of the few optimization techniques that can accommodate both system and hydrologic complexity, that is, a large number of reservoirs and diverse hydrologic information. However, current SDDP formulations are unable to capture the long‐term persistence of the streamflow process found in some regions. In this paper, we present an extension of the SDDP algorithm that can handle the long‐term persistence and provide reservoir operating policies that explicitly capture regime shifts. Using the Senegal River Basin as a case study, we illustrate the potential gain associated with reservoir operating policies tailored to climate states.

Multi-stage stochastic optimization of carbon risk management

Emissions trading within the Emissions Trading Scheme of the European Union (EU ETS) strongly influences European industrial companies. The companies must choose their strategy of reduction the costs of emissions allowances as possible. The changing system’s conditions and volatile prices of allowances make this decision challenging. The main aim of this study is to compare different ways of risk management: banking (i.e., buying the allowances in forward) and using derivatives: futures and options. Despite several studies devoted to the relationship between the EU ETS and companies have already been published, there is still a gap in this field. Namely, the published studies have been substantially simplified so far by ignoring the risk of driving parameters. We construct a realistic large-scale stochastic optimization model, which avoids the mentioned simplifications. We use the Markov Stochastic Dual Dynamic Programming algorithm (MSDDP) to find the optimal solution. We apply the model to the data of a real-life industrial company. We find that banking is the most costly way of risk reduction, while using derivatives is efficient in risk reduction. Surprisingly, out of the derivatives, it is always optimal to use futures and not to use options. These results are confirmed by a thorough sensitivity analysis. The preference of the futures over options is mainly due to the less price of futures in comparison to options reducing risk equivalently.

Multistage Stochastic Programming for VPP Trading in Continuous Intraday Electricity Markets

The stochastic nature of renewable energy sources has increased the need for intraday trading in electricity markets. Intradaymarkets provide the possibility to the market participants to modify their market positions based on their updated forecasts. In this paper, we propose a multistage stochastic programming approach to model the trading of a Virtual Power Plant (VPP), comprising thermal, wind and hydro power plants, in the Continuous Intraday (CID) electricity market. The order clearing in the CID market is enabled by the two presented models, namely the Immediate Order Clearing (IOC) and the Partial Order Clearing (POC). We tackle the proposed problem with a modified version of Stochastic Dual Dynamic Programming (SDDP) algorithm. The functionality of our model is demonstrated by performing illustrative and large scale case studies and comparing the performance with a benchmark model.

Distributed Multistage Optimization of Large-Scale Microgrids Under Stochasticity

Microgrids are recognized as a relevant tool to absorb decentralized renewable energies in the energy mix. However, the sequential handling of multiple stochastic productions and demands, and of storage, make their management a delicate issue. We add another layer of complexity by considering microgrids where different buildings stand at the nodes of a network and are connected by the arcs; some buildings host local production and storage capabilities, and can exchange with others their energy surplus. We formulate the problem as a multistage stochastic optimization problem, corresponding to the minimization of the expected temporal sum of operational costs, while satisfying the energy demand at each node, for all time. The resulting mathematical problem has a large-scale nature, exhibiting both spatial and temporal couplings. However, the problem displays a network structure that makes it amenable to a mix of spatial decomposition-coordination with temporal decomposition methods. We conduct numerical simulations on microgrids of different sizes and topologies, with up to 48 nodes and 64 state variables. Decomposition methods are faster and provide more efficient policies than a state-of-the-art Stochastic Dual Dynamic Programming algorithm. Moreover, they scale almost linearly with the state dimension, making them a promising tool to address more complex microgrid optimal management problems.

Combining Polyhedral Approaches and Stochastic Dual Dynamic Integer Programming for Solving the Uncapacitated Lot-Sizing Problem Under Uncertainty

We study the uncapacitated lot-sizing problem with uncertain demand and costs. The problem is modeled as a multistage stochastic mixed-integer linear program in which the evolution of the uncertain parameters is represented by a scenario tree. To solve this problem, we propose a new extension of the stochastic dual dynamic integer programming algorithm (SDDiP). This extension aims at being more computationally efficient in the management of the expected cost-to-go functions involved in the model, in particular by reducing their number and by exploiting the current knowledge on the polyhedral structure of the stochastic uncapacitated lot-sizing problem. The algorithm is based on a partial decomposition of the problem into a set of stochastic subproblems, each one involving a subset of nodes forming a subtree of the initial scenario tree. We then introduce a cutting plane–generation procedure that iteratively strengthens the linear relaxation of these subproblems and enables the generation of an additional strengthened Benders’ cut, which improves the convergence of the method. We carry out extensive computational experiments on randomly generated large-size instances. Our numerical results show that the proposed algorithm significantly outperforms the SDDiP algorithm at providing good-quality solutions within the computation time limit. Summary of Contribution: This paper investigates a combinatorial optimization problem called the uncapacitated lot-sizing problem. This problem has been widely studied in the operations research literature as it appears as a core subproblem in many industrial production planning problems. We consider a stochastic extension in which the input parameters are subject to uncertainty and model the resulting stochastic optimization problem as a multistage stochastic integer program. To solve this stochastic problem, we propose a novel extension of the recently published stochastic dual dynamic integer programming (SDDiP) algorithm. The proposed extension relies on two main ideas: the use of a partial decomposition of the scenario tree and the exploitation of existing knowledge on the polyhedral structure of the stochastic uncapacitated lot-sizing problem. We provide the results of extensive computational experiments carried out on large-size randomly generated instances. These results show that the proposed extended algorithm significantly outperforms the SDDiP at providing good-quality solutions for the stochastic uncapacitated lot-sizing problem. Although the paper focuses on a basic lot-sizing problem, the proposed algorithmic framework may be useful to solve more complex practical production planning problems.

Parallel and distributed computing for stochastic dual dynamic programming

We study different parallelization schemes for the stochastic dual dynamic programming (SDDP) algorithm. We propose a taxonomy for these parallel algorithms, which is based on the concept of parallelizing by scenario and parallelizing by node of the underlying stochastic process. We develop a synchronous and asynchronous version for each configuration. The parallelization strategy in the parallelscenario configuration aims at parallelizing the Monte Carlo sampling procedure in the forward pass of the SDDP algorithm, and thus generates a large number of supporting hyperplanes in parallel. On the other hand, the parallel-node strategy aims at building a single hyperplane of the dynamic programming value function in parallel. The considered algorithms are implemented using Julia and JuMP on a high performance computing cluster. We study the effectiveness of the methods in terms of achieving tight optimality gaps, as well as the scalability properties of the algorithms with respect to an increasing number of CPUs. In particular, we study the effects of the different parallelization strategies on performance when increasing the number of Monte Carlo samples in the forward pass, and demonstrate through numerical experiments that such an increase may be harmful. Our results indicate that a parallel-node strategy presents certain benefits as compared to a parallel-scenario configuration.

Mature offshore oil field development: Solving a real options problem using stochastic dual dynamic integer programming

Oil and gas companies are facing low output prices and are forced to focus on the development of mature fields. Relevant investment decisions for operators include lifetime-enhancing activities, such as drilling new wells or permanent shutdown. We study the problem of optimal timing of investments in mature oil and gas fields in the presence of price uncertainty, which is an example of a complex real options problem consisting of a portfolio of interdependent options. We formulate a multistage stochastic integer programming model that incorporates a detailed representation of the uncertain oil price, and demonstrate how such a complex real options problem can be efficiently solved using the Stochastic Dual Dynamic Integer Programming algorithm. The paper presents a numerical example based on realistic data and discusses our computational results. We find that only a small number of Markov states are required to represent the uncertain price process, while obtaining convergence of the lower and upper bounds of the objective function. The value of stochastic solution of 11% is considerable in this example. It is concluded that the shutdown decision tends to be postponed as a result of high decommissioning costs, high discount rates, high price uncertainty and low operational expenditures, while it generally is accelerated if the decommissioning costs increase over time.