Stochastic Dual Dynamic Programming Algorithm Research Articles

Stochastic dual dynamic programming for optimal power flow problems under uncertainty

Planning in the power sector has to take into account the physical laws of alternating current (AC) power flows as well as uncertainty in the data of the problems, both of which greatly complicate optimal decision making. We propose a computationally tractable framework to solve multi-stage stochastic optimal power flow (OPF) problems in AC power systems. Our approach uses recent results on dual convex semi-definite programming (SDP) relaxations of OPF problems in order to adapt the stochastic dual dynamic programming (SDDP) algorithm for problems with a Markovian structure. We show that the usual SDDP lower bound remains valid and that the algorithm converges to a globally optimal policy of the stochastic AC-OPF problem as long as the SDP relaxations are tight. To test the practical viability of our approach, we set up a case study of a storage siting, sizing, and operations problem. We show that the convex SDP relaxation of the stochastic problem is usually tight and discuss ways to obtain near-optimal physically feasible solutions when this is not the case. The algorithm finds a physically feasible policy with an optimality gap of 3% and yields a significant added value of 27% over a rolling deterministic policy, which leads to overly optimistic policies and underinvestment in flexibility. This suggests that the common industry practice of assuming direct current and deterministic problems should be reevaluated by considering models that incorporate realistic AC flows and stochastic elements in the data as potentially more realistic alternatives.

Approximating hydropower systems by feasibility spaces in stochastic dual dynamic programming

This work investigates and improves methodology for approximating hydropower systems by feasibility spaces, which can be embedded in the stochastic dual dynamic programming algorithm and applied in the context long-term hydrothermal scheduling. The feasibility spaces are derived from optimization of the detailed hydropower system and are expressed in few dimensions to facilitate efficient computations. Test results from a case study for the Norwegian power system demonstrate how feasibility spaces serve to realistically constrain the hydropower system.

Solving constrained consumption–investment problems by decomposition algorithms

Consumption–investment problems with maximizing utility agents are usually considered from a theoretical viewpoint, aiming at closed-form solutions for the optimal policy. However, such an approach requires that the model be relatively simple: even the inclusion of nonnegativity constraints can prevent the derivation of explicit solutions. In such cases, it is necessary to solve the problem numerically, but standard dynamic programming algorithms can only solve small problems due to the curse of dimensionality. In this paper, we adapt the Stochastic Dual Dynamic Programming (SDDP) algorithm to solve dynamic constrained consumption–investment problems with stochastic labor income numerically. Unlike classical dynamic programming approaches, SDDP allows us to analyze problems with multiple assets, and an internal sampling procedure allows the problems to have a very large, or even infinite, number of scenarios. We start with a simpler problem for which a closed-form solution is known and compare it to the optimal policy obtained by SDDP. We then illustrate the flexibility of our approach by solving a defined contribution pension fund problem with multiple assets, for which no closed-form solution is available.

Uncertainty dynamics in energy planning models: An autoregressive and Markov chain modeling approach

This paper deals with the modeling of stochastic processes in long-term multi-stage energy planning problems when limited information is available about the distributions of such processes. Starting from simple estimates of variation intervals for uncertain parameters, such as energy demands and costs, we model the temporal correlation of these parameters through carefully constructed autoregressive (AR) models that respect the intervals defined in each period. We introduce a coefficient for “zigzag” effects in the evolution of uncertain processes, which controls the correlation across periods and also the likelihood of extreme scenarios. To preserve the convexity of the stochastic problem, we discretize the AR models associated with the cost parameters involved in the objective function by Markov chains. The resulting formulation is then solved using a Stochastic Dual Dynamic Programming (SDDP) algorithm available in the literature that handles finite-state Markov chains. Our numerical experiments, carried out on the Swiss energy system, show a very desirable adaptation strategy of investment decisions to uncertainty scenarios, a behavior that is not observed when the temporal correlation is ignored. Moreover, the solutions lead to better out-of-sample cost performances, especially on extreme scenario realizations, than the non-correlated ones which usually result in overcapacities to protect against high, but unlikely, parameter variations over time.

Medium-term stochastic hydrothermal scheduling with short-term operational effects for large-scale power and water networks

The high integration of variable renewable sources in electric power systems entails a series of challenges inherent to their intrinsic variability. A critical challenge is to correctly value the water available in reservoirs in hydrothermal systems, considering the flexibility that it provides. In this context, this paper proposes a medium-term multistage stochastic optimization model for the hydrothermal scheduling problem solved with the stochastic dual dynamic programming algorithm. The proposed model includes operational constraints and simplified mathematical expressions of relevant operational effects that allow more informed assessment of the water value by considering, among others, the flexibility necessary for the operation of the system. In addition, the hydrological uncertainty in the model is represented by a vector autoregressive process, which allows capturing spatio-temporal correlations between the different hydro inflows. A calibration method for the simplified mathematical expressions of operational effects is also proposed, which allows a detailed short-term operational model to be correctly linked to the proposed medium-term linear model. Through extensive experiments for the Chilean power system, the results show that the difference between the expected operating costs of the proposed medium-term model, and the costs obtained through a detailed short-term operational model was only 0.1%, in contrast to the 9.3% difference obtained when a simpler base model is employed. This shows the effectiveness of the proposed approach. Further, this difference is also reflected in the estimation of the water value, which is critical in water shortage situations.

Batch Learning SDDP for Long-Term Hydrothermal Planning

We consider the stochastic dual dynamic programming (SDDP) algorithm - a widely employed algorithm applied to multistage stochastic programming - and propose a variant using experience replay - a batch learning technique from reinforcement learning. To connect SDDP with reinforcement learning, we cast SDDP as a Q-learning algorithm and describe its application in both risk-neutral and risk-averse settings. We demonstrate the superiority of the algorithm over conventional SDDP by benchmarking it against PSR's SDDP software using a large-scale instance of the long-term planning problem of inter-connected hydropower plants in Colombia. We find that SDDP with batch learning is able to produce tighter optimality gaps in a shorter amount of time than conventional SDDP. We also find that batch learning improves the parallel efficiency of SDDP backward passes.

A stochastic policy algorithm for seasonal hydropower planning

AbstractHydropower producers need to plan several months or years ahead to estimate the opportunity value of water stored in their reservoirs. The resulting large-scale optimization problem is computationally intensive, and model simplifications are often needed to allow for efficient solving. Alternatively, one can look for near-optimal policies using heuristics that can tackle non-convexities in the production function and a wide range of modelling approaches for the price- and inflow dynamics. We undertake an extensive numerical comparison between the state-of-the-art algorithm stochastic dual dynamic programming (SDDP) and rolling forecast-based algorithms, including a novel algorithm that we develop in this paper. We name it Scenario-based Two-stage ReOptimization abbreviated as STRO. The numerical experiments are based on convex stochastic dynamic programs with discretized exogenous state space, which makes the SDDP algorithm applicable for comparisons. We demonstrate that our algorithm can handle inflow risk better than traditional forecast-based algorithms, by reducing the optimality gap from 2.5 to 1.3% compared to the SDDP bound.

Medium-term multi-stage distributionally robust scheduling of hydro–wind–solar complementary systems in electricity markets considering multiple time-scale uncertainties

Joint trading of hydro–wind–solar complementary systems (HWSCSs) in the electricity market (EM) helps to reduce the imbalance cost and increase profits. However, multiple energy resources and market price uncertainties affect the trading strategies. Existing medium-term (MT) scheduling approaches assume that the probability distribution of the random variable is perfectly known. Short-term variations were also ignored, which led to revenue loss and trading risk. To address the above issues, this paper proposes an MT multi-stage distributionally robust optimization (MDRO) scheduling approach for a price-taking HWSCS in the EM. Firstly, hourly unit commitment (HUC) constraints are incorporated into the MT scheduling model to accurately capture short-term variations. A novel ambiguity set is designed based on the modified chi-square distance to address probability distribution uncertainties at two different time scales. Subsequently, an MDRO scheduling model is proposed to optimize the trading strategy. Finally, the proposed MDRO model is converted to a large-scale multi-stage integer programming problem based on linearization and reformation. The stochastic dual dynamic integer programming algorithm is modified to ensure computational tractability. Xiluodu-Xiangjiaba HWSCS, located in the Jinsha River in China, was selected as a case study. The results show that: 1) the MDRO model is more robust to distributional uncertainties than the multi-stage stochastic programming (MSSP) model. When the probability distribution of the random variable changes, the MDRO model yields a higher expected revenue (+2.43%) and a lower standard deviation (-60.8%) of revenue, which illustrates lower trading risk. 2) Compared with MSSP, deterministic, two-stage stochastic programming, and distributionally robust optimization models, the MDRO model exhibits the best out-of-sample performance in terms of the highest expected revenue and lowest trading risk. 3) Incorporating HUC constraints into the MDRO model helps to increase the total revenue (+3.53%) and energy generation (+3.31%) at the expense of increasing the computational burden.

A Nonconvex Regularization Scheme for the Stochastic Dual Dynamic Programming Algorithm

We propose a new nonconvex regularization scheme to improve the performance of the stochastic dual dynamic programming (SDDP) algorithm for solving large-scale multistage stochastic programs. Specifically, we use a class of nonconvex regularization functions, namely folded concave penalty functions, to improve solution quality and the convergence rate of the SDDP procedure. We develop a strategy based on mixed-integer programming to guarantee global optimality of the nonconvex regularization problem. Moreover, we establish provable convergence guarantees for our customized SDDP algorithm. The benefits of our regularization scheme are demonstrated by solving large-scale instances of two multistage stochastic optimization problems. History: Pascal Van Hentenryck, Area Editor for Computational Modeling: Methods & Analysis. Supplemental Material: The software that supports the findings of this study is available within the paper and its Supplemental Information ( https://pubsonline.informs.org/doi/suppl/10.1287/ijoc.2021.0255 ) as well as from the IJOC GitHub software repository ( https://github.com/INFORMSJoC/2021.0255 ). The complete IJOC Software and Data Repository is available at https://informsjoc.github.io/ .

Multi-stage distributionally robust optimization for hybrid energy storage in regional integrated energy system considering robustness and nonanticipativity

Hybrid energy storage system (HESS) has advantages in coping with the uncertainty of renewable energy and improving the stability of regional integrated energy systems (RIES). Few of the current HESS uncertainty scheduling methods consider both robustness and nonanticipativity, which may result in scheduling strategies that are not feasible in practice. Therefore, this research constructs a HESS (containing electric, thermal, hydrogen and natural gas storage devices) and forms a hybrid energy storage operator (IHESO) with an independent market position based on the HESS to provide energy services. Subsequently, a novel multi-stage distributionally robust optimization (MSDRO) method is established to maximize HESS revenue. Robustness and nonanticipativity can be satisfied in the uncertainty scheduling process of this method. After that, the stochastic dual dynamic integer programming (SDDiP) algorithm is introduced to solve multi-stage nested problems of complex systems. Finally, simulations are conducted in the actual system of three seasons and five scenarios. The conclusions are as follows: 1) The scheduling strategy provided by the MSDRO model and SDDiP algorithm can ensure effective operation based on satisfying robustness and nonanticipativity. 2) Compared with MSO, TSDRO and MSRO, the method proposed can improve the HESS economy and stability, increase renewable energy consumption and reduce carbon emissions.

Dual SDDP for risk-averse multistage stochastic programs

Risk-averse multistage stochastic programs appear in multiple areas and are challenging to solve. Stochastic Dual Dynamic Programming (SDDP) is a well-known tool to address such problems under time-independence assumptions. We show how to derive a dual formulation for these problems and apply an SDDP algorithm, leading to converging and deterministic upper bounds for risk-averse problems.

A Nonconvex Regularization Scheme for the Stochastic Dual Dynamic Programming Algorithm

A Nonconvex Regularization Scheme for the Stochastic Dual Dynamic Programming Algorithm

Hydropower Aggregation by Spatial Decomposition—An SDDP Approach

The balance between detailed technical description, representation of uncertainty and computational complexity is central in long-term scheduling models applied to hydro-dominated power system. The aggregation of complex hydropower systems into equivalent energy representations (EER) is a commonly used technique to reduce dimensionality and computation time in scheduling models. This work presents a method for coordinating the EERs with their detailed hydropower system representation within a model based on stochastic dual dynamic programming (SDDP). SDDP is applied to an EER representation of the hydropower system, where feasibility cuts derived from optimization of the detailed hydropower are used to constrain the flexibility of the EERs. These cuts can be computed either before or during the execution of the SDDP algorithm and allow system details to be captured within the SDDP strategies without compromising the convergence rate and state-space dimensionality. Results in terms of computational performance and system operation are reported from a test system comprising realistic hydropower watercourses.

A stochastic dual dynamic integer programming based approach for remanufacturing planning under uncertainty

ABSTRACT We seek to optimize the production planning of a three-echelon remanufacturing system under uncertain input data. We consider a multi-stage stochastic integer programming approach and use scenario trees to represent the uncertain information structure. We introduce a new dynamic programming formulation that relies on a partial nested decomposition of the scenario tree. We then propose a new approximate stochastic dual dynamic integer programming algorithm based on this partial decomposition. Our numerical results show that the proposed solution approach is able to provide near-optimal solutions for large-size instances with a reasonable computational effort.

A variant SDDP approach for periodic-review approximately optimal pricing of a slow-moving a item in a duopoly under price protection with end-of-life return and retail fixed markdown policy

In this paper, we examine a selling environment where a manufacturer-controlled retailer and an independent retailer sell a slow-moving A item. The manufacturer offers the independent retailer a price protection contract stipulating that the manufacturer reimburses the independent retailer in case of a reduction in the wholesale price. The price set by the independent retailer is assumed to be determined by Retail Fixed Markdown (RFM) policy. The manufacturer also offers the independent retailer a special discount rate for the replenishment orders and the retailers are assumed to follow (R, S) inventory replenishment policy. The manufacturer adopts a periodic-review pricing strategy and the mean demand observed by each retailer in a given period depends on the prices. We also take the customers choosing no-purchase option into account. We employ multinomial logit (MNL) models to forecast customers’ preferences based on retail prices. The retailers’ market shares are estimated by customized choice probability functions. We propose stochastic programming models to determine the manufacturer’s pricing strategy. Then, we propose a variant Stochastic Dual Dynamic Programming (SDDP) algorithm to determine the manufacturer’s approximately optimal pricing strategy by getting around three curses of dimensionality. Then, we move on to the observations on the impact of four critically important contractual parameters on the price, the market shares and the expected total net profits and finally discuss some possible approaches for the selection of the best compromise values of those contractual parameters.

Robust Portfolio Optimization with Respect to Spectral Risk Measures Under Correlation Uncertainty

This paper proposes a distributionally robust multi-period portfolio model with ambiguity on asset correlations with fixed individual asset return mean and variance. The correlation matrix bounds can be quantified via corresponding confidence intervals based on historical data. We employ a general class of coherent risk measures namely the spectral risk measure, which includes the popular measure conditional value-at-risk (CVaR) as a particular case, as our objective function. Specific choices of spectral risk measure permit flexibility for capturing risk preferences of different investors. A semi-analytical solution is derived for our model. The prominent stochastic dual dynamic programming (SDDP) algorithm adapted with intricate modifications is developed as a numerical method under the discrete distribution setting. In particular, our new formulation accounts for the unknown worst-case distribution in each iteration. We verify the convergence property of this algorithm under the setting of finite scenarios. Our results show that the optimal solution favours a certain degree of anti-diversification due to dependence ambiguity and exhibits its protection ability during the financial crisis period.

Optimizing the Management of Multireservoir Systems Under Shifting Flow Regimes

AbstractOver the past few decades, significant research efforts have been devoted to the development of tools and techniques to improve the operational effectiveness of multireservoir systems. One of those efforts focuses on the incorporation of relevant hydrologic information into reservoir operation models. This effort is particularly relevant in regions characterized by low‐frequency climate signals, where time series of river discharges exhibit regime‐like behavior. Failure to properly capture such regime‐like behavior yields suboptimal operating policies, especially in systems characterized by large storage capacity such as large multireservoir systems. Hidden Markov Modeling is a class of hydrological models that can accommodate both overdispersion and serial dependence in time series, two essential hydrological properties that must be captured when modeling a system where the climate is switching between different states (e.g., dry, normal, and wet). In terms of reservoir operation, Stochastic Dual Dynamic Programming (SDDP) is one of the few optimization techniques that can accommodate both system and hydrologic complexity, that is, a large number of reservoirs and diverse hydrologic information. However, current SDDP formulations are unable to capture the long‐term persistence of the streamflow process found in some regions. In this paper, we present an extension of the SDDP algorithm that can handle the long‐term persistence and provide reservoir operating policies that explicitly capture regime shifts. Using the Senegal River Basin as a case study, we illustrate the potential gain associated with reservoir operating policies tailored to climate states.

Multi-stage stochastic optimization of carbon risk management

Emissions trading within the Emissions Trading Scheme of the European Union (EU ETS) strongly influences European industrial companies. The companies must choose their strategy of reduction the costs of emissions allowances as possible. The changing system’s conditions and volatile prices of allowances make this decision challenging. The main aim of this study is to compare different ways of risk management: banking (i.e., buying the allowances in forward) and using derivatives: futures and options. Despite several studies devoted to the relationship between the EU ETS and companies have already been published, there is still a gap in this field. Namely, the published studies have been substantially simplified so far by ignoring the risk of driving parameters. We construct a realistic large-scale stochastic optimization model, which avoids the mentioned simplifications. We use the Markov Stochastic Dual Dynamic Programming algorithm (MSDDP) to find the optimal solution. We apply the model to the data of a real-life industrial company. We find that banking is the most costly way of risk reduction, while using derivatives is efficient in risk reduction. Surprisingly, out of the derivatives, it is always optimal to use futures and not to use options. These results are confirmed by a thorough sensitivity analysis. The preference of the futures over options is mainly due to the less price of futures in comparison to options reducing risk equivalently.

Multistage Stochastic Programming for VPP Trading in Continuous Intraday Electricity Markets

The stochastic nature of renewable energy sources has increased the need for intraday trading in electricity markets. Intradaymarkets provide the possibility to the market participants to modify their market positions based on their updated forecasts. In this paper, we propose a multistage stochastic programming approach to model the trading of a Virtual Power Plant (VPP), comprising thermal, wind and hydro power plants, in the Continuous Intraday (CID) electricity market. The order clearing in the CID market is enabled by the two presented models, namely the Immediate Order Clearing (IOC) and the Partial Order Clearing (POC). We tackle the proposed problem with a modified version of Stochastic Dual Dynamic Programming (SDDP) algorithm. The functionality of our model is demonstrated by performing illustrative and large scale case studies and comparing the performance with a benchmark model.

Distributed Multistage Optimization of Large-Scale Microgrids Under Stochasticity

Microgrids are recognized as a relevant tool to absorb decentralized renewable energies in the energy mix. However, the sequential handling of multiple stochastic productions and demands, and of storage, make their management a delicate issue. We add another layer of complexity by considering microgrids where different buildings stand at the nodes of a network and are connected by the arcs; some buildings host local production and storage capabilities, and can exchange with others their energy surplus. We formulate the problem as a multistage stochastic optimization problem, corresponding to the minimization of the expected temporal sum of operational costs, while satisfying the energy demand at each node, for all time. The resulting mathematical problem has a large-scale nature, exhibiting both spatial and temporal couplings. However, the problem displays a network structure that makes it amenable to a mix of spatial decomposition-coordination with temporal decomposition methods. We conduct numerical simulations on microgrids of different sizes and topologies, with up to 48 nodes and 64 state variables. Decomposition methods are faster and provide more efficient policies than a state-of-the-art Stochastic Dual Dynamic Programming algorithm. Moreover, they scale almost linearly with the state dimension, making them a promising tool to address more complex microgrid optimal management problems.