Bayesian Stochastic Dynamic Programming Research Articles

Information theory for ranking and selection

AbstractWe study the classical ranking and selection problem, where the ultimate goal is to find the unknown best alternative in terms of the probability of correct selection or expected opportunity cost. However, this paper adopts an alternative sampling approach to achieve this goal, where sampling decisions are made with the objective of maximizing information about the unknown best alternative, or equivalently, minimizing its Shannon entropy. This adaptive learning is formulated via a Bayesian stochastic dynamic programming problem, by which several properties of the learning problem are presented, including the monotonicity of the optimal value function in an information‐seeking setting. Since the state space of the stochastic dynamic program is unbounded in the Gaussian setting, a one‐step look‐ahead approach is used to develop a policy. The proposed policy seeks to maximize the one‐step information gain about the unknown best alternative, and therefore, it is called information gradient (IG). It is also proved that the IG policy is consistent, that is, as the sampling budget grows to infinity, the IG policy finds the true best alternative almost surely. Later, a computationally efficient estimate of the proposed policy, called approximated information gradient (AIG), is introduced and in the numerical experiments its performance is tested against recent benchmarks alongside several sensitivity analyses. Results show that AIG performs competitively against other algorithms from the literature.

Bayesian Stochastic Dynamic Programming for Hydropower Generation Operation Based on Copula Functions

Bayesian stochastic dynamic programming (BSDP) has been widely used in hydropower generation operation, as natural inflow and forecast uncertainties can be easily determined by transition probabilities. In this study, we propose a theoretical estimation method (TEM) based on copula functions to calculate the transition probability under conditions of limited historical inflow samples. The explicit expression of the conditional probability is derived using copula functions and then used to calculate prior and likelihood probabilities, and the prior probability can be revised to the posterior probability once new forecast information is available by Bayesian formulation. The performance of BSDP models in seven forecast scenarios and two extreme conditions considering no or perfect forecast information is evaluated and compared. The case study in the Ertan hydropower station in China shows that (1) TEM can avoid the shortcomings of empirical estimation method (EMM) in calculating the transition probability, so that the prior and likelihood probability matrices can be distributed more uniformly with less zeros, and the problem that the posterior probability cannot be calculated can be avoided; (2) there is a positive correlation between operating benefit and forecast accuracy; and (3) the operating policy considering reliable forecast information can improve hydropower generation. However, an incorrect decision may be made in the case of low forecast accuracy.

A reservoir operation method that accounts for different inflow forecast uncertainties in different hydrological periods

Inflow forecast is an important input for reservoir operation. Due to limited forecast techniques, the inflow forecast always has uncertainties (usually in the form of forecast errors). Handling inflow forecast uncertainties effectively is important for optimizing reservoir operation. The magnitude of this uncertainty depends on both the forecast lead times and the hydrological periods (i.e., wet season vs. dry season). Two-stage Bayesian stochastic dynamic programming (TBSDP) is an effective way to address the forecast uncertainties. However, current TBSDP methods pay more attention to the uncertainty differences caused by forecast lead times, neglecting the uncertainty differences in wet and dry seasons. Starting with a TBSDP model, we proposed a new two-stage and two-period version of the model (TTBSDP), which can not only consider the forecast uncertainty differences caused by forecast lead times, but also separately accounts for forecast uncertainty in wet and dry seasons. Because dealing separately with the uncertainties in the two periods increases the computational complexity, we further explored the conditions under which degree of forecast uncertainties this separation is necessary. We compared our new model’s results (in terms of cumulative annual power generation) with the previous Bayesian stochastic dynamic programming models. With increasing forecast uncertainty in the first stage of the overall forecast horizon (in the wet season), the TTBSDP model produced superior results at higher uncertainty, with the most stable performance (the smallest variation of cumulative annual power generation when the forecast uncertainty coefficient increases). The proposed new model is benefit for hydropower operations when the magnitude of forecast uncertainty in the wet seasons is large.

Effects of reservoir operation methods on downstream ecological disturbance and economic benefits

AbstractThe natural flow regime can sustain the ecological integrity of riverine ecosystems. Different reservoir operation polices differ in their effects on the degree of alteration of natural flow regimes. Dynamic programming plays an important role in developing operation policies. When using dynamic programming models to develop operation policies, the discrete number of storage states (DNSS), which is a key factor affecting the reservoirs operation policies, is always been determined based on computational efficiency and economic benefits. Little consideration has been given to the ecological disturbance caused by different DNSS‐based operation policies. To analyze the impact of DNSS, we built a deterministic dynamic programming model to explore the relationship among DNSS, the flow regime alteration (ecological disturbance), and the cumulative annual power generation (economic benefits) by setting a range of DNSS scenarios. We used three reservoirs with different storage coefficients (ratios of usable storage to annual average runoff) as examples and used the range of variability approach to assess the ecological disturbance under these scenarios. We compared the results with those of a stochastic dynamic programming (SDP) model and a Bayesian SDP (BSDP) model. We found that when DNSS is low, increasing DNSS improves economic benefits but causes a more severe ecological disturbance; when DNSS is high, increasing DNSS improves the economic benefits only slightly, without exacerbating the ecological disturbance; for a given DNSS, the BSDP model provides higher economic benefits than the SDP model and a similar disturbance of the riverine ecosystem; and larger reservoirs more often cause more severe disturbance of riverine ecosystems because monthly mean flows and annual extreme flows change more drastically. Our results will help to protect the riverine ecosystems and improve economic benefits if reservoir operation managers consider DNSS using dynamic programming models.

Water Management in Tarbela Dam By using Bayesian Stochastic Dynamic Programming in Extreme Inflow Season

Existing method of forecasting inflows at Tarbela have some limitations, also system needs an adequate operating policy model to deal with highly volatile inflow of summer months of June, July, August and September. In this paper, historical data of inflows from 1986 to 2014 have been used to forecast upcoming inflows at dam. Bayesian predictive distribution is used to predict future inflows. These forecasted inflows were further incorporated into operating policy model to determine the optimal release during the prescribed months. Weather volatility is a major factor causing unstable inflows. High temperature during summer period cause high inflows at dam. Considering weather volatility, this policy model is proposed for the flood season (15th June to 30th September), in which inflows and outflows are higher than rest of the year. This model maximizes the expected profit from hydro power production, minimizes the expected loss from flood damage and updates the proper estimate of current stage of reservoir storage.

An aggregation-decomposition bayesian stochastic optimization model for cascade hydropower reservoirs using medium-range precipitation forecasts

The forecast information is essential to improve the utilization efficiency of hydropower resources. To address the uncertainties of forecasting inflow, the Aggregation-Decomposition Bayesian Stochastic Dynamic Programming (AD-BSDP) model is presented in the present paper by using the 10-days precipitation value of the Quantitative Precipitation Forecasts from Global Forecast System (GFS-QPFs). The application in China’s Hun River cascade hydropower reservoirs shows that the GFS-QPFs are beneficial for hydropower generation and the performance of AD-BSDP is more efficiency and reliability than the others models.

Bayesian learning of dose–response parameters from a cohort under response-guided dosing

There has been a surge of clinical interest in the idea of response-guided dosing (RGD). The goal in RGD is to tailor drug-doses to the stochastic evolution of each individual patient’s disease condition over the treatment course. The hope is that this form of individualized therapy will deliver the right dose to the right patient at the right time. Several expert panels have observed that despite the excitement surrounding RGD, quantitative, data-driven decision-making approaches that learn patients’ dose–response and incorporate this information into adaptive dosing strategies are lagging behind. This situation is particularly exacerbated in clinical trials. For instance, fixed design clinical studies for estimating the key parameter of a dose–response function might not treat trial patients optimally. Similarly, the dosing strategies employed in clinical trials for RGD often appear ad-hoc.We study the problem of finding optimal RGD policies while learning the distribution of a dose–response parameter from a cohort of patients. We provide a Bayesian stochastic dynamic programming (DP) formulation of this problem. Exact solution of Bellman’s equations for this problem is computationally intractable. We therefore present two approximate control schemes and mathematically analyze the monotonicity, stationarity, and separability structures of the resulting dosing strategies. These structures are then exploited in efficient, approximate solution of our problem. Computer simulations using the Michaelis–Menten dose–response function are included as an example wherein we study the effect of cohort size and of prior misspecification.

Two Monthly Continuous Dynamic Model Based on Nash Bargaining Theory for Conflict Resolution in Reservoir System

So far many optimization models based on Nash Bargaining Theory associated with reservoir operation have been developed. Most of them have aimed to provide practical and efficient solutions for water allocation in order to alleviate conflicts among water users. These models can be discussed from two viewpoints: (i) having a discrete nature; and (ii) working on an annual basis. Although discrete dynamic game models provide appropriate reservoir operator policies, their discretization of variables increases the run time and causes dimensionality problems. In this study, two monthly based non-discrete optimization models based on the Nash Bargaining Solution are developed for a reservoir system. In the first model, based on constrained state formulation, the first and second moments (mean and variance) of the state variable (water level in the reservoir) is calculated. Using moment equations as the constraint, the long-term utility of the reservoir manager and water users are optimized. The second model is a dynamic approach structured based on continuous state Markov decision models. The corresponding solution based on the collocation method is structured for a reservoir system. In this model, the reward function is defined based on the Nash Bargaining Solution. Indeed, it is used to yield equilibrium in every proper sub-game, thereby satisfying the Markov perfect equilibrium. Both approaches are applicable for water allocation in arid and semi-arid regions. A case study was carried out at the Zayandeh-Rud river basin located in central Iran to identify the effectiveness of the presented methods. The results are compared with the results of an annual form of dynamic game, a classical stochastic dynamic programming model (e.g. Bayesian Stochastic Dynamic Programming model, BSDP), and a discrete stochastic dynamic game model (PSDNG). By comparing the results of alternative methods, it is shown that both models are capable of tackling conflict issues in water allocation in situations of water scarcity properly. Also, comparing the annual dynamic game models, the presented models result in superior results in practice. Furthermore, unlike discrete dynamic game models, the presented models can significantly reduce the runtime thereby avoiding dimensionality problems.

Optimal Bayesian Learning of Dose-Response Parameters from a Cohort

There has been a surge of clinical interest in the idea of response-guided dosing (RGD). The goal in RGD is to tailor drug-doses to the stochastic evolution of each individual patient's disease condition over the treatment course. The hope is that this form of individualized therapy will deliver the right dose to the right patient at the right time. Several expert panels have observed that despite the excitement surrounding RGD, quantitative, data-driven decision-making approaches that learn patients' dose-response and incorporate this information into adaptive dosing strategies are lagging behind. This situation is particularly exacerbated in clinical trials. For instance, fixed design clinical studies for estimating the key parameter of a dose-response function might not treat trial patients optimally. Similarly, the dosing strategies employed in clinical trials for RGD often appear ad-hoc. In this paper, we study the problem of finding optimal RGD policies while learning the distribution of a dose-response parameter from a cohort of patients. We provide a Bayesian stochastic dynamic programming (DP) formulation of this problem. Exact solution of Bellman's equations for this problem is computationally intractable. We therefore present two approximate control schemes and mathematically analyze the monotonicity, stationarity, and separability structures of the resulting dosing strategies. These structures are then exploited in efficient, approximate solution of the optimal learning problem. Computer simulations using the Michaelis-Menten dose-response function are included as an example wherein we study the effect of cohort size and prior misspecification on learning, and also compare the performance of various dosing policies.

A two stage Bayesian stochastic optimization model for cascaded hydropower systems considering varying uncertainty of flow forecasts

AbstractThis paper presents a new Two Stage Bayesian Stochastic Dynamic Programming (TS‐BSDP) model for real time operation of cascaded hydropower systems to handle varying uncertainty of inflow forecasts from Quantitative Precipitation Forecasts. In this model, the inflow forecasts are considered as having increasing uncertainty with extending lead time, thus the forecast horizon is divided into two periods: the inflows in the first period are assumed to be accurate, and the inflows in the second period assumed to be of high uncertainty. Two operation strategies are developed to derive hydropower operation policies for the first and the entire forecast horizon using TS‐BSDP. In this paper, the newly developed model is tested on China's Hun River cascade hydropower system and is compared with three popular stochastic dynamic programming models. Comparative results show that the TS‐BSDP model exhibits significantly improved system performance in terms of power generation and system reliability due to its explicit and effective utilization of varying degrees of inflow forecast uncertainty. The results also show that the decision strategies should be determined considering the magnitude of uncertainty in inflow forecasts. Further, this study confirms the previous finding that the benefit in hydropower generation gained from the use of a longer horizon of inflow forecasts is diminished due to higher uncertainty and further reveals that the benefit reduction can be substantially mitigated through explicit consideration of varying magnitudes of forecast uncertainties in the decision‐making process.

Development of Integrated Drought Evaluation and Monitoring System: Case Study of Aharchay River Basin

AbstractIn this paper, an integrated drought management scheme (IDMS) is developed for the Aharchay River Basin, located in northwestern Iran. This region is located in a semiarid region and has experienced considerable drought damage in recent years. Therefore, application of the proposed IDMS could greatly help to mitigate drought damage. The developed IDMS includes different tools and models needed for drought management. The utilized tools are classified into four groups, as follows: (1) data collection and validation, (2) rainfall-runoff prediction, (3) reservoir operation, and (4) drought analysis and monitoring. Future rainfall and runoff are projected using two computer models. Optimal reservoir operation is determined using a Bayesian stochastic genetic algorithm and Bayesian stochastic dynamic programming. These models are capable of incorporating new data of reservoir inflow and expert opinion in decision-making. For reservoir operation in drought conditions, hedging rules are also developed ba...

A Novel Solution for Stochastic Dynamic Game of Water Allocation from a Reservoir Using Collocation Method

In this study, a continuous model of stochastic dynamic game for water allocation from a reservoir system was developed. The continuous random variable of inflow in the state transition function was replaced with a discrete approximant rather than using the mean of the random variable as is done in a continuous model of deterministic dynamic game. As a result, a new solution method was used to solve the stochastic model of game based on collocation method. The collocation method was introduced as an alternative to linear-quadratic (LQ) approximation methods to resolve a dynamic model of game. The collocation method is not limited to the first and second degree approximations, compared to LQ approximation, i.e. Ricatti equations. Furthermore, in spite of LQ related problems, consideration of the stochastic nature of game on the action variables in the collocation method would be possible. The proposed solution method was applied to the real case of reservoir operation, which typically requires considering the effect of uncertainty on decision variables. The results of the solution of the stochastic model of game are compared with the results of a deterministic solution of game, a classical stochastic dynamic programming model (e.g. Bayesian Stochastic Dynamic Programming model, BSDP), and a discrete stochastic dynamic game model (PSDNG). By comparing the results of alternative methods, it is shown that the proposed solution method of stochastic dynamic game is quite capable of providing appropriate reservoir operating policies.

A Bayesian Stochastic Optimization Model for a Multi-Reservoir Hydropower System

This paper presents the development of an operating policy model for a multi-reservoir system for hydropower generation by addressing forecast uncertainty along with inflow uncertainty. The stochastic optimization tool adopted is the Bayesian Stochastic Dynamic Programming (BSDP), which incorporates a Bayesian approach within the classical Stochastic Dynamic Programming (SDP) formulation. The BSDP model developed in this study considers, the storages of individual reservoirs at the beginning of period t, aggregate inflow to the system during period t and forecast for aggregate inflow to the system for the next time period t + 1, as state variables. The randomness of the inflow is addressed through a posterior flow transition probability, and the uncertainty in flow forecasts is addressed through both the posterior flow transition probability and the predictive probability of forecasts. The system performance measure used in the BSDP model is the square of the deviation of the total power generated from the total firm power committed and the objective function is to minimize the expected value of the system performance measure. The model application is demonstrated through a case study of the Kalinadi Hydroelectric Project (KHEP) Stage I, in Karnataka state, India.

Development of stochastic dynamic Nash game model for reservoir operation. I. The symmetric stochastic model with perfect information

Increasing water demands, higher standards of living, depletion of resources of acceptable quality and excessive water pollution due to agricultural and industrial expansions have caused intense social and political predicaments, and conflicting issues among water consumers. The available techniques commonly used in reservoir optimization/operation do not consider interaction, behavior and preferences of water users, reservoir operator and their associated modeling procedures, within the stochastic modeling framework. In this paper, game theory is used to present the associated conflicts among different consumers due to limited water. Considering the game theory fundamentals, the Stochastic Dynamic Nash Game with perfect information (PSDNG) model is developed, which assumes that the decision maker has sufficient (perfect) information regarding the associated randomness of reservoir operation parameters. The simulated annealing approach (SA) is applied as a part of the proposed stochastic framework, which makes the PSDNG solution conceivable. As a case study, the proposed model is applied to the Zayandeh-Rud river basin in Iran with conflicting demands. The results are compared with alternative reservoir operation models, i.e., Bayesian stochastic dynamic programming (BSDP), sequential genetic algorithm (SGA) and classical dynamic programming regression (DPR). Results show that the proposed model has the ability to generate reservoir operating policies, considering interactions of water users, reservoir operator and their preferences.

UNCERTAINTY BASED OPERATION OF LARGE SCALE RESERVOIR SYSTEMS: DEZ AND KAROON EXPERIENCE1

ABSTRACT: Reservoir operation involves a complex set of human decisions depending upon hydrologic conditions in the supply network including watersheds, lakes, transfer tunnels, and rivers. Water releases from reservoirs are adjusted in an attempt to provide a balanced response to different demands. When a system involves more than one reservoir, computational burdens have been a major obstacle in incorporating uncertainties and variations in supply and demand. A new generation of stochastic dynamic programming was developed in the 1980s and 1990s to incorporate the forecast and demand uncertainties. The Bayesian Stochastic Dynamic Programming (BSDP) model and its extension, Demand Driven Stochastic Dynamic Programming (DDSP) model, are among those models. Recently, a Fuzzy Stochastic Dynamic Programming model (FSDP) also was developed for a single reservoir to model the errors associated with discretizing the variables using fuzzy set theory. In this study the DDSP and the FSDP models were extended and simplified for a complex system of Dez and Karoon reservoirs in the southwestern part of Iran. The simplified models are called Condensed Demand Driven Stochastic Programming (CDDSP) and Condensed Fuzzy Stochastic Dynamic Programming (CFSDP). The optimal operating policies developed by the CDDSP and the CFSDP models were simulated in a classical model and a fuzzy simulation model, respectively. The case study was used to demonstrate the advantages of implementing the proposed algorithm, and the results show the significant value of the proposed fuzzy based algorithm.

Value of Seasonal Flow Forecasts in Bayesian Stochastic Programming

This paper presents a Bayesian Stochastic Dynamic Programming (BSDP) model to investigate the value of seasonal flow forecasts in hydropower generation. The proposed BSDP framework generates monthly operating policies for the Skagit Hydropower System (SHS), which supplies energy to the Seattle metropolitan area. The objective function maximizes the total benefits resulting from energy produced by the SHS and its interchange with the Bonneville Power Administration. The BSDP-derived operating policies for the SHS are simulated using historical monthly inflows, as well as seasonal flow forecasts during 60 years from January 1929 through December 1988. Performance of the BSDP model is compared with alternative stochastic dynamic programming models. To illustrate the potential advantage of using the seasonal flow forecasts and other hydrologic information, the sensitivity of SHS operation is evaluated by varying (1) the reservoir capacity; (2) the energy demand; and (3) the energy price. The simulation results demonstrate that including the seasonal forecasts is beneficial to SHS operation.

Bayesian stochastic optimization of reservoir operation using uncertain forecasts

Operation of reservoir systems using stochastic dynamic programming (SDP) and Bayesian decision theory (BDT) is investigated in this study. The proposed model, called Bayesian stochastic dynamic programming (BSDP), which includes inflow, storage, and forecast as state variables, describes streamflows with a discrete lag 1 Markov process, and uses BDT to incorporate new information by updating the prior probabilities to posterior probabilities, is used to generate optimal reservoir operating rules. This continuous updating can significantly reduce the effects of natural and forecast uncertainties in the model. In order to test the value of the BSDP model for generating optimal operating rules, real‐time reservoir operation simulation models are constructed using 95 years of monthly historical inflows of the Gunpowder River to Loch Raven reservoir in Maryland. The rules generated by the BSDP model are applied in an operation simulation model and their performance is compared with an alternative stochastic dynamic programming (ASDP) model and a classical stochastic dynamic programming (SDP) model. BSDP differs from the other two models in the selection of state variables and the way the transition probabilities are formed and updated.