Deep Reinforcement Learning for Cascaded Hydropower Reservoirs Considering Inflow Forecasts

Abstract

This paper develops a deep reinforcement learning (DRL) framework for intelligence operation of cascaded hydropower reservoirs considering inflow forecasts, in which two key problems of large discrete action spaces and uncertainty of inflow forecasts are addressed. In this study, a DRL framework is first developed based on a newly defined knowledge sample form and a deep Q-network (DQN). Then, an aggregation-disaggregation model is used to reduce the multi-dimension spaces of state and action for cascaded reservoirs. Following, three DRL models are developed respectively to evaluate the performance of the newly defined decision value functions and modified decision action selection approach. In this paper, the DRL methodologies are tested on China’s Hun River cascade hydropower reservoirs system. The results show that the aggregation-disaggregation model can effectively reduce the dimensions of state and action, which also makes the model structure simpler and has higher learning efficiency. The Bayesian theory in the decision action selection approach is useful to address the uncertainty of inflow forecasts, which can improve the performance to reduce spillages during the wet season. The proposed DRL models outperform the comparison models (i.e., stochastic dynamic programming) in terms of annual hydropower generation and system reliability. This study suggests that the DRL has the potential to be implemented in practice to derive optimal operation strategies.