Novel data-driven energy management of a hybrid photovoltaic-reverse osmosis desalination system using deep reinforcement learning

Abstract

This paper proposes a novel deep reinforcement learning-accelerated energy management system for a hybrid grid-connected photovoltaic-reverse osmosis-pressure retarded osmosis desalination plant. The energy management problem is formulated as a partially observable Markov decision process by using historical photovoltaic (PV) power data in order to cope with uncertainties related to the generation of solar power and provide more information regarding the true state of the system. The soft actor-critic (SAC) algorithm is employed as the core of the energy management system to maximize water production rate and contaminant removal efficiency while minimizing the supplied power from the external grid. We introduce 1-dimensional convolutional neural networks (1-D CNNs) to the actor, critic, and value function networks of the SAC algorithm to address the partial observability dilemma involved in PV-powered energy systems, extract essential features from the PV power time series, and achieve immensely improved performance ultimately. Furthermore, it is assumed that the proposed CNN-SAC algorithm does not have access to the current output power data of the PV system. The development of more practical energy management systems necessitates this assumption, and we demonstrate that the proposed method is capable of forecasting the current PV power data. The superiority of the CNN-SAC model is verified by comparing its learning performance and simulation results with those of four state-of-the-art deep reinforcement learning algorithms: Deep deterministic policy gradient (DDPG), proximal policy optimization (PPO), twin delayed DDPG (TD3), and vanilla SAC. The results show that the CNN-SAC model outperforms the benchmark methods in terms of effective solar energy exploitation and power scheduling, manifesting the necessity of exploiting historical PV power data and 1-D CNNs. Moreover, the CNN-SAC algorithm is benchmarked against a powerful energy management system we developed in our previous investigation by studying three scenarios, and it is demonstrated that considerable improvement in energy efficiency can be obtained without using any solar power generation forecasting algorithm. By conducting ablation studies, the critical contribution of the introduced 1-D CNN is demonstrated, and we highlight the significance of providing historical PV power data for substantial performance enhancement. The average and standard deviation of evaluation scores obtained during the last stages of training reveal that the 1-D CNN significantly improves the final performance and stability of the SAC algorithm. These results demonstrate that the modifications we detail in our investigation render deep reinforcement learning algorithms extremely powerful for the energy management of PV-powered microgrids, including PV-driven reverse osmosis desalination plants.