Improved reinforcement learning-based real-time energy scheduling for prosumer with elastic loads in smart grid

Abstract

In this study, we investigate the dynamic energy-scheduling problem of a prosumer (producer/consumer) with an energy storage device (EDS) and elastic loads. The goal is to develop efficient real-time scheduling strategies for prosumers, and to minimise their total long-term costs (i.e. cost of energy purchased from the external grid and depreciation cost of the EDS). The challenges are twofold: the uncertainty of the energy output of the prosumer and the time-coupling constraints of the EDS and elastic loads. To address these challenges, we first describe the dynamic energy-scheduling problem as a Markov decision process. Then, an approximate state dual-agent Q-learning algorithm is proposed to solve the optimal dynamic scheduling problem by improving the model-free reinforcement learning(RL) method. Compared with the traditional RL method, the proposed algorithm reduces the system-state dimensions and exhibits improved performance. The proposed algorithm can only be assisted by mutual interactions between the environment with a reward feedback mechanism to dynamically respond to uncertain changes in the environments, without modelling or predicting the system environments. Finally, extensive empirical evaluations using real-world traces are conducted to study the effectiveness of the proposed algorithm. The results show that the proposed algorithm can reduce the total cost of the prosumer by up to 6.3%, 11.7% and 22.4% compared with the traditional RL method, Lyapunov optimisation and greedy algorithm, respectively.