An Improved Two-Stage Deep Reinforcement Learning Approach for Regulation Service Disaggregation in a Virtual Power Plant

Abstract

Managing numerous distributed energy resources (DERs) within the virtual power plant (VPP) is challenging due to inaccurate parameters and unknown dynamic characteristics. To address these obstacles, a two-stage deep reinforcement learning approach is proposed for the VPP to provide frequency regulation services and issue the disaggregation commands to DER aggregators in real-time operation. In the offline-stage, an offline simulator is formulated to learn the dynamic characteristics of DER aggregators, through which the soft actor-critic (SAC) algorithm is employed to train the control policy. In the online-stage, the trained control policy is updated continuously in the practical environment, which can ameliorate the performance of the start-up process with prior knowledge. Moreover, a novel sharpness-aware minimization based soft actor-critic (SAM-SAC) algorithm is proposed to improve the robustness and adaptability of the deep reinforcement learning approach. Simulation results illustrate that the proposed approach enables the VPP to manage the DER aggregators to track the regulation requests more accurately and economically than the state-of-the-art methods.