Learning to Operate Distribution Networks With Safe Deep Reinforcement Learning

Abstract

In this paper, we propose a safe deep reinforcement learning (SDRL) based method to solve the problem of optimal operation of distribution networks (OODN). We formulate OODN as a constrained Markov decision process (CMDP). The objective is to achieve adaptive voltage regulation and energy cost minimization considering the uncertainty of renewable resources (RSs), nodal loads and energy prices. The control actions include the number of in-operation units of the switchable capacitor banks (SCBs), the tap position of the on-load tap-changers (OLTCs) and voltage regulators (VRs), the active and reactive power of distributed generators (DGs), and the charging and discharging power of battery storage systems (BSSs). To optimize the discrete and continuous actions simultaneously, a stochastic policy built upon a joint distribution of mixed random variables is designed and learned through a neural network approximator. To guarantee that safety constraints are satisfied, constrained policy optimization (CPO) is employed to train the neural network. The proposed approach enables the agent to learn a cost-effective operating strategy through exploring safe scheduling actions. Compared to traditional deep reinforcement learning (DRL) methods that allow agents to freely explore any behaviors during training, the proposed approach is more practical to be applied in a real system. Simulation results on a modified IEEE-34 node system and a modified IEEE-123 node system demonstrate the effectiveness of the proposed method.