Safe multi-agent deep reinforcement learning for real-time decentralized control of inverter based renewable energy resources considering communication delay

Abstract

The increasing penetration of distributed renewable energy resources brings a great challenge for real-time voltage security of distribution grids. The paper proposes a safe multi-agent deep reinforcement learning (MADRL) algorithm for real-time control of inverter-based Volt-Var control (VVC) in distribution grids considering communication delay to minimize the network power loss, while maintaining the nodal voltages in a safe range. The multi-agent VVC is modeled as a constrained Markov game, which is solved by the MADRL algorithm. In the training stage, the safety projection is added to the combined policy to analytically solve an action correction formulation to promote more efficient and safe exploration. In the real-time decision-making stage, a state synchronization block is designed to impute the data under the latest timestamp as the input of the agents deployed in a distributed manner, to avoid instability caused by communication delay. The simulation results show that the proposed algorithm performs well in safe exploration, and also achieves better performance under communication delay.