Two-stage data-driven optimal energy management and dynamic real-time operation in networked microgrid based on a deep reinforcement learning approach

Abstract

Given the significant challenges posed by the vast and diverse data in energy management, this study introduces a two-stage approach: optimal energy management system (OEMS) and dynamic real-time operation (DRTOP). These stages employ a multi-agent policy-oriented deep reinforcement learning (DRL) approach, aiming to minimize operating and energy exchange costs through interactions in the networked microgrid (NMG) energy market. The primary objectives include minimizing the distribution system operator (DSO) cost and optimizing the exchanged power between DSO and NMG, and the power transmission losses and the secondary include minimizing MG’s operating cost, optimal use of renewable energy resources (RER) and energy storage systems (ESS), minimizing the exchanged power cost with the main grid and, risk analysis. The OEMS&DRTOP model is developed based on the Stackelberg game theory and the DRL structure. The DRL model is developed in two offline learning and online distributed operation phases to minimize the computational burden, time, and DRL operation process. This study’s results show the high efficiency of the presented approach to minimizing the operating cost, the exchanged power based on the price uncertainty, power transmission losses, and, RER and ESSs optimal participation. In addition, regarding computational load, the proposed concept demonstrates a 12.9% reduction compared to the dueling deep Q-network method and a 17% reduction compared to the deep Q-network method. Also regarding computational time, the proposed concept demonstrates a 17.13% reduction compared to the dueling deep Q-network method and a 25.6% reduction compared to the deep Q-network method.