Safe reinforcement learning based optimal low-carbon scheduling strategy for multi-energy system

Abstract

Multi-energy system with distributed energy resources has become the inevitable trend in recent years due to their potential for creating the efficient and sustainable energy infrastructure, with a strong ability on carbon emission reduction. To accommodate the uncertainties of renewable energy generation and energy demand, model-free deep reinforcement learning methods are emerging for energy management in multi-energy system. However, traditional reinforcement learning methods still have operation safety issue of violating the physical constraints of multi-energy system. To address the challenges, a low-carbon scheduling strategy based on safe soft actor-critic algorithm is proposed in this paper. Firstly, an electricity-thermal-carbon joint scheduling framework is constructed, where carbon trading mechanism is incorporated to further motivate carbon emission reductions. Secondly, the energy cost and carbon trading cost are simultaneously integrated in the objective function, and the dynamic optimization problem of multi-energy system is modeled as a constrained Markov decision process by taking into account the diverse uncertainties. Then, a novel safe soft actor-critic method is proposed to achieve the benefits of economic and carbon emissions, where the security networks and Lagrangian relaxation are introduced to deal with operation constraints. The case study validates that the proposed scheduling strategy can reduce the energy cost and carbon trading cost by up to 26.24% and 33.73% within constraints, compared with existing methods.