Integrated energy hub dispatch with a multi-mode CAES–BESS hybrid system: An option-based hierarchical reinforcement learning approach

Abstract

The high penetration of renewable energy sources (RES) in power generation has driven demand for advanced integrated energy management systems (IEMS). In this study, to address the challenges of insufficient adaptability to dynamic supply–demand, a multi-type energy IEMS combining compressed air energy storage (CAES) and a battery energy storage system (BESS) is proposed, which operates under a multi-mode energy storage (MES) mechanism with rapid response, long-term balance, and synergic adjustment modes. To address the complexity of sequential decisions, an option-critic based twin delayed deep deterministic policy gradient (OCTD3) algorithm is firstly proposed within the hierarchical reinforcement learning (HRL) framework, enhancing efficiency through encapsulation of subtasks within ”options”. Additionally, model precision is refined by fitting the electricity–gas–heat conversion dynamics of CAES under off-design conditions. Dispatch tasks are modeled as an option-based Semi-Markov Decision Process (SMDP) and optimized by the OCTD3 to improve the power fluctuations index (PFI), comprehensive costs index (CCI), and system response synergy index (SRSI). Comparative simulations reveal that the MES mechanism boosts SRSI by 91.8%, showcasing high adaptability to varied supply–demand scenarios. The OCTD3 algorithm develops five hybrid strategies for CAES–BESS across three modes, effectively cutting costs by reducing electricity purchases and fluctuations expenses, and lowering PFI by 42.2% through balancing peak–valley loads and swiftly responding to transient shifts.