Multi-agent deep reinforcement learning-based multi-time scale energy management of urban rail traction networks with distributed photovoltaic–regenerative braking hybrid energy storage systems

Abstract

The integration of photovoltaics (PVs), regenerative braking (RB) techniques, and energy storage devices has become crucial to promote energy conservation and emission reduction for a sustainable future of urban rail traction networks (URTNs). This paper proposes a tri-level multi-time scale energy management framework for the economic and low-carbon operation of URTNs with PV–RB hybrid energy storage systems (HESSs) based on multi-agent deep reinforcement learning (MADRL). A two-stage stochastic scheduling approach is developed to minimize daily operation and carbon trading costs at the upper level and correct day-ahead scheduling deviations against multi-source uncertainties at the middle level. A MADRL-based real-time energy management strategy is established to optimize the PV–RB power flow and promote its utilization by coordinating distributed HESSs at the lower level. The HESS control problem is formulated as a decentralized partially observable Markov decision process and solved by a multi-agent control algorithm based on monotonic value function factorization. A Copula-based spatio-temporal dependency model is devised to characterize the PV, passenger flow, and traction load uncertainties and generate daily URTN operation scenarios for enhancing day-ahead and intraday decisions. Comparative studies demonstrate the effectiveness of the proposed framework in terms of a cost reduction by 11.98% and a PV–RB energy utilization improvement by 13.94%.