Parallel-Reinforcement-Learning-Based Online Energy Management Strategy for Energy Storage Traction Substations in Electrified Railroad

Abstract

The traction power supply system is the only source of power for electric locomotives. The huge power fluctuations and complex operating conditions of the traction power supply system pose a challenge to the efficient operation of energy storage traction substations. The existing energy management strategies are difficult to achieve accurate charging and discharging, difficult to modify the control rules in real-time, and have poor migration capability. For comparison, the reinforcement learning algorithms can address the shortcomings of rule-based energy management strategies due to their model-free feature. Therefore, this paper proposes an energy management strategy based on parallel reinforcement learning to improve the efficiency of energy utilization while speeding up the convergence of the algorithm. More specifically, a Markov decision framework is established for capturing the energy management process. The Monte Carlo sampling process is also improved to achieve offline optimization by parallel reinforcement learning algorithms and reduce the impact of low-value power fragments on iteration speed. Meanwhile, the algorithm is modified to enable online updates. The case study shows that compared with other energy management strategies, the parallel reinforcement learning-based energy management strategy has faster convergence speed, higher energy exchange efficiency, better migration capability, and can adapt to various complex working conditions.