A soft actor-critic reinforcement learning framework for optimal energy management in electric vehicles with hybrid storage

Abstract

The efficient energy management of electric vehicles (EVs) equipped with hybrid energy storage systems (HESS) poses a significant challenge due to its vast search space, numerous control variables, and intricate driving conditions. In response to this challenge, this paper introduces a novel deep reinforcement learning (DRL) algorithm, specifically the soft actor-critic (SAC), tailored for optimizing energy distribution within EVs equipped with battery-supercapacitor (SC) HESS. The proposed SAC-based energy management system (EMS) is designed to address inherent limitations present in most existing DRL algorithms, such as slower convergence rates, discretization errors, unstable training dynamics, and suboptimal optimization effects. The SAC-based EMS undergoes training in a continuous action space through self-play, utilizing a complex driving cycle and a novel reward function. The algorithm demonstrates its efficiency by rapidly maximizing cumulative rewards and refining its decision-making policies. Afterward, extensive experiments showcase the superiority of the proposed SAC-based EMS over rule-based (RB) techniques, deep deterministic policy gradient (DDPG), and battery-only configurations across various driving cycles. Notably, the approach effectively allocates high-power surges and braking energy to the SC, reducing the frequency of charge/discharge cycles and thereby extending the battery's lifespan. Additionally, the learned SAC-based EMS policy achieves substantial reductions in electricity consumption by 39.6 %, 47.87 %, and 45 % compared to battery-only configuration, and 42.56 %, 47.87 %, and 45.83 % compared to RB technique under the NYCC, LA92, and FTP cycles, respectively. In contrast, the DDPG algorithm tends to rely heavily on the SC to deliver the total power required, even under normal driving conditions.