Energy management strategies based on soft actor critic reinforcement learning with a proper reward function design based on battery state of charge constraints

Abstract

The environmental concerns have pushed our society to find solutions to reduce the emission of transportation systems. One promising solution is the fuel cell hybrid electric vehicle (FCHEV), with a proton exchange membrane fuel cell (PEMFC) and a lithium-ion battery. The presence of two different energy sources requires a smart energy management strategy (EMS) which must not only optimize energy consumption (conventional approaches), but also mitigate component’s degradation (New approaches : Health-conscious strategies). For that, a wide range of techniques has been proposed in the literature, and the most recent techniques are based on machine learning. Reinforcement learning (RL) is usually the most chosen strategy, but as in the optimization techniques, a strong design of the objective function (called reward function in RL) is required. The main challenge in the reward function is to encourage a proper battery state of charge (SOC) management, while reducing the energy consumption and avoiding the factors that accelerate the FC aging. The current advancement in the literature shows satisfactory results, but the reward function design have great improvement potential in the chosen approach. Indeed, most approaches in RL for SOC management with continuous actions are based on the reference SOC principle, which reduces the EMS ability to optimize the other part of the reward function. This paper reveals the challenges associated with the introduction of SOC limits in the reward function and proposes an approach, based on SOC boundaries in the reward function. The contribution allows to better consider the parts to optimize, as it gives more freedom than previous reference SOC techniques all reducing the consumption by 12.9%.