A Deep Reinforcement Learning Based Hierarchical Eco-Driving Strategy for Connected and Automated HEVs

Abstract

The advancement in vehicle connectivity and autonomy has fostered the development of eco-driving technology, aimed at optimizing driving behaviors to reduce vehicle energy consumption. This study proposed a real-time deep reinforcement learning based hierarchical eco-driving control strategy to optimally control a connected and automated hybrid electric vehicle in a traffic scenario with multiple constraints. The speed optimization layer of the proposed strategy employs a twin-delayed deep deterministic policy gradient (TD3) based reference speed planning strategy to compute the optimized speed, based on a pre-trained optimal policy and observed environmental states. Specifically, to learn the optimal policy, a multi-objective reward function is designed that integrates fuel consumption reward and shaping reward involving car-following and road speed limit. Additionally, a rule-based competition-decision model is embedded within the speed optimization layer to ensure compliance with traffic light rules. In the vehicle controller layer, a real-time controller is implemented to specify appropriate actuator variables for the hybrid powertrain to track the reference speed and conduct energy management control. Simulation results show that the proposed TD3 based eco-driving strategy achieves remarkable energy saving performance by optimizing the speed. Besides, the proposed eco-driving strategy is capable of satisfying the constraint of diverse traffic scenarios, including car-following and traffic light, while also being computationally lightweight.