Model-Based Reinforcement Learning for Eco-Driving Control of Electric Vehicles

Heeyun Lee,Namwook Kim,Suk Won Cha

doi:10.1109/access.2020.3036719

Abstract

With the development of autonomous vehicles, research on energy-efficient eco-driving is becoming increasingly important. The optimal control problem of determining the speed profile of the vehicle for minimizing energy consumption is a challenging problem that necessitates the consideration of various aspects, such as the vehicle energy consumption, slope of the road, and driving environment, e.g., the traffic and other vehicles on the road. In this study, an approach using reinforcement learning was applied to the eco-driving problem for electric vehicles considering road slopes. A novel model-based reinforcement learning algorithm for eco-driving was developed, which separates the vehicle's energy consumption approximation model and driving environment model. Thus, the domain knowledge of vehicle dynamics and the powertrain system is utilized for the reinforcement learning process, while model-free characteristics are maintained by updating the approximation model using experience replay. The proposed algorithm was tested via a vehicle simulation and compared with a solution obtained using dynamic programming (DP), and as well as conventional cruise control driving with constant speed. The simulation results indicated that the speed profile optimized using model-based reinforcement learning had similar behavior to the global solution obtained via DP and energy saving performance compared with cruise control.

Highlights

R ECENTLY, diverse technologies for autonomous vehicles have been developing rapidly, which has led to advancements in autonomous driving
The contribution of the present study is as follows: We developed a new algorithm for the eco-driving control problem using the reinforcement learning approach, and through this, we confirmed that the reinforcement learning method can be well applied to the eco-driving problem
A reinforcement learning algorithm was developed for the eco-driving of an electric vehicles (EVs), and through this, we confirmed that the reinforcement learning method can be well applied to the eco-driving problem

Summary

INTRODUCTION

R ECENTLY, diverse technologies for autonomous vehicles have been developing rapidly, which has led to advancements in autonomous driving. A bi-level methodology was used for the predictive energy management of parallel HEVs, where the optimal velocity was calculated first in the outer loop using a Krylov subspace method, and in the inner loop, the optimal torque split and gear shift were determined using PMP based on the model predictive control (MPC) framework Applying these eco-driving strategies to realworld driving situations is not easy and has many limitations. In the automated car-following scenario, the pulse-and-gliding strategy was implemented based on the switching logic in a servo-loop controller to minimize the fuel consumption These approaches have limitations in that MPC and periodic control are focused on finding the local optimal for the near future, rather than the global optimal solution with entire travel distances. General speed profile optimization for an eco-driving strategy for longitudinal driving considering the road slope using model-based reinforcement learning (MBRL) was investigated.

VEHICLE MODELING

OPTIMAL CONTROL PROBLEM FORMULATION

MODEL-BASED REINFORCEMENT LEARNING

COMPARISON WITH DETERMINISTIC DP RESULT AND CRUISE CONTROL RESULT

PERFORMANCE FOR LEARNING WITH COMBINED DRIVING CYCLES

CONCLUSION