Offline Reinforcement Learning for Autonomous Driving with Real World Driving Data

Abstract

Since traditional reinforcement learning (RL) approaches need active online interaction with the environment, previous works are mainly investigated in the simulation environment rather than the real world environment, especially for safety-critical applications. Offline RL has recently emerged as a promising data-driven learning paradigm to learn a policy from offline dataset directly. It seems that offline RL is well suited for autonomous driving, as it is feasible to collect offline naturalized driving dataset. However, it remains unclear how to deploy offline RL with real world driving dataset only including observation data, and whether current offline RL algorithms work well to learn a driving policy than imitation learning? In this paper, we provide an offline RL benchmark for autonomous driving including the dataset, baselines, and a data driven simulator <sup xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">1</sup> <sup xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">1</sup> Code: https://github.com/weiaiF/offiineRL-INTERACTION. First, we summarize and introduce the popular offline RL baseline methods. Then, we construct an offline RL dataset for the car following task based on the real world driving dataset INTERACTION. A data driven simulator is applied to obtain augmented data and test the driving policy. Further, we deploy four popular offline algorithms and analyze their performances under different datasets including real world driving data and augmented data. Finally, related conclusions and discussions are given to analyze the critical challenge for offline RL in autonomous driving.