Receding-Horizon Reinforcement Learning Approach for Kinodynamic Motion Planning of Autonomous Vehicles

Abstract

Kinodynamic motion planning is critical for autonomous vehicles with high maneuverability in dynamic environments. However, obtaining near-optimal motion planning solutions with low computational costs and inaccurate prior model information is challenging. To address this issue, this paper proposes a receding-horizon reinforcement learning approach for kinodynamic motion planning (RHRL-KDP) of autonomous vehicles in the presence of inaccurate dynamics information and moving obstacles. Specifically, a receding-horizon actor-critic reinforcement learning algorithm is presented, resulting in a neural network-based planning strategy that can be learned both offline and online. A neural network-based model is built and learned online to approximate the modeling uncertainty of the prior nominal model in order to improve planning performance. Furthermore, active collision avoidance in dynamic environments is realized by constructing safety-related terms in actor and critic networks using potential fields. In theory, the uniformly ultimate boundedness property of the modeling uncertainty’s approximation error is proven, and the convergence of the proposed RHRL-KDP is analyzed. Simulation tests show that our approach outperforms the previously developed motion planners based on model predictive control (MPC), safe RL, and RRT <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"><tex-math notation="LaTeX">${^\star }$</tex-math></inline-formula> in terms of planning performance. Furthermore, in both online and offline learning scenarios, RHRL-KDP outperforms MPC and RRT <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"><tex-math notation="LaTeX">$^{\star }$</tex-math></inline-formula> in terms of computational efficiency.