A Risk-Averse Preview-Based Q-Learning Algorithm: Application to Highway Driving of Autonomous Vehicles

Abstract

A risk-averse preview-based <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"> <tex-math notation="LaTeX">$Q$</tex-math> </inline-formula> -learning planner is presented for navigation of autonomous vehicles (AVs). To this end, the multilane road ahead of a vehicle is represented by a finite-state nonstationary Markov decision process (MDP). A risk assessment unit module is then presented, which leverages the preview information provided by sensors along with a stochastic reachability module to assign reward values to the MDP states and update them as scenarios develop. A sampling-based risk-averse preview-based <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"> <tex-math notation="LaTeX">$Q$</tex-math> </inline-formula> -learning algorithm is finally developed, which generates samples using the preview information and reward function to learn risk-averse optimal planning strategies without actual interaction with the environment. The risk factor is imposed on the objective function to avoid fluctuation of the <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"> <tex-math notation="LaTeX">$Q$</tex-math> </inline-formula> values, which can jeopardize the vehicle’s safety and/or performance. The overall hybrid automaton model of the system is leveraged to develop a feasibility check unit module that detects unfeasible plans and enables the planner system to react proactively to the changes of the environment. Finally, to verify the efficiency of the presented algorithm, its implementation on two highway driving scenarios of an AV in a varying traffic density is considered.