Autonomous Vehicle Evaluation: A Comprehensive Survey on Modeling and Simulation Approaches

Abstract

In recent years, autonomous vehicles (AVs), which observe the driving environment and lead a few or all of the driving tasks, have garnered tremendous success. The field of AVs has been rapidly developing and has found many applications. As a safety requirement established by policymakers, these vehicles must be evaluated before their deployment. The evaluation process for AVs is challenging because crashes are rare events, and AVs can escape passing predefined test scenarios. Therefore, capturing crashes and creating real test scenarios should be considered in order to develop an evaluation approach that represents real-world scenarios. One evaluation approach is based on the naturalistic field operational test (N-FOT), in which prototype AVs are driven on roads by volunteers or test engineers. Unfortunately, this approach is time-consuming and costly because thousands of miles need to be driven to experience a police-reported collision and nearly millions of miles for a fatal crash. Another approach is the accelerated evaluation method. The core idea of the accelerated evaluation approach is to modify the statistics of naturalistic driving so that safety-critical events are emphasized. This paper presents a brief survey of the advances that have occurred in the area of the evaluation of partially or fully autonomous vehicles, starting with naturalistic field operational tests (N-FOTs). The review covers the test matrix evaluation, worst-case scenario evaluation (WCSE), Monte Carlo simulations, and accelerated evaluation (AE). We also present all the simulation-based and agent-based modeling approaches that do not follow any evaluation protocol listed above. This study provides a scientific analysis of each evaluation techniques, focusing on their advantages/disadvantages, inherent restrictions, practicability, and optimality. The results reveal that the accelerated evaluation approach outperforms naturalistic field operational tests (N-FOTs), test matrix evaluation, worst-case scenario evaluation (WCSE), and Monte Carlo simulation methods in some of the car-following and lane-change studies when using specific models. Moreover, the agent-based model and augmented and virtual reality approaches show promising results in AV evaluation. Furthermore, integrating machine and deep learning into the available AV evaluation methods can improve their performance and generate encouraging outcomes.