Scalable evaluation methods for autonomous vehicles

Abstract

Effective intelligent driving test and evaluation methods can improve the development and deployment process of autonomous vehicles (AVs). However, due to the extreme complexity and high dimensionality of driving behavior, how to objectively and efficiently evaluate the multi-dimensional performance of AVs in simulation and real-world environments is a long-standing problem. This paper proposes an objective multi-dimensional comprehensive evaluation (OMDCE) method that divides the intelligent driving test into four modules: test scenarios, scenario complexity model, simulation test platform, and automated evaluation system. The scenario complexity model is proposed to bridge the gap between the test scenarios and the automated evaluation system, thus enabling adaptive scaling of the evaluation scale for test scenarios with different difficulty levels. Besides, based on the existing four evaluation metrics for ego performance, altruism performance evaluation is first proposed to comprehensively portray AVs’ intelligence degree. The OMDCE method was evaluated by testing two different intelligent driving algorithms and conducting real vehicle testing. The experimental results demonstrated that the OMDCE method can effectively evaluate AVs in various scenarios and quantitatively measure their multi-dimensional performance. The proposed method reduces the subjectivity of manual evaluation and speeds up the evaluation. Moreover, it improves the universality and scalability of the evaluation metrics system.