Test Coverage Index for ADAS/ADS Assessment Based on Various Real-World Information Points

Abstract

In this article, we propose a novel index that expresses the comprehensiveness of the real scenes using various information points (e.g., map information, road structure rules, and traffic investigation) to correctly assess an advanced driving assistance system (ADAS)/autonomous driving system (ADS). There are two keypoints. One is to establish an index that enables us to judge whether a test has been sufficiently performed and locate ``dropouts'' in the ADAS/ADS assessment by constructing a database of the scene structure in the real world, instead of the current index which is the running distance of the field operational test (FOT) based on knowledges/experiences. The other is that designed evaluation scenarios with the proposed index enable to guarantee that the scenes targeted by the ADAS/ADS are mostly covered and to grasp the priority of the target scenes without the bias of the appearance frequency. Specifically, we defined the real world as a combination of five types of scene features. Then, we formulated the test coverage index by integrating the existence and appearance frequency of real-world information corresponding to each scene feature. Furthermore, we searched optimal evaluation courses by maximizing the score on each road segment based on the index. In experiments, we showed the results of visualizing the test coverage ratios which enable to compare both optimal designed course and manually designed course that assumes the current process. We also showed the test coverage ratios in several countries and in several scene feature patterns toward the various quantitative ADAS/ADS assessment.