Less is More: Simplification of Test Scenarios for Autonomous Driving System Testing

Abstract

Simulation-based testing is a popular approach for testing autonomous driving systems (ADS), in which different types of scenario are designed to test the ADS under different driving conditions. Given a specific test goal, a generation approach (e.g., search-based testing) is usually employed to find a scenario covering such goal; for example, it can find a scenario in which the autonomous vehicle collides. The generated scenarios may contain some elements that are irrelevant for the achievement of the test goal; if this is the case for a scenario that exposes a failure, for ADS engineers it is difficult to identify the root cause, as the ADS interacts with several traffic participants and it is not clear which of these are essential to trigger the failure. This problem emerged during the collaboration with our industry partner, for which, in the past, we proposed different test generation approaches for their ADS path planner, but these may produce test scenarios that are not minimal. To tackle this problem, in this paper, we propose an approach that, given an ADS test scenario, simplifies it by removing all the traffic participants that are not needed. As output, the approach provides a scenario that still covers the test goal as the initial scenario, but contains the minimum number of traffic participants. The approach consists in iteratively generating simplified scenarios by removing some traffic participants and determining, by observing the test execution, which can be actually removed and which must be kept in the scenario. Three policies are investigated to remove traffic participants whose classification is not know: single policy, binary policy, and adaptive policy. Experiments have been conducted on several scenarios generated for the path planner. Results show that the binary policy is the one that usually can find the minimal scenario with the minimum number of simplification attempts, but, in particular cases, the adaptive policy is better.