Sampling-based test scenarios searching method for autonomous system robustness evaluation

Abstract

The robustness evaluation for autonomous systems is essential before implementing them in safety-critical fields. Searching the challenging test scenarios by adaptive sampling algorithm in the transition regions between different performance modes, named performance boundary, becomes an efficient method to evaluate the robustness of systems. This paper proposes a novel model-independent nearest neighbors adaptive sampling algorithm considering gradient and distance, named GDNNAS. GDNNAS designs a special evaluation criterion to identify samples within boundary regions and a searching method to generate new samples. A novel elimination criterion is proposed to ensure the quality of samples and an adaptive stopping criterion is proposed to stop the sampling process adaptively. Considering the lack of unified data sets and metrics to assess adaptive sampling methods for searching the performance boundary, we proposed a comparison platform consisting of nine simulated systems under test (SSUT) as adaptive sampling benchmarks and four evaluation metrics to evaluate the adaptive sampling method quantitatively. We assess GDNNAS and two other adaptive sampling algorithms with the nine SSUTs. The experimental result proves that samples generated by GDNNAS are precisely located at and cover the performance boundary regions. In addition, we analyze the effectiveness of the comparison platform. Experimental result shows that the proposed SSUTs and evaluation metrics can comprehensively reflect the performance of the algorithm from various aspects and reveal the characteristics of the adaptive sampling algorithms. To verify the practicability of GDNNAS, a path planning system is used for testing GDNNAS, and the experiment result proves that GDNNAS performs outstandingly in practical system robustness evaluations.