Pareto efficient multi-objective black-box test case selection for simulation-based testing

Abstract

Abstract Context: In many domains, engineers build simulation models (e.g., Simulink) before developing code to simulate the behavior of complex systems (e.g., Cyber-Physical Systems). Those models are commonly heavy to simulate which makes it difficult to execute the entire test suite. Furthermore, it is often difficult to measure white-box coverage of test cases when employing such models. In addition, the historical data related to failures might not be available. Objective: The objective of the approach presented in this paper is to cost-effectively select test cases without making use of white-box coverage information or historical data related to fault detection. Method: We propose a cost-effective approach for test case selection that relies on black-box data related to inputs and outputs of the system. The approach defines in total six effectiveness measures and one cost measure followed by deriving in total 21 objective combinations and integrating them within Non-Dominated Sorting Genetic Algorithm-II (NSGA-II). The proposed six effectiveness metrics are specific to simulation models and are based on anti-patterns and similarity measures. Results: We empirically evaluated our approach with these 21 combinations using six case studies by employing mutation testing to assess the fault revealing capability. We compared our approach with Random Search (RS), two many-objective algorithm, as well as three white-box metrics. The results demonstrated that our approach managed to improve Random Search by up to around 28% in terms of the Hypervolume quality indicator. Similarly, black-box metrics-based test case selection also significantly outperformed those of white-box metrics. Conclusion: We demonstrate that test case selection is a non-trivial problem in the context of simulation models. We also show that the proposed effectiveness metrics performed significantly better than traditional white-box metrics. Thus, we show that black-box test selection approaches are appropriate to solve the test case selection problem within simulation models.