Search-based test case generation for Cyber-Physical Systems

Abstract

The test case generation of Cyber-Physical Systems (CPSs) face critical challenges that traditional methods such as Model-Based Testing cannot deal with. As a result, simulation-based testing is one of the most commonly used techniques for testing CPSs despite sometimes being computationally too expensive. This paper proposes a search-based approach which is implemented on top of Non-dominated Sorting Genetic Algorithm II (NSGA-II), the most commonly applied multi-objective search algorithm for cost-effectively generating executable test cases in order to test CPSs. With the aim of guiding the generation of the optimal set of so-called reactive test cases, the approach formally defines three cost-effectiveness measures: requirements coverage, test case similarity and test execution time. Furthermore, we design one crossover operator and three mutation operators (i.e., mutation at test suite level named Mu TS, mutation at test case level named Mu TC and mutation at both levels named Mu BO) for test case generation. We evaluate our approach by comparing with Random Search (RS) using four case studies (one of them is an industrial system). Moreover, we evaluate the three mutation operators using the four case studies. The results of the experiment (with a rigorous statistical analysis) indicated that our approach in conjunction with the crossover operator operation and three mutation operators significantly outperformed RS. In general, Mu BO achieved the best performance among the three mutation operators and managed to improve on average the test execution time by 14%, the requirements coverage by 34%, and the test similarity by 75% as compared with RS.