An empirical evaluation of mutation and crossover operators for multi-objective uncertainty-wise test minimization

Abstract

Multi-objective uncertainty-wise test case minimization focuses on selecting a minimum number of test cases to execute out of all available ones while maximizing effectiveness (e.g., coverage), minimizing cost (e.g., time to execute test cases), and at the same time optimizing uncertainty-related objectives. In our previous unpublished work1, we developed four uncertainty-wise test case minimization strategies relying on Uncertainty Theory and multi-objective search (NSGA-II with default settings), which were evaluated with one real Cyber-Physical System (CPS) with inherent uncertainty. However, a fundamental question to answer is whether these default settings of NSGA-II are good enough to provide optimized solutions. In this direction, we report one of the preliminary empirical evaluations, where we performed an experiment with three different mutation operators and three crossover operators, i.e., in total nine combinations with NSGA-II for the four uncertainty-wise test case minimization strategies using a real CPS case study. Results show that the Blend Alpha crossover operator together with the polynomial mutation operator permits NSGA-II achieving the best performance for solving our uncertainty-wise test minimization problems.