Test Suite Minimization with Mutation Testing-Based Many-Objective Evolutionary Optimization

Abstract

Context: As software evolves, the test suite tends to grow, regression testing has become prohibitively expensive. Test suite minimization is one of the most important approaches for reducing test cost. The process of test suite minimization is a trade-off between cost and other value criteria and is appropriate to be described as a many-objective optimization problem. Objective: To identify the most efficient test suite for reducing the redundant degree of test data and improving test efficiency without decreasing the defect detection ability of test data. Method: We introduce a mutation testing-based many-objective optimization approach, which gives higher priority to the fault detection ability and takes mutation score as a major objective, together with cost and three standard code coverage criteria for test suite minimization. Six classical evolutionary many-objective optimization algorithms are applied to identify efficient test suite. Three programs from the SIR repository and one larger program, space are applied for empirical study and effectiveness evaluation. Results: On the one hand, in three SIR programs experiments NSGA-II with tuning was the most effective technique. However, MOEA/D-PBI outperformed NSGA-II on the larger program (Space). On the other hand, the test cost of the optimal test suite which obtained by the many-objective optimization approach with mutation score is much lower than the one without it in tcas. Conclusions: The experimental results prove that the many-objective optimization model with the guidance of mutation score is indeed effective in reducing the test suite redundancy.