Analyzing the effectiveness of One-Op Mutation against the minimal set of mutants

Abstract

Mutation testing is a fault-based criterion that helps testers to evaluate the adequacy of a test set applied to a given program. In order to do that, mutation operators are applied to the software being tested, creating versions with few syntactic deviations from the original program, called mutants. The choice of mutation operator plays an important role when applying mutation testing regarding the test case design and cost analysis. However, some generated mutants are considered as redundant and may be considered unnecessary during mutation testing. This paper addresses this topic and investigates the effectiveness of test suites designed to kill mutants of a specific operator to kill mutants that belong to the minimal set of mutants. We performed an experimental evaluation using a set of 39 programs written in C language. Guided by previous studies, we evaluated the effectiveness of deletion operators, and due to the results, we extended our investigation to all mutant operators. Concerning deletion operators, the results showed that even combining deletion operators, the best mutation score achieved was, on average, 0.460 using the SODL (SSDL + OODL) operator. The following analyses showed that the VDTR operator performed better than the deletion operators both in terms of mutation score and in terms of cost, proving to be the best approach to design test cases that have a higher impact on killing mutants that belong to the minimal set of mutants.