Empirical Evaluation of Multiobjective Optimization Algorithms Searching for Higher Order Mutants

Abstract

ABSTRACTFirst order mutation testing is used to evaluate the quality of a given set of test cases by inserting single changes into the program under test to produce first order mutants (FOMs) of the original program, and then checking whether tests are good enough to detect the artificially injected defects. However, mutation testing is not yet widely used due to the problems of a large number of generated mutants and limited realism of introduced changes that do not necessarily reflect real software defects. Furthermore, many of the generated mutants are equivalent, i.e., they keep the program semantics unchanged and, thus, cannot be detected by any test suite. Higher order mutation testing has been coined as a promising solution for overcoming these limitations of FOM testing. In particular, finding strongly subsuming higher order mutants (SSHOMs), which are able to replace all of their constituent FOMs without scarifying test effectiveness while being able to reflect complex, real defects that require more than one change to correct them, is considered an important research challenge and is the focus of this work. The contribution of this article is a new, extended classification of higher order mutants (HOMs) to cover all cases of generated HOMs. Fitness functions and empirical comparison of four different multiobjective optimization algorithms are used to generate and evaluate HOMs as well as search for valuable high-quality and reasonable HOMs (strongly subsuming and coupled HOMs) and ten other types of HOMs. The main goal of this study is to assert the effect of applying multiobjective optimization algorithms in the area of higher order-mutation testing, while asserting the correctness of the proposed HOMs classification, objectives, and fitness functions. Our experimental results show that the total number of generated HOMs is smaller (about 70%) in comparison to FOMs, while the mean ratio of reasonable HOMs (subsuming HOMs) to all found HOMs is over 56%, and the mean ratio of high-quality and reasonable HOMs (strongly subsuming and coupled HOMs) to all found reasonable HOMs (subsuming HOMs) is fairly high (around 8.74%).