Do we need high-order mutation in fault-based Boolean-specification testing?

Abstract

Fault-based Boolean-specification testing is an essential weak mutation testing technique since the executing paths of the programs are usually dependent on those Boolean expressions in predicate statements. In mutation testing, the coupling effect hypothesis assumes that a test set that detects all the simple faults in the program can detect a high percentage of complex faults. Such a hypothesis suggests that we do not need to perform high-order mutation testing in practice. Many empirical results have shown the existence of the coupling effect. However, as fault-based Boolean-specification testing is mainly used in safety-critical scenarios, the question of whether high-order mutation testing is necessary for Boolean-specifications needs to be answered more strictly. To answer whether we need high-order mutation testing in fault-based Boolean-specification testing, we conduct an empirical study on general-form Boolean expressions. Experimental results show us that: (1) Generally, it is slightly easier to kill high-order mutants (HOMs) than first-order mutants (FOMs) since fault detection probabilities for complex faults (HOMs) are marginally higher than that for simple faults (FOMs) for almost all the cases. (2) However, for any fault types in Boolean-specifications, the coupling effect hypothesis is not guaranteed to hold with absolute certainty. (3) When HOMs involve the expression negation fault (ENF) or the operator reference fault (ORF), the percent of killed HOMs is lower. Such results suggest that, if the testing resource is sufficient, high-order mutation testing could be carried out on fault types ENF and ORF in fault-based Boolean-specification testing.