Investigating the Relationship Between Mutants and Real Faults with Respect to Mutated Code

Abstract

Mutation testing aims to evaluate the fault detection capability of a test suite. This evaluation substitutes faults with mutants by transforming program code to be defective. Evidences of the relationship between the detection rates of mutants and real faults have supported the use of mutants. It has also been known that the test suite size was a significant factor affecting the relationship. Our study revealed that the selection of the mutated code was another factor affecting the relationship. We generated mutants by transforming the code modified to fix defects, while the modified code was located at three granularity levels. The experiments conducted on the defects4j dataset demonstrated that the granularity level caused a significant difference in the relationship; the detection rate of mutants was more strongly correlated with and more indicative of the fault detection capability at a fine level than at a coarse level. Moreover, the influence of the test suite size was different at each granularity level. These findings implied a strong correlation between the detection rates of mutants and real faults, independently of test suite size, when the error-prone code was located precisely.