The impact of fault type on the relationship between code coverage and fault detection

Abstract

Structural coverage criteria are commonly used to determine the adequacy of a test suite. However, studies investigating structural coverage and fault-finding capabilities have mixed results. Some studies have shown generating test suites to satisfy structural coverage criteria has a positive effect on finding faults, while other studies show the opposite. These mixed results indicate there are factors not yet known that affect the ability of test suites satisfying structural coverage criteria to find faults. In order to improve the fault-finding capabilities of test suites, it is essential to understand what factors are causing this variance. Unfortunately very little work has been done to investigate the variance observed in the relationship between structural coverage criteria and fault-finding capabilities. In this paper, we investigate one possible source of variation in the results observed: fault type. We provide an empirical study which narrows down the focus of the relationship between structural coverage and fault-finding capabilities by focusing on object-oriented bugs. Specifically, we investigated 26 different types of object-oriented faults and evaluated how effectively test suites with high coverage percentages were able to detect each type of fault. We found that a test suite's ability to find faults varied significantly according to the type of fault (ranging from a rate of 0% to 87.5% mutants detected per fault type. We also found there are particular types of faults that were consistently found less frequently across all object programs.