Assessing the Effectiveness of Coverage-Based Fault Localizations Using Mutants

Abstract

Empirical studies show that coverage-based fault localizations are very effective in testing and debugging software applications. It is also a commonly held belief that no software testing techniques would perform best for all programs with various data structures and complexity. An important research question posed in this paper is whether the type and complexity of faults in a given program has any influence on the performance of these fault localization techniques. This paper investigates the performance of coverage-based fault localizations for different types of faults. We explore and compare the accuracy of these techniques for two large groups of faults often observed in object-oriented programs. First, we explore different types of traditional method-level faults grouped into six categories including those related to arithmetic, relational, conditional, logical, assignment, and shift. We then focus on class-level faults related to object-oriented features and group them into four categories including inheritance, overriding, Java-specific features, and common programming mistakes. The results show that coverage-based fault localizations are less effective for class-level faults associated with object-oriented features of programs. We therefore advocate the needs for designing more effective fault localizations for debugging object-oriented and class-level defects.