A program slicing-based method for effective detection of coincidentally correct test cases

Abstract

Despite the proven applicability of the spectrum-based fault localization (SBFL) methods, their effectiveness may be degraded due to the presence of coincidental correctness, which occurs when faults fail to propagate, i.e., their execution does not result in failures. This article aims at improving SBFL effectiveness by mitigating the effect of coincidentally correct test cases. In this regard, given a test suite in which each test has been classified as failing or passing and each faulty program has a single-bug, we present a program slicing-based technique to identify a set of program entities that directly affect the program output when executed with failing test cases, called failure candidate causes (FCC). We then use FCC set to identify test cases that can be marked as being coincidentally correct. These tests are identified based on two heuristics: the average suspiciousness score of the statements that directly affect the program output and the coverage ratio of those statements. To evaluate our approach, we used several evaluation metrics and conducted extensive experiments on programs containing single and multiple bugs, including both real and seeded faults. The empirical results demonstrate that the proposed heuristics can alleviate the coincidental correctness problem and improve the accuracy of SBFL techniques.