Exploring the Instability of Spectra Based Fault Localization Performance

Abstract

Spectra Based Fault Localization (SBFL) is a technique to improve the efficiency of software fault localization. The performance of SBFL largely depends on the input information provided by an executed test suite. Due to the randomness existing in the testing process, the output of SBFL may not be stable. In practice, testers do not have the chance to run the whole testing process many times. They are not sure whether the actually obtained SBFL output has a large deviation from the ideal output (i.e. the SBFL output obtained under the assumption that the amount of testing resources is unlimited). Thus, concerning the application of SBFL in real cases, such instability of its performance (SBFL instability for short) is a challenge. In this paper, the SBFL instability is discussed and its characteristics are further explored. Specifically, we define SBFL instability as a stochastic quantity and introduce the measure of StabilityLevel to quantify it. Then, based on the definition and measurement, we conduct experimental studies to demonstrate that SBFL instability is indeed a prevalent phenomenon and also a serious problem. Besides, two factors which influence the intensity of SBFL instability, i.e. the test suite size and risk evaluation formula, are observed and analyzed.