Statistical Fault Localization Research Articles

Locating faults with program slicing: an empirical analysis

Statistical fault localization is an easily deployed technique for quickly determining candidates for faulty code locations. If a human programmer has to search the fault beyond the top candidate locations, though, more traditional techniques of following dependencies along dynamic slices may be better suited. In a large study of 457 bugs (369 single faults and 88 multiple faults) in 46 open source C programs, we compare the effectiveness of statistical fault localization against dynamic slicing. For single faults, we find that dynamic slicing was eight percentage points more effective than the best performing statistical debugging formula; for 66% of the bugs, dynamic slicing finds the fault earlier than the best performing statistical debugging formula. In our evaluation, dynamic slicing is more effective for programs with single fault, but statistical debugging performs better on multiple faults. Best results, however, are obtained by a hybrid approach: If programmers first examine at most the top five most suspicious locations from statistical debugging, and then switch to dynamic slices, on average, they will need to examine 15% (30 lines) of the code. These findings hold for 18 most effective statistical debugging formulas and our results are independent of the number of faults (i.e. single or multiple faults) and error type (i.e. artificial or real errors).

Spectrum-Based and Program Slicing Statistical Fault Localization

Software fault localization is a necessary and time-consuming process in program debugging. An effective fault location method can help developers quickly locate faults and improve debug efficiency .In this paper, a new SIFL (Spectrum-Based Improved Fault Localization) method is proposed to improve the accuracy of fault location, by adding program slicing technology and test case selection technology to traditional spectrum-based fault localization. In the experimental section, this paper uses the public fault data set SIEMENT[1] to test the accuracy of the SIFL method. According to the experimental results, the SIFL method significantly improves the accuracy of fault location compared to SFL method. And using the SIFL method, the average fault location ratio is 3.076% of the program, which showing SIFL method can greatly narrow the scope of fault and improve debug efficiency.

FPA-FL: Incorporating static fault-proneness analysis into statistical fault localization

Despite the proven applicability of the statistical methods in automatic fault localization, these approaches are biased by data collected from different executions of the program. This biasness could result in unstable statistical models which may vary dependent on test data provided for trial executions of the program. To resolve the difficulty, in this article a new fault-proneness-aware statistical approach based on Elastic-Net regression, namely FPA-FL is proposed. The main idea behind FPA-FL is to consider the static structure and the fault-proneness of the program statements in addition to their dynamic correlations with the program termination state. The grouping effect of FPA-FL is helpful for finding multiple faults and supporting scalability. To provide the context of failure, cause-effect chains of program faults are discovered. FPA-FL is evaluated from different viewpoints on well-known test suites. The results reveal high fault localization performance of our approach, compared with similar techniques in the literature.

Fault Localization Using Function Call Sequences

Fault localization techniques locate faults in a program. Many predicate based statistical fault localization techniques have been proposed in order to locate faults effectively and efficiently. Most of the faults exist in independent functions rather than independent predicates in a program and the majority of these faults will cause abnormal sequences of function calls. This paper proposes a novel function-level sequence matching fault localization technique called SMFL, which use abnormal start points of function call sequences between normal traces and faulty traces. Our approach runs a program to calculate the suspiciousness of each function, and gives a ranking list of all functions in descending order of the suspiciousness. The disparities of function call sequences in different versions of the same software program on a test case are very small and these disparities provide information about faulty functions. We use Siemens program as our subjects. The experimental results show that our approach can improve the effectiveness of fault localization.

Causal inference based fault localization for numerical software with NUMFL

SummaryThis paper presents NUMFL, a value‐based causal inference technique for localizing faults in numerical software. NUMFL combines causal and statistical analyses to characterize the causal effects of individual numerical expressions on output errors. Given value‐profiles for an expression's variables, NUMFL uses generalized propensity scores or covariate balancing propensity scores to reduce confounding bias caused by evaluation of other, faulty expressions. It estimates the average failure‐causing effect of an expression using statistical regression models fit within generalized propensity score or covariate balancing propensity score subclasses (strata). This paper also reports on an empirical evaluation of NUMFL involving components from four Java numerical libraries, in which it was compared with five alternative statistical fault localization metrics. The results indicate that NUMFL is more effective than competitive statistical fault localization techniques. The results also indicate NUMFL that works surprisingly well with data from failing runs alone. Copyright © 2016 John Wiley & Sons, Ltd.

Slice-based statistical fault localization

Recent techniques for fault localization statistically analyze coverage information of a set of test runs to measure the correlations between program entities and program failures. However, coverage information cannot identify those program entities whose execution affects the output and therefore weakens the aforementioned correlations. This paper proposes a slice-based statistical fault localization approach to address this problem. Our approach utilizes program slices of a set of test runs to capture the influence of a program entity's execution on the output, and uses statistical analysis to measure the suspiciousness of each program entity being faulty. In addition, this paper presents a new slicing approach called approximate dynamic backward slice to balance the size and accuracy of a slice, and applies this slice to our statistical approach. We use two standard benchmarks and three real-life UNIX utility programs as our subjects, and compare our approach with a sufficient number of fault localization techniques. The experimental results show that our approach can significantly improve the effectiveness of fault localization.

On the adoption of MC/DC and control-flow adequacy for a tight integration of program testing and statistical fault localization

ContextTesting and debugging consume a significant portion of software development effort. Both processes are usually conducted independently despite their close relationship with each other. Test adequacy is vital for developers to assure that sufficient testing effort has been made, while finding all the faults in a program as soon as possible is equally important. A tight integration between testing and debugging activities is essential. ObjectiveThe paper aims at finding whether three factors, namely, the adequacy criterion to gauge a test suite, the size of a prioritized test suite, and the percentage of such a test suite used in fault localization, have significant impacts on integrating test case prioritization techniques with statistical fault localization techniques. MethodWe conduct a controlled experiment to investigate the effectiveness of applying adequate test suites to locate faults in a benchmark suite of seven Siemens programs and four real-life UNIX utility programs using three adequacy criteria, 16 test case prioritization techniques, and four statistical fault localization techniques. We measure the proportion of code needed to be examined in order to locate a fault as the effectiveness of statistical fault localization techniques. We also investigate the integration of test case prioritization and statistical fault localization with postmortem analysis. ResultThe main result shows that on average, it is more effective for a statistical fault localization technique to utilize the execution results of a MC/DC-adequate test suite than those of a branch-adequate test suite, and is in turn more effective to utilize the execution results of a branch-adequate test suite than those of a statement-adequate test suite. On the other hand, we find that none of the fault localization techniques studied can be sufficiently effective in suggesting fault-relevant statements that can fit easily into one debug window of a typical IDE. ConclusionWe find that the adequacy criterion and the percentage of a prioritized test suite utilized are major factors affecting the effectiveness of statistical fault localization techniques. In our experiment, the adoption of a stronger adequacy criterion can lead to more effective integration of testing and debugging.

How well does test case prioritization integrate with statistical fault localization?

ContextEffective test case prioritization shortens the time to detect failures, and yet the use of fewer test cases may compromise the effectiveness of subsequent fault localization. ObjectiveThe paper aims at finding whether several previously identified effectiveness factors of test case prioritization techniques, namely strategy, coverage granularity, and time cost, have observable consequences on the effectiveness of statistical fault localization techniques. MethodThis paper uses a controlled experiment to examine these factors. The experiment includes 16 test case prioritization techniques and four statistical fault localization techniques using the Siemens suite of programs as well as grep, gzip, sed, and flex as subjects. The experiment studies the effects of the percentage of code examined to locate faults from these benchmark subjects after a given number of failures have been observed. ResultsWe find that if testers have a budgetary concern on the number of test cases for regression testing, the use of test case prioritization can save up to 40% of test case executions for commit builds without significantly affecting the effectiveness of fault localization. A statistical fault localization technique using a smaller fraction of a prioritized test suite is found to compromise its effectiveness seriously. Despite the presence of some variations, the inclusion of more failed test cases will generally improve the fault localization effectiveness during the integration process. Interestingly, during the variation periods, adding more failed test cases actually deteriorates the fault localization effectiveness. In terms of strategies, Random is found to be the most effective, followed by the ART and Additional strategies, while the Total strategy is the least effective. We do not observe sufficient empirical evidence to conclude that using different coverage granularity levels have different overall effects. ConclusionThe paper empirically identifies that strategy and time–cost of test case prioritization techniques are key factors affecting the effectiveness of statistical fault localization, while coverage granularity is not a significant factor. It also identifies a mid-range deterioration in fault localization effectiveness when adding more test cases to facilitate debugging.

Focus section on program debugging

Focus section on program debugging

In quest of the science in statistical fault localization

SUMMARYMany researchers employ various statistical methods to locate faults in faulty programs. Like other researchers, we sometimes have made mistakes in the quest of making statistical fault localization both practical and scientific. In this experience report, we reflect on our work conducted on this topic, organize our isolated experiences in the format of models and errors, and cast them in the context of statistics. Copyright © 2011 John Wiley & Sons, Ltd.

Is non-parametric hypothesis testing model robust for statistical fault localization?

Fault localization is one of the most difficult activities in software debugging. Many existing statistical fault-localization techniques estimate the fault positions of programs by comparing the program feature spectra between passed runs and failed runs. Some existing approaches develop estimation formulas based on mean values of the underlying program feature spectra and their distributions alike. Our previous work advocates the use of a non-parametric approach in estimation formulas to pinpoint fault-relevant positions. It is worthy of further study to resolve the two schools of thought by examining the fundamental, underlying properties of distributions related to fault localization. In particular, we ask: Can the feature spectra of program elements be safely considered as normal distributions so that parametric techniques can be soundly and powerfully applied? In this paper, we empirically investigate this question from the program predicate perspective. We conduct an experimental study based on the Siemens suite of programs. We examine the degree of normality on the distributions of evaluation biases of the predicates, and obtain three major results from the study. First, almost all examined distributions of evaluation biases are either normal or far from normal, but not in between. Second, the most fault-relevant predicates are less likely to exhibit normal distributions in terms of evaluation biases than other predicates. Our results show that normality is not common as far as evaluation bias can represent. Furthermore, the effectiveness of our non-parametric predicate-based fault-localization technique weakly correlates with the distributions of evaluation biases, making the technique robust to this type of uncertainty in the underlying program spectra.