Spectrum-based Fault Localization Research Articles

On the empirical evaluation of similarity coefficients for spreadsheets fault localization

Spreadsheets are by far the most prominent example of end-user programs of ample size and substantial structural complexity. They are usually not thoroughly tested so they often contain faults. Debugging spreadsheets is a hard task due to the size and structure, which is usually not directly visible to the user, i.e., the functions are hidden and only the computed values are presented. A way to locate faulty cells in spreadsheets is by adapting software debugging approaches for traditional procedural or object-oriented programming languages. One of such approaches is spectrum-based fault localization (Sfl). In this paper, we study the impact of different similarity coefficients on the accuracy of Sfl applied to the spreadsheet domain. Our empirical evaluation shows that three of the 42 studied coefficients (Ochiai, Jaccard and Sorensen-Dice) require less effort by the user while inspecting the diagnostic report, and can also be used interchangeably without a loss of accuracy. In addition, we illustrate the influence of the number of correct and incorrect output cells on the diagnostic report.

Fault density, fault types, and spectra-based fault localization

This paper presents multiple empirical experiments that investigate the impact of fault quantity and fault type on statistical, coverage-based fault localization techniques and fault-localization interference. Fault-localization interference is a phenomenon revealed in earlier studies of coverage-based fault localization that causes faults to obstruct, or interfere, with other faults' ability to be localized. Previously, it had been asserted that a fault-localization technique's effectiveness was negatively correlated to the quantity of faults in the program. To investigate these beliefs, we conducted an experiment on six programs consisting of more than 72,000 multiple-fault versions. Our data suggests that the impact of multiple faults exerts a significant, but slight influence on fault-localization effectiveness. In addition, faults were categorized according to four existing fault-taxonomies and found no correlation between fault type and fault-localization interference. In general, even in the presence of many faults, at least one fault was found by fault localization with similar effectiveness. Additionally, our data exhibits that fault-localization interference is prevalent and exerts a meaningful influence that may cause a fault's localizability to vary greatly. Because almost all real-world software contains multiple faults, these results affect the practical use and understanding of statistical fault-localization techniques.

Effective Fault Localization Using Weighted Test Cases

Locating faults in a program is prohibitively time-consuming and tedious, and therefore, many automated fault localization techniques have been proposed to assist in the debugging process. Spectrum based fault localization are promising techniques that can guide developers to the possible locations of faults. These techniques make a summary on the number of passing and failing tests cases to prioritize suspicious statements according to likelihood of containing program bugs for each statement. Though results are encouraging, these techniques treat all test cases as equally important, which ignore individual error diagnosis ability for different test cases. In this paper, we present an approach to exploit varying weights for individual test cases in the computation of suspiciousness scores so as to improve the effectiveness of spectrum based fault localization techniques. To validate our method, experiments were performed on eight typical SFL techniques using two standard benchmarks. Results are suggestive that for the studied SFL techniques, our method can significantly improve the fault localization effectiveness in most situations and in other cases it does not introduce much adverse impact on the techniques original performance.

A dynamic code coverage approach to maximize fault localization efficiency

Spectrum-based fault localization is amongst the most effective techniques for automatic fault localization. However, abstractions of program execution traces, one of the required inputs for this technique, require instrumentation of the software under test at a statement level of granularity in order to compute a list of potential faulty statements. This introduces a considerable overhead in the fault localization process, which can even become prohibitive in, e.g., resource constrained environments. To counter this problem, we propose a new approach, coined dynamic code coverage (DCC), aimed at reducing this instrumentation overhead. This technique, by means of using coarser instrumentation, starts by analyzing coverage traces for large components of the system under test. It then progressively increases the instrumentation detail for faulty components, until the statement level of detail is reached. To assess the validity of our proposed approach, an empirical evaluation was performed, injecting faults in six real-world software projects. The empirical evaluation demonstrates that the dynamic code coverage approach reduces the execution overhead that exists in spectrum-based fault localization, and even presents a more concise potential fault ranking to the user. We have observed execution time reductions of 27% on average and diagnostic report size reductions of 77% on average.

Improving spectrum-based fault-localization through spectra cloning for fail test cases beyond balanced test suite

Spectrum-based Fault Localization (SBFL) has been widely studied as a debugging technique to reduce time and effort in locating faulty code in software. In SBFL, execution profiles (spectra) of pass and fail test cases are analyzed with SBFL metric to rank software code according to their likeliness to be faulty. However, there are significantly more pass test cases than fail test cases in typical test suites for faulty programs. The domination of pass test cases creates imbalance test suites that have a negative impact on the performance of SBFL. This is attributed to the fact that the execution profiles of fail test cases provide the essential information on the location of faulty code. In this paper, we propose to clone the spectra of fail test cases beyond balanced test suite to improve the performance of SBFL. Our empirical study shows that the proposed cloning method significantly improves the performance of commonly used SBFL metrics. Furthermore, we attempt to identify the amount of cloning required to achieve the optimal performance for the proposed cloning method.

Incremental spectrum cloning algorithm for optimization of spectrum-based fault localization

Incremental spectrum cloning algorithm for optimization of spectrum-based fault localization

Noise Reduction for Spectrum-based Fault Localization

Spectrum-based Fault Localization (SBFL) has been proven to be an effective technique to locate faulty statement in program code. SBFL metrics exploit the records of statement execution (spectra) by pass and fail test cases for each statement to rank its likeliness to be faulty. However, in some cases, these spectra contain duplicated and ambiguous information or noise which may deteriorate the performance of SBFL metrics. We propose six noise reduction schemes to eliminate test cases which provide duplicated and ambiguous information and evaluate the resulting performance improvements in SBFL metrics. Based on our findings, we further provide a guide for SBFL practitioners to select the best performing noise reduction scheme for the SBFL metrics that they use.

A theoretical analysis of the risk evaluation formulas for spectrum-based fault localization

An important research area of Spectrum-Based Fault Localization (SBFL) is the effectiveness of risk evaluation formulas. Most previous studies have adopted an empirical approach, which can hardly be considered as sufficiently comprehensive because of the huge number of combinations of various factors in SBFL. Though some studies aimed at overcoming the limitations of the empirical approach, none of them has provided a completely satisfactory solution. Therefore, we provide a theoretical investigation on the effectiveness of risk evaluation formulas. We define two types of relations between formulas, namely, equivalent and better. To identify the relations between formulas, we develop an innovative framework for the theoretical investigation. Our framework is based on the concept that the determinant for the effectiveness of a formula is the number of statements with risk values higher than the risk value of the faulty statement. We group all program statements into three disjoint sets with risk values higher than, equal to, and lower than the risk value of the faulty statement, respectively. For different formulas, the sizes of their sets are compared using the notion of subset. We use this framework to identify the maximal formulas which should be the only formulas to be used in SBFL.

Debugging in the Extreme: Spectrum-based Fault Localization with Limited Test Cases

Spectrum-based Fault Localization (SBFL) is a well-known debugging technique that locates fault in program code by utilizing execution profiles (spectra) of pass and fail test cases. Hence, the performance of SBFL depends on the test cases executed and the test results. In the most extreme scenarios, the debugging process may have to be conducted with only one fail test case, one pass test case or no pass test case. These scenarios might occur due to extremely high or extremely low failure rates or when software testers decide to stop running more test cases due to time and resource constraints. However, limited test case execution profiles may reduce the accuracy of SBFL metrics. In view of this, we evaluate the performance of SBFL metrics in these extreme scenarios to identify the best performing SBFL metric for each of these scenarios. From the experiment results, we have further discovered the convergence in performance for SBFL metrics under these extreme scenarios.

Spectrum-based Fault Localization: A Pair Scoring Approach

Spectrum-based Fault Localization (SBFL) is a popular fault localization technique that ranks statements in a program according to their suspiciousness to be faulty based on the statement execution records (spectra) of pass and fail test cases. Many SBFL metrics have been proposed with varying accuracies in ranking of faulty statement. In this paper we proposed a new SBFL metric based on a pair scoring approach. We evaluated the performance of the proposed metric and compare it with other existing SBFL metrics. Despite its simplicity, we found the proposed metric outperformed majority of the existing SBFL metrics. 

A New Spectrum-based Fault Localization Method by Using Clustering Algorithm

Locating the errors or faults in the software debugging process is very expensive and time-consuming. Therefore, automated debugging has become popular due to its capability to reduce cost and increase the efficiency. Spectrum-based method is one of the most important automated fault localization methods in the Software debugging techniques. Most of the existing spectrum-based fault localization methods against some others, whereas the comparison of presented methods is not comprehensive due to the different test case set is utilized. This paper implemented the existing 28 methods to evaluate the spectrumbased fault localization methods in the same test case set. Furthermore, we proposed a new spectrumbased fault localization method, which utilize k-means algorithm, to obtain a suspicious ranking of program statements and capture the statement with accepted high suspiciousness, so as to improve the effectiveness of fault-localization. An experiment is designed and we apply the proposed method to our java plug-in project of spectrum-based fault localization. The experimental results show that this method can locate faults in the software effectively and has good performance.

AI for the win

A considerable amount of time in software engineering is spent in debugging. In practice, mainly debugging tools which allow for executing a program step-by-step and setting break points are used. This debugging method is however very time consuming and cumbersome. There is a need for tools which undertake the task of narrowing down the most likely fault locations. These tools must complete this task with as little user interaction as possible and the results computed must be beneficial so that such tools appeal to programmers. In order to come up with such tools, we present three variants of the well-known spectrum-based fault localization technique that are enhanced by using methods from Artificial Intelligence. Each of the three combined approaches outperforms the underlying basic method concerning diagnostic accuracy. Hence, the presented approaches support the hypothesis that combining techniques from different areas is beneficial. In addition to the introduction of these techniques, we perform an empirical evaluation, discuss open challenges of debugging and outline possible solutions.

Metamorphic slice: An application in spectrum-based fault localization

ContextBecause of its simplicity and effectiveness, Spectrum-Based Fault Localization (SBFL) has been one of the popular approaches towards fault localization. It utilizes the execution result of failure or pass, and the corresponding coverage information (such as program slice) to estimate the risk of being faulty for each program entity (such as statement). However, all existing SBFL techniques assume the existence of a test oracle to determine the execution result of a test case. But, it is common that test oracles do not exist, and hence the applicability of SBFL has been severely restricted. ObjectiveWe aim at developing a framework that can extend the application of SBFL to the common situations where test oracles do not exist. MethodOur approach uses a new concept of metamorphic slice resulting from the integration of metamorphic testing and program slicing. In SBFL, instead of using the program slice and the result of failure or pass for an individual test case, a metamorphic slice and the result of violation or non-violation of a metamorphic relation are used. Since we need not know the execution result for an individual test case, the existence of a test oracle is no longer a requirement to apply SBFL. ResultsAn experimental study involving nine programs and three risk evaluation formulas was conducted. The results show that our proposed solution delivers a performance comparable to the performance observed by existing SBFL techniques for the situations where test oracles exist. ConclusionWith respect to the problem that SBFL is only applicable to programs with test oracles, we propose an innovative solution. Our solution is not only intuitively appealing and conceptually feasible, but also practically effective. Consequently, test oracles are no longer mandatory for SBFL, and hence the applicability of SBFL is significantly extended.

Effective Fault Localization Approach Using Feedback

At the stage of software debugging, the effective interaction between software debugging engineers and fault localization techniques can greatly improve fault localization performance. However, most fault localization approaches usually ignore this interaction and merely utilize the information from testing. Due to different goals of testing and fault localization, the lack of interaction may lead to the issue of information inadequacy, which can substantially degrade fault localization performance. In addition, human work is costly and error-prone. It is vital to study and simulate the pattern of debugging engineers as they apply their knowledge and experience to this interaction to promote fault localization effectiveness and reduce their workload. Thus this paper proposes an effective fault localization approach to simulate this interaction via feedback. Based on results obtained from fault localization techniques, this approach utilizes test data generation techniques to automatically produce feedback for interacting with these fault localization techniques, and then iterate this process to improve fault localization performance until a specific stopping condition is satisfied. Experiments on two standard benchmarks demonstrate the significant improvement of our approach over a promising fault localization technique, namely the spectrum-based fault localization technique.

Test-driven Fault Navigation for Debugging Reproducible Failures

Debugging failing test cases, particularly the search for failure causes, is often a laborious and timeconsuming activity. With the help of spectrum-based fault localization developers are able to reduce the potentially large search space by detecting anomalies in tested program entities. However, such anomalies do not necessarily indicate defects and so developers still have to analyze numerous candidates one by one until they find the failure cause. This procedure is inefficient since it does not take into account how suspicious entities relate to each other, whether another developer is better qualified for debugging this failure, or how erroneous behavior comes to be. We present test-driven fault navigation as an interconnected debugging guide that integrates spectrumbased anomalies and failure causes. By analyzing failure-reproducing test cases, we reveal suspicious system parts, developers most qualified for addressing localized faults, and erroneous behavior in the execution history. The Paths tool suite realizes our approach: PathMap supports a breadth first search for narrowing down failure causes and recommends developers for help; PathFinder is a lightweight back-in-time debugger that classifies failing test behavior for easily following infection chains back to defects. The evaluation of our approach illustrates the improvements for debugging test cases, the high accuracy of recommended developers, and the fast response times of our corresponding tool suite.

Spectra-Based Fault Localization by Increasing Marginal Weight

Spectra-based fault localization technique uses coverage information to calculate every statement's likelihood of having a bug.And then rank the likelihood in a decreasing order to find the faulty statement.This paper improves the spectra-base fault localization technique by increasing the marginal weight of the failed test cases.That means that as the number of the failed test case increases,the weight of the failed test case also increases.Comparing with reducing or sustaining the weight of covered statement's successful/failed test case,the experimental result shows that increasing the marginal weight of the covered statement's failed test case can promote the fault localization efficiency.

Simultaneous debugging of software faults

(Semi-)automated diagnosis of software faults can drastically increase debugging efficiency, improving reliability and time-to-market. Current automatic diagnosis techniques are predominantly of a statistical nature and, despite typical defect densities, do not explicitly consider multiple faults, as also demonstrated by the popularity of the single-fault benchmark set of programs. We present a reasoning approach, called Zoltar-M(ultiple fault), that yields multiple-fault diagnoses, ranked in order of their probability. Although application of Zoltar-M to programs with many faults requires heuristics (trading-off completeness) to reduce the inherent computational complexity, theory as well as experiments on synthetic program models and multiple-fault program versions available from the software infrastructure repository (SIR) show that for multiple-fault programs this approach can outperform statistical techniques, notably spectrum-based fault localization (SFL). As a side-effect of this research, we present a new SFL variant, called Zoltar-S(ingle fault), that is optimal for single-fault programs, outperforming all other variants known to date.

A practical evaluation of spectrum-based fault localization

Spectrum-based fault localization (SFL) shortens the test–diagnose–repair cycle by reducing the debugging effort. As a light-weight automated diagnosis technique it can easily be integrated with existing testing schemes. Since SFL is based on discovering statistical coincidences between system failures and the activity of the different parts of a system, its diagnostic accuracy is inherently limited. Using a common benchmark consisting of the Siemens set and the space program, we investigate this diagnostic accuracy as a function of several parameters (such as quality and quantity of the program spectra collected during the execution of the system), some of which directly relate to test design. Our results indicate that the superior performance of a particular similarity coefficient, used to analyze the program spectra, is largely independent of test design. Furthermore, near-optimal diagnostic accuracy (exonerating over 80% of the blocks of code on average) is already obtained for low-quality error observations and limited numbers of test cases. In addition to establishing these results in the controlled environment of our benchmark set, we show that SFL can effectively be applied in the context of embedded software development in an industrial environment.