Effectiveness Of Fault Localization Research Articles

Delta4Ms: Improving mutation‐based fault localization by eliminating mutant bias

AbstractFault localization is a complex, costly and time‐consuming task in software debugging. Numerous automated techniques have been developed to expedite this process. Mutation‐based fault localization (MBFL) is one of the most widely studied techniques which uses mutation analysis to generate mutants for revealing potential faults in the program. However, our theoretical analysis exposes an inherent conflict between the fundamental assumption and the essential meaning of existing MBFL suspiciousness. This conflict is caused by mutant bias. Intuitively, the suspiciousness can be corrected by eliminating the mutant bias for more accurately measuring the faulty probability of the corresponding mutant statement. In this paper, we introduce Delta4Ms, a fault localization approach designed to eliminate mutant bias. Delta4Ms integrates the principles of signal theory, modelling the actual suspiciousness and mutant bias as the desired and false signal components, respectively. Based on theoretical derivation, the average suspiciousness of mutants serves as an estimate of mutant bias. Delta4Ms effectively mitigates mutant bias, extracting the desired signal and yielding corrected suspiciousness for fault localization. To precisely estimate mutant bias, higher order mutants (HOMs) are incorporated. We conduct an extensive experimental evaluation of Delta4Ms on 320 real‐fault programs from Codeflaws. The results indicate that our model significantly outperforms existing SBFL and MBFL techniques, showing a considerable improvement in fault localization effectiveness. We further assessed the robustness of Delta4Ms by examining different HOM ratios and HOM generation strategies. Moreover, Delta4Ms achieves a substantial reduction in mutation execution cost and minimal accuracy loss through the implementation of test case reduction. Finally, we perform preliminary experiments on 15 real‐fault programs from the Defects4J benchmark to assess the generalization of the model's fault localization effectiveness.

The MADAG Strategy for Fault Location Techniques

Spectrum-based fault localization (SBFL), which utilizes spectrum information of test cases to calculate the suspiciousness of each statement in a program, can reduce developers’ effort. However, applying redundant test cases from a test suite to fault localization incurs a heavy burden, especially in a restricted resource environment, and it is expensive and infeasible to inspect the results of each test input. Prioritizing/selecting appropriate test cases is important to enable the practical application of the SBFL technique. In addition, we must ensure that applying the selected tests to SBFL can achieve approximately the effectiveness of fault localization with whole tests. This paper presents a test case prioritization/selection strategy, namely the Minimal Aggregate of the Diversity of All Groups (MADAG). The MADAG strategy prioritizes/selects test cases using information on the diversity of the execution trace of each test case. We implemented and applied the MADAG strategy to 233 faulty versions of the Siemens and UNIX programs from the Software-artifact Infrastructure Repository. The experiments show that (1) the MADAG strategy uses only 8.99 and 14.27 test cases, with an average of 18, from the Siemens and UNIX test suites, respectively, and the SBFL technique has approximate effectiveness for fault localization on all test cases and outperforms the previous best test case prioritization method; (2) we verify that applying whole tests from the test suite may not achieve the better effectiveness in fault localization compared with the tests selected by MADAG strategy.

Effectively Combining Risk Evaluation Metrics for Precise Fault Localization

Spectrum-based fault localization (SBFL) is an automated fault localization technique that uses risk evaluation metrics to compute the suspiciousness scores from program spectra. Thus, risk evaluation metrics determine the technique’s performance. However, the existing experimental studies still show no optimal metric for different program structures and error types. It is possible to further optimize SBFL’s performance by combining different metrics. Therefore, this paper effectively explores the combination of risk evaluation metrics for precise fault localization. Based on extensive experiments using 92 faults from SIR and 357 faults from Defects4J repositories, we highlight what and which risk evaluation metrics to combine to maximize the efficiency and accuracy of fault localization. The experimental results show that combining risk evaluation metrics with high negative correlation values can improve fault localization effectiveness. Similarly, even though the combination of positively correlated effective risk evaluation metrics can outperform most negatively correlated non-effective ones, it still cannot improve the fault localization effectiveness. Furthermore, low-correlated risk evaluation metrics should also be considered for fault localization. The study concluded that getting highly negatively correlated risk evaluation metrics is almost impossible. The combination of such risk evaluation metrics would improve fault localization accuracy.

Detraque: Dynamic execution tracing techniques for automatic fault localization of hardware design code.

In an error-prone development process, the ability to localize faults is a crucial one. Generally speaking, detecting and repairing errant behavior at an early stage of the development cycle considerably reduces costs and development time. The debugging of the Verilog program takes much time to read the waveform and capture the signal, and in many cases, problem-solving relies heavily on experienced developers. Most existing Verilog fault localization methods utilize the static analysis method to find faults. However, using static analysis methods exclusively may result in some types of faults being inevitably ignored. The use of dynamic analysis could help resolve this issue. Accordingly, in this work, we propose a new fault localization approach for Verilog, named Detraque. After obtaining dynamic execution through test cases, Detraque traces these executions to localize faults; subsequently, it can determine the likelihood of any Verilog statement being faulty and sort the statements in descending order by suspicion score. Through conducting empirical research on real Verilog programs with 61 faulty versions, Detraque can achieve an EXAM score of 18.3%. Thus, Detraque is verified as able to improve Verilog fault localization effectiveness when used as a supplement to static analysis methods.

The improved dynamic slicing for spectrum-based fault localization.

BackgroundSpectrum-based Fault localization have proven to be useful in the process of software testing and debugging. However, how to improve the effectiveness of software fault localization has always been a research hot spot in the field of software engineering. Dynamic slicing can extract program dependencies under certain conditions. Thus, this technology is expected to benefit for locating fault.MethodsWe propose an improved dynamic slicing for spectrum-based fault localization under a general framework. We first obtain the dynamic slice of program execution. Secondly, we construct a mixed slice spectrum matrix from the dynamic slice of each test case and the corresponding test results. Finally, we compute the suspiciousness value of each statement in the mixed-slice spectram matrix.ResultsTo verify the performance of our method, we conduct an empirical study on 15 widely used open-source programs. Experimental results show that our approach achieves significant improvement than the compared techniques.ConclusionsOur approach can reduce approximately 1% to 17.79% of the average cost of code examined significantly.

A Fault Localization Approach Based on BiRNN and Multi-Dimensional Features

Software fault localization is notoriously tedious and time-consuming. Developed rapidly, machine learning techniques have been adopted for fault localization by researchers. Most existing approaches use the test coverage information as feature input to the learning model, ignoring the limited ability of the single-dimensional features. The effectiveness of fault localization is not greatly improved. To overcome the limitation, we propose a fault localization approach based on Bidirectional Recurrent Neural Networks (BiRNNs) and multi-dimensional features. Our approach collects suspiciousness-based, text similarity-based and fault-proneness-based features from the traditional fault localization areas and software metrics. To evaluate our approach, the experiments have been studied on the real-fault benchmark Defects4J and seeded fault program NanoXML. The experimental results show that our approach effectively improves fault localization accuracy.

Feature-FL: Feature-Based Fault Localization

Fault localization aims at developing an effective methodology identifying suspicious statements potentially responsible for program failures. The spectrum-based fault localization is the widely used methodology by analyzing the statistical coincidences viewed from the spectrum to evaluate the suspiciousness of each statement of being faulty. However, just analyzing statistical coincidences in the coverage information perspective and without combining diverse amount of information may restrict fault localization effectiveness. Thus, this article proposes feature-based fault localization ( <monospace xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">Feature-FL</monospace> ): A family fault localization methodology of feature-based metrics by combining the feature diversity from the view of program features into suspiciousness evaluation. Specifically, <monospace xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">Feature-FL</monospace> defines a concept of branching execution probability to abstract program behaviors as the values of features. Then, <monospace xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">Feature-FL</monospace> uses feature selection (i.e., a family of feature-based metrics) to evaluate the relevance of each feature with program failures. Finally, <monospace xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">Feature-FL</monospace> associates each feature with its corresponding statement, and uses the relevance as the suspiciousness to locate suspicious statements. We present six feature-based metrics for <monospace xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">Feature-FL</monospace> , and conduct an extensive study to evaluate the effectiveness of <monospace xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">Feature-FL</monospace> and its potential over the state-of-the-art spectrum-based formulas. Our results provide insight into the potential among different feature-based metrics and also show <monospace xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">Feature-FL</monospace> significantly outperforms the state-of-the-art spectrum-based formulas, e.g., an average <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">saving</i> of at least 30% over spectrum-based formulas in case of real faults.

An Empirical Study on Higher-Order Mutation-Based Fault Localization

Fault localization is one of the most expensive activities in software debugging. Mutation-based fault localization (MBFL) is a commonly studied technique that applied mutation analysis to find the location of faults in the programs. Previous studies showed that MBFL adopted First-Order-Mutants (FOMs) that could achieve promising results in single-fault localization, but it did not perform well in multiple-fault localization. Recently, Higher-Order-Mutants (HOMs) were proposed for modeling complex faults but whether HOMs can help in fault localization is still unknown. In this paper, we investigate the performance of MBFL with FOMs and HOMs on single- and multiple-fault localization. Moreover, to study the characteristics of HOMs, we divide HOMs into three groups (i.e. Accurate HOMs, Partially accurate HOMs, and Inaccurate HOMs) by considering different mutation locations. Based on the empirical results on 186 versions of six real-world programs, we find that (1) In single-fault localization, FOMs can achieve better performance than HOMs. (2) However, in multiple-fault localization, HOMs (2-HOMs) localize more faults than FOMs. (3) Furthermore, different types of HOMs have different fault localization effectiveness, where Accurate HOMs outperform the other two HOMs categories. Therefore, the researchers should propose methods to find HOMs more useful for fault localization.

Effective fault localization and context‐aware debugging for concurrent programs

SummaryConcurrent programs are difficult to debug because concurrency faults usually occur under specific inputs and thread interleavings. Fault localization techniques for sequential programs are often ineffective because the root causes of concurrency faults involve memory accesses across multiple threads rather than single statements. Previous research has proposed techniques to analyse passing and failing executions obtained from running a set of test cases for identifying faulty memory access patterns. However, stand‐alone access patterns do not provide enough contextual information, such as the path leading to the failure, for developers to understand the bug. We present an approach, Coadec, to automatically generate interthread control flow paths that can link memory access patterns that occurred most frequently in the failing executions to better diagnose concurrency bugs. Coadec consists of two phases. In the first phase, we use feature selection techniques from machine learning to localize suspicious memory access patterns based on failing and passing executions. The patterns with maximum feature diversity information can point to the most suspicious pattern. We then apply a data mining technique and identify the memory access patterns that occurred most frequently in the failing executions. Finally, Coadec identifies faulty program paths by connecting both the frequent patterns and the suspicious pattern. We also evaluate the effectiveness of fault localization using test suites generated from different test adequacy criteria. We introduce and have evaluated Coadec on 10 real‐world multithreaded Java applications. Results indicate that Coadec outperforms state‐of‐the‐art approaches for localizing concurrency faults and that Coadec's context debugging can help developers understand concurrency fault by inspecting a small percentage of code.

Fault localization based on wide & deep learning model by mining software behavior

Many learning based fault localization approaches haven been proposed to improve the effectiveness by fusing various dimension of fault diagnosis features. However, method calls behavior has been neglected, and the interaction between features has not been fully explored. To solve this problem, firstly, a fault localization method by mining software behavior graphs has been proposed to improve the effectiveness of localizing function call related faults. Then, a fault localization approach by wide & deep learning on multi-feature groups has been proposed. Not only the spectrum based and mutation based suspiciousness features have been analyzed, but also the behavior based and invariants based suspiciousness, the static metrics, as well as the combined features of crash stack trace with the invariants change features have been integrated. Wide & Deep model is adopted as the ranking model, to explore the relationships between these features, so as to improve the effectiveness of fault localization. Experiments on practical software defects benchmark Defects4J have shown that our model outperforms the traditional spectrum-based and mutation-based approaches, it also outperforms the state-art-of learning-based approaches on the capability of early fault detection.

Locating faults with program slicing: an empirical analysis

Statistical fault localization is an easily deployed technique for quickly determining candidates for faulty code locations. If a human programmer has to search the fault beyond the top candidate locations, though, more traditional techniques of following dependencies along dynamic slices may be better suited. In a large study of 457 bugs (369 single faults and 88 multiple faults) in 46 open source C programs, we compare the effectiveness of statistical fault localization against dynamic slicing. For single faults, we find that dynamic slicing was eight percentage points more effective than the best performing statistical debugging formula; for 66% of the bugs, dynamic slicing finds the fault earlier than the best performing statistical debugging formula. In our evaluation, dynamic slicing is more effective for programs with single fault, but statistical debugging performs better on multiple faults. Best results, however, are obtained by a hybrid approach: If programmers first examine at most the top five most suspicious locations from statistical debugging, and then switch to dynamic slices, on average, they will need to examine 15% (30 lines) of the code. These findings hold for 18 most effective statistical debugging formulas and our results are independent of the number of faults (i.e. single or multiple faults) and error type (i.e. artificial or real errors).

A Framework for Improving Fault Localization Effectiveness Based on Fuzzy Expert System

Many spectrum-based fault localization (SBFL) techniques have been proposed in order to improve debugging efficiency. These SBFL techniques were designed according to different underlying assumptions and then adopt different fault locator functions to evaluate the likelihood of each statement being faulty, called suspiciousness. So far no single SBFL technique claims that it can outperform all of the others under every scenario. That is, the effectiveness of fault localization may vary considerably by just adopting a single SBFL technique. Due to the aforementioned reasons, this study presents a framework for improving fault localization effectiveness by using a Fuzzy Expert System (FES) to integrate different SBFL techniques. In the presented framework, the outputs of several SBFL techniques will be put into the fuzzification and then transferred to fuzzy input sets. After undergoing the fuzzy inference based on the given fuzzy rules, the fuzzy input sets will be transferred to a fuzzy output set. Finally, the fuzzy set will be transferred to a crisp output (called a weighted suspiciousness value). The code statements will then be ranked according to their weighted suspiciousness values. In other words, no additional instrumentations and analyses on the source code and the test suite are necessary for our approach. Our experiment results indicate that our FES-based framework is effective at combining the SBFL techniques from different equivalent groups and achieves high effectiveness on the nine subject programs. It is also noted that in the literature, most of the approaches that combine multiple SBFL techniques are learning-based and they are suitable for long-term projects with sufficient historical data. Since our approach does not reference historical data for model training, it can be applied to new software projects. Thus, the application scenarios of our approach should be complementary to those of the state-of-the-art learning-based approaches.

A study of effectiveness of deep learning in locating real faults

Abstract Context: The recent progress of deep learning has shown its promising learning ability in making sense of data, and many fields have utilized this learning ability to learn an effective model, successfully solving their problems. Fault localization has explored and used deep learning to server an aid in debugging, showing the promising results on fault localization. However, as far as we know, there is no detailed studies on evaluating the benefits of using deep learning for locating real faults present in programs. Objective: To understand the benefits of deep learning in locating real faults, this paper explores more about deep learning by studying the effectiveness of fault localization using deep learning for a set of real bugs reported in the widely used programs. Method: We use three representative deep learning architectures (i.e. convolutional neural network, recurrent neural network and multi-layer perceptron) for fault localization, and conduct large-scale experiments on 8 real-world programs equipped with all real faults to evaluate their effectiveness on fault localization. Results: We observe that the localization effectiveness varies considerably among three neural networks in the context of real faults. Specifically, convolutional neural network performs the best in locating real faults, showing an average of 38.97% and 26.22% saving over multi-layer perceptron and recurrent neural network respectively; recurrent neural network and multi-layer perceptron yield comparable effectiveness even if the effectiveness of recurrent neural network is marginally higher than multi-layer perceptron. Conclusion: In context of real faults, convolutional neural network is the most effective for fault localization among the investigated architectures, and we suggest potential factors of deep learning for improving fault localization.

Improving deep‐learning‐based fault localization with resampling

AbstractMany fault localization approaches recently utilize deep learning to learn an effective localization model showing a fresh perspective with promising results. However, localization models are generally learned from class imbalance datasets; that is, the number of failing test cases is much fewer than passing test cases. It may be highly susceptible to affect the accuracy of learned localization models. Thus, in this paper, we explore using data resampling to reduce the negative effect of the imbalanced class problem and improve the accuracy of learned models of deep‐learning‐based fault localization. Specifically, for deep‐learning‐based fault localization, its learning feature may require duplicate essential data to enhance the weak but beneficial experience incurred by the class imbalance datasets. We leverage the property of test cases (i.e., passing or failing) to identify failing test cases as the duplicate essential data and propose an iterative oversampling approach to resample failing test cases for producing a class balanced test suite. We apply the test case resampling to representative localization models using deep learning. Our empirical results on eight large‐sized programs with real faults and four large‐sized programs with seeded faults show that the test case resampling significantly improves fault localization effectiveness.

Fault Localization With Weighted Test Model in Model Transformations

Model transformations and model-driven engineering (MDE) have been applied widely in service-oriented architecture based information systems. To support the development of such a service-oriented information system, it is necessary to guarantee the quality of model transformations. With the increasing complexity and scale of model transformations, debugging of model transformations becomes more and more time-consuming and difficult, there is an increasing need to rely on efficient and accurate fault localization approaches to help with debugging. Among the existing fault localization approaches, the spectrum-based fault localization (SBFL), as a dynamic analysis method, mainly used the coverage information and execution results of the rules of model transformation to estimate the probability of each rule may be faulty. However, there are many false-positive and false-negative results in the rule coverage information, the accuracy of the SBFL is not ideal, so we consider mining the impact of covered range in different test models to further improve the effectiveness of fault localization. In this paper, we propose an optimized strategy of fault localization based on the impact of the test model, according to the covered range of test models, weight values are assigned to different test models, and the statistical coverage information of rules are weighted and adjusted accordingly. We compare the proposed approach with the SBFL, the experimental results show that under the same techniques for computing the suspiciousness, our approach can help locate around 26% more faulty rules in the ranking Top-1 of the suspicious list than the SBFL, the effectiveness of fault localization techniques can be improved by 50.42% in the best case and 8.9% in the average case.

How test suites impact fault localisation starting from the size

Although a test suite is indispensable for conducting effective fault localisation, not much work has been done to study how the test suite impacts fault localisation. This study presents a systematic study for a deeper understanding of their relation. Specifically, the authors' study reveals an interesting fact that there is no strong correlation between localisation effectiveness and the size of the test suite. Furthermore, they show that, in a test suite, (i) the passing test cases that do not execute the faulty statements and the failing test cases have a positive effect on the fault localisation effectiveness, while (ii) the passing test cases that exercise the faulty statements have a negative impact on localisation performance. Their result is drawn from a large-scale empirical analysis on the localisation effectiveness with respect to randomly sampled test suites. This study presents the details of the study and their follow-up investigation on the findings. Their work provides a new perspective on fault localisation and suggests fresh directions of research on an extensively studied topic.

Test case selection using multi-criteria optimization for effective fault localization

As spectra-based fault localization techniques report suspicious statements by analyzing the coverage of test cases, the effectiveness of the results is highly dependent on the composition of test suites. This paper proposes an approach for selecting a subset of the passed test suite when a failure revealed by a failed test case. The goal is to obtain a more effective fault localization using a minimal number of test cases than using the originally given large number of test cases. A novelty is that a prioritization criterion and a selection criterion are defined. Different from previous studies, the failed trace is fully considered. The prioritization criterion partitions statements in the failed trace into more suspicious and less suspicious, and then ranks passed test cases by their ability in distinguishing the more suspicious statements from the less suspicious ones. The selection criterion selects the minimal passed test suite which can maximize the number of coverage equivalent classes in the failed trace, so as to distinguish the suspicious statements and meanwhile reduce the size of the test suite. Another novelty is that our approach turns the test case selection into a multi-criteria optimization to make the prioritization and the selection criteria complement each other. This approach was evaluated with 5 fault localization techniques, 8 subject programs and 35,392 test cases. The results show that the fault localization effectiveness can be significantly improved with less than 5% passed test cases. Our approach has advantages over the statement- based and vector-based test suite reduction approaches in both fault localization effectiveness and test suite reduction rate.

Kernel-based detection of coincidentally correct test cases to improve fault localisation effectiveness

Although empirical studies have confirmed the effectiveness of spectrum-based fault localization (SBFL) techniques, their performance may be degraded due to presence of some undesired circumstances such as the existence of coincidental correctness (CC) where one or more passing test cases exercise a faulty statement and thus causing some confusion to decide whether the underlying exercised statement is faulty or not. This article aims at improving SBFL effectiveness by mitigating the effect of CC test cases. In this regard, a new method is proposed that uses a support vector machine (SVM) with a customized kernel function. To build the kernel function, we applied a new sequence-matching algorithm that measures the similarities between passing and failing executions. We conducted some experiments to assess the proposed method. The results show that our method can effectively improve the performance of SBFL techniques.

Simultaneous Localization of Software Faults Based on Complex Network Theory

Effective debugging is necessary for producing high quality and reliable software. Fault localization plays a vital role in the debugging process. However, fault localization is the most tedious and expensive activity in program debugging, as such, effective fault localization techniques that can identify the exact location of faults is eminent. Despite various fault localization techniques proposed, their application in multiple-fault programs is limited. The presence of multiple faults in a program reduces the efficacy of the existing fault localization techniques to locate faults effectively due to fault interference. Moreover, most of these techniques are unable to localize faults simultaneously. This has led researchers to adopt alternative approaches, such as one-fault-at-a-time debugging and parallel debugging. In this paper, we propose a novel fault localization technique based on complex network theory (FLCN) to improve localization effectiveness in programs with single-fault and multiple-fault and to aid developers to localize multiple faults simultaneously in a single diagnosis rank list. The proposed technique ranks faulty statements based on their behavioral abnormalities and distance between faulty statements in both passed and failed test executions. Two graph-based centrality measures are adopted for fault diagnosis, namely, degree centrality and closeness centrality and a new ranking formula is proposed. FLCN is evaluated across 14 subjects with both single-fault and multiple-fault programs. Our experimental results show that FLCN is more effective at locating faults when compared with existing state-of-the-art fault localization techniques with improvement of fault localization effectiveness in both single-fault and multiple-fault programs.

Kernel-based detection of coincidentally correct test cases to improve fault localisation effectiveness

Although empirical studies have confirmed the effectiveness of spectrum-based fault localisation (SBFL) techniques, their performance may be degraded due to presence of some undesired circumstances such as the existence of coincidental correctness (CC) where one or more passing test cases exercise a faulty statement and thus causing some confusion to decide whether the underlying exercised statement is faulty or not. This article aims at improving SBFL effectiveness by mitigating the effect of CC test cases. In this regard, a new method is proposed that uses a support vector machine (SVM) with a customised kernel function. To build the kernel function, we applied a new sequence-matching algorithm that measures the similarities between passing and failing executions. We conducted some experiments to assess the proposed method. The results show that our method can effectively improve the performance of SBFL techniques.