Spectrum-based Fault Localization Research Articles

Boosting Spectrum-Based Fault Localization via Multi-Correct Programs in Online Programming

Boosting Spectrum-Based Fault Localization via Multi-Correct Programs in Online Programming

Understanding the use of spectrum‐based fault localization

SummaryDevelopers spend significant time locating and fixing bugs, which is often performed manually. Although spectrum‐based fault localization (SFL) techniques aim at helping developers to locate faults, they are not yet used in practice. Recent studies have investigated how developers use SFL, presenting different conclusions about their effectiveness and usefulness. We carried out a user study to further enhance the understanding of SFL. We assessed whether SFL improves the developers' performance and to what extent SFL leads developers to inspect faulty code excerpts. We also investigated the intention of the developers to use SFL and how they interact with SFL. Twenty‐six participants performed debugging tasks using real programs, with and without using the Jaguar SFL tool. Using SFL, more developers located and fixed the bugs. SFL also led more developers to inspect the faulty code and locate the faulty method. However, they did not spend less time locating the faults. SFL was well‐accepted by the participants, who showed intention to use it in their daily activities. Our results indicate that SFL is useful even when the fault is not ranked among the first positions, leading developers to reach faulty code regions and find the bugs.

Improving fault localization via weighted execution graph and graph attention network

AbstractSoftware fault localization is commonly recognized as arduous and time consuming. Spectrum‐based fault localization (SBFL) has been widely used due to its lightness. However, the effectiveness of SBFL is limited since it only considers simple statistics on the coverage information, ignoring the tie problem that the spectrum matrixes of some statements are the same. Most existing deep learning‐based fault localization (DLFL) techniques convert the coverage information into a vector, which utilizes the spectrum in a simplified manner and still has limitations in practice. To solve the above problem, we propose an approach via the weighted execution graph and graph attention network (WEGAT). We use a graph structure to represent the coverage information between test cases and program elements. Then, we generate a weighted execution graph by applying the predicate execution sequence. Furthermore, we combine the weighted execution graph with the AST as an integrated graph, which is the input of the GAT for fault localization. We evaluate WEGAT in within‐project and cross‐project prediction scenarios on the Defects4J benchmark. Experimental results show that our approach outperforms traditional SBFL (Ochiai, DStar and Tarantula) and DLFL (TraPT, CNN‐FL, Grace, and AGFL) methods, effectively improving the accuracy of fault localization.

Fault localization in DSLTrans model transformations by combining symbolic execution and spectrum-based analysis

AbstractThe verification of model transformations is important for realizing robust model-driven engineering technologies and quality-assured automation. Many approaches for checking properties of model transformations have been proposed. Most of them have focused on the effective and efficient detection of property violations by contract checking. However, there remains the fault localization step between identifying a failing contract for a transformation based on verification feedback and precisely identifying the faulty rules. While there exist fault localization approaches in the model transformation verification literature, these require the creation and maintenance of test cases, which imposes an additional burden on the developer. In this paper, we combine transformation verification based on symbolic execution with spectrum-based fault localization techniques for identifying the faulty rules in DSLTrans model transformations. This fault localization approach operates on the path condition output of symbolic transformation checkers instead of requiring a set of test input models. In particular, we introduce a workflow for running the symbolic execution of a model transformation, evaluating the defined contracts for satisfaction, and computing different measures for tracking the faulty rules. We evaluate the effectiveness of spectrum-based analysis techniques for tracking faulty rules and compare our approach to previous works. We evaluate our technique by introducing known mutations into five model transformations. Our results show that the best spectrum-based analysis techniques allow for effective fault localization, showing an average EXAM score below 0.30 (less than 30% of the transformation needs to be inspected). These techniques are also able to locate the faulty rule in the top-three ranked rules in 70% of all cases. The impact of the model transformation, the type of mutation and the type of contract on the results is discussed. Finally, we also investigate the cases where the technique does not work properly, including discussion of a potential pre-check to estimate the prospects of the technique for a certain transformation.

Diagnosis of intermittent faults in Multi-Agent Systems: An SFL approach

Multi-Agent Systems (MAS) can be found in a wide variety of applications, including industrial systems, transportation, software systems and more. In such systems, agents may experience faults that affect the performance of the whole system. However, faulty agents might not consistently experience their fault, but rather in certain conditions. For example, a robot with a faulty rotating mechanism will appear healthy if it is tasked to only move in a straight line. Those faults are called Intermittent Faults. Such faults may cause the entire system to fail, but not always. Previous work proposed diagnosis algorithms for MAS, assuming faulty agents persistently behave abnormally. To the best of our knowledge, intermittent faults in MAS have not been concretely explored. In this paper we formally present a novel problem called Diagnosis of Intermittent Faults in Multi-Agent Systems (DIFMAS): a group of agents are observed across multiple runs. In each run, the success/failure of the agents and the system is observed, aiming to explain all the failed runs by diagnosing which agent(s) are faulty. The contributions of this paper are: (1) formalizing DIFMAS as a Model-Based Diagnosis problem, (2) solving it by presenting a Spectrum-Based Fault Localization (SFL) based method, called Multi-Run SFL-based Diagnosis Algorithm (MRSD). Experiments demonstrate that MRSD's outperforms competing SFL-based algorithms. Moreover, the algorithm's performance increases if planned interactions are considered.

FEDRak: Federated Learning-Based Symmetric Code Statement Ranking Model for Software Fault Forecasting

Software Fault Forecasting (SFF) pertains to timely identifying sections in software projects that are prone to faults and may result in significant development expenses. Deep learning models have become widespread in software fault monitoring and management, and these models rely on the design metrics and the code pattern features for classifying the code as erroneous or safe. The proposed model works based on the collective formulation of the fault localization model, which acquires the model-specific metadata for building a global model that would perform software fault forecasting globally. The proposed model works by ranking the suspicious code blocks based on the symmetry of the semantic features of the erroneous code and the implementation code. The feature selection and scaling process is initially performed to precisely identify the features contributing to fault forecasting. The data extraction that is portrayed as the intermediate phase would assist in focusing on the code statements and ranking them based on the impact of the fault. A fine-tuned spectrum-based fault localization technique is used in ranking the statements. The FEDRak model facilitates ongoing adaptation in instances where there are adjustments in the feature contribution of data over time. The federated learning model would update the feature weights of the global model based on the weights synchronized by locally built fault forecasting approaches. FEDRak is statistically analyzed in relation to other contemporary techniques in fault localization in terms of metrics like sensitivity, specificity, accuracy, F1-score, and ROC curves. The proposed model’s performances are analyzed in terms of local and global models.

VsusFL: Variable-suspiciousness-based Fault Localization for novice programs

Automatically localizing faulty statements is a desired feature for effective learning programming. Most of the existing automated fault localization techniques are developed and evaluated on commercial or well-known open-source projects, which performed poorly on novice programs. In this paper, we propose a novel fault localization technique VsusFL (Variable-suspiciousness-based Fault Localization) for novice programs. VsusFL is inspired by simulating the manual program debugging process and takes advantage of variable value sequences. VsusFL can trace variable value changes, determine whether the intermediate state of the variables is correct, and report the potential faulty statements for novice programs. This paper presents the implementation of VsusFL and conducts empirical studies on 422 real faulty novice programs. Experimental results show that VsusFL performs much better than Grace, ANGELINA, VSBFL, Spectrum-Based Fault Localization (SBFL), and Variable-based Fault Localization (VFL) in terms of TOP-1, TOP-3, and TOP-5 metrics. Specifically, VsusFL can localize 90%, 35% and 9% more faulty statements than the best-performing baseline Grace. Moreover, We analyze the correlation between VsusFL and other techniques and find a weak correlation since they perform well on different programs, indicating the potential to further enhance fault localization performance through strategic integration of VsusFL with other methods.

Multiple fault localization based on ant colony algorithm via genetic operation

Debugging programs involves a considerable amount of time and resources spent on identifying the source of errors. Spectrum-based fault localization techniques are becoming increasingly popular due to their quick and effective performance when dealing with programs that have a single fault. However, these methods do not take into account the reality that most programs tend to have multiple faults. To address the above challenges, we propose an automatic approach using ant colony algorithm via genetic operation for multiple fault localization (ACOMFL). Our approach first constructs the search graph by the coverage information. Secondly, it designs the fitness function to select the best paths and dynamically adjusts the pheromone updating. Finally, it computes the suspiciousness ranking of each statement by the ant path information in the optimal path set. Four open-source benchmarks are utilized to validate the performance of the ACOMFL approach. The experimental results show ACOMFL achieves better results for multi-fault programs in comparison with six spectrum-based fault localization approaches and a genetic algorithm-based multiple fault localization approach. The ACOMFL is effective at locating multi-fault, serving as a helpful method for locating faults.

Distributed Spectrum-Based Fault Localization

Spectrum-Based Fault Localization (SFL) is a popular approach for diagnosing faulty systems. SFL algorithms are inherently centralized, where observations are collected and analyzed by a single diagnoser. Applying SFL to diagnose distributed systems is challenging, especially when communication is costly and there are privacy concerns. We propose two SFL-based algorithms that are designed for distributed systems: one for diagnosing a single faulty component and one for diagnosing multiple faults. We analyze these algorithms theoretically and empirically. Our analysis shows that the distributed SFL algorithms we developed output identical diagnoses to centralized SFL while preserving privacy.

Software multiple-fault localization using particle swarm optimization via genetic operation

Recently, spectrum-based fault localization approaches have been widely used for its fast and perform well for programs with only one fault.However, most of the existing methods do not consider the fact that the programs tend to have multiple faults. To address the above issue, we propose a Particle Swarm Optimization with genetic operation based Multiple-Fault Localization (PSOMFL). Our method models the software multiple-fault localization process as a search process for the particle swarm algorithm, which can quickly find the optimal solution in the multi-dimensional hyper-volume, and finally analyzes the optimal solution set to obtain the locations of multiple faults. We have implemented a prototype and conducted several experiments to compare PSOMFL against the existing fault localization approaches. The experimental results show that PSOMFL outperforms the compared methods and can reduce the costs by 5%-25% on average.

The MADAG Strategy for Fault Location Techniques

Spectrum-based fault localization (SBFL), which utilizes spectrum information of test cases to calculate the suspiciousness of each statement in a program, can reduce developers’ effort. However, applying redundant test cases from a test suite to fault localization incurs a heavy burden, especially in a restricted resource environment, and it is expensive and infeasible to inspect the results of each test input. Prioritizing/selecting appropriate test cases is important to enable the practical application of the SBFL technique. In addition, we must ensure that applying the selected tests to SBFL can achieve approximately the effectiveness of fault localization with whole tests. This paper presents a test case prioritization/selection strategy, namely the Minimal Aggregate of the Diversity of All Groups (MADAG). The MADAG strategy prioritizes/selects test cases using information on the diversity of the execution trace of each test case. We implemented and applied the MADAG strategy to 233 faulty versions of the Siemens and UNIX programs from the Software-artifact Infrastructure Repository. The experiments show that (1) the MADAG strategy uses only 8.99 and 14.27 test cases, with an average of 18, from the Siemens and UNIX test suites, respectively, and the SBFL technique has approximate effectiveness for fault localization on all test cases and outperforms the previous best test case prioritization method; (2) we verify that applying whole tests from the test suite may not achieve the better effectiveness in fault localization compared with the tests selected by MADAG strategy.

CRMF: A fault localization approach based on class reduction and method call frequency

AbstractIdentifying the location of faults in real‐world programs is one of the costly processes during software debugging. To reduce the debugging effort, various fault localization techniques have been proposed in recent years. Spectrum‐based fault localization (SBFL) is one kind of widely investigated fault localization technique. Most SBFL techniques first calculate the suspiciousness of program elements (such as statements, methods) to be faulty using the coverage information and execution results of tests. Then a rank list of program elements is generated according to their suspiciousness. However, some SBFL techniques only consider the binary coverage information (i.e., whether the program element is covered) but ignore some of the tests' running behaviors, such as the execution frequency when faults occur in the iteration entities or loop bodies, which are more likely to be faulty followed the propagation‐infection‐execution model. The execution frequency based techniques only replace the feature items of the existing formula limiting their effectiveness in fault localization. In this article, we propose a fault localization technique, class reduction and method call frequency (CRMF), which utilizes mutation analysis and information retrieval techniques. In particular, CRMF first uses mutation analysis to identify and reduce the classes, in which the program elements with a low probability of being faulty. Then we propose a new suspiciousness formula that applies information retrieval and considers method call frequency. To evaluate the effectiveness of CRMF, we conduct empirical studies on 264 real‐world programs from the Defects4J benchmark. Final results show that CRMF outperforms the statement frequency based technique FLSF and SBFL techniques (i.e., Ochiai, OP2, Tarantula, and Dstar) in both single‐fault programs and multiple‐fault programs. Specifically, CRMF can rank 29, 74, and 112 faults at the top 1, 3, 5 ranks and achieve a higher mean reciprocal rank for single‐fault programs and multiple‐fault programs. Finally, we discuss the essence of CRMF and analyze its effectiveness on multi‐fault programs in detail.

Effectively Combining Risk Evaluation Metrics for Precise Fault Localization

Spectrum-based fault localization (SBFL) is an automated fault localization technique that uses risk evaluation metrics to compute the suspiciousness scores from program spectra. Thus, risk evaluation metrics determine the technique’s performance. However, the existing experimental studies still show no optimal metric for different program structures and error types. It is possible to further optimize SBFL’s performance by combining different metrics. Therefore, this paper effectively explores the combination of risk evaluation metrics for precise fault localization. Based on extensive experiments using 92 faults from SIR and 357 faults from Defects4J repositories, we highlight what and which risk evaluation metrics to combine to maximize the efficiency and accuracy of fault localization. The experimental results show that combining risk evaluation metrics with high negative correlation values can improve fault localization effectiveness. Similarly, even though the combination of positively correlated effective risk evaluation metrics can outperform most negatively correlated non-effective ones, it still cannot improve the fault localization effectiveness. Furthermore, low-correlated risk evaluation metrics should also be considered for fault localization. The study concluded that getting highly negatively correlated risk evaluation metrics is almost impossible. The combination of such risk evaluation metrics would improve fault localization accuracy.

The improved dynamic slicing for spectrum-based fault localization.

BackgroundSpectrum-based Fault localization have proven to be useful in the process of software testing and debugging. However, how to improve the effectiveness of software fault localization has always been a research hot spot in the field of software engineering. Dynamic slicing can extract program dependencies under certain conditions. Thus, this technology is expected to benefit for locating fault.MethodsWe propose an improved dynamic slicing for spectrum-based fault localization under a general framework. We first obtain the dynamic slice of program execution. Secondly, we construct a mixed slice spectrum matrix from the dynamic slice of each test case and the corresponding test results. Finally, we compute the suspiciousness value of each statement in the mixed-slice spectram matrix.ResultsTo verify the performance of our method, we conduct an empirical study on 15 widely used open-source programs. Experimental results show that our approach achieves significant improvement than the compared techniques.ConclusionsOur approach can reduce approximately 1% to 17.79% of the average cost of code examined significantly.

Using contextual knowledge in interactive fault localization

Tool support for automated fault localization in program debugging is limited because state-of-the-art algorithms often fail to provide efficient help to the user. They usually offer a ranked list of suspicious code elements, but the fault is not guaranteed to be found among the highest ranks. In Spectrum-Based Fault Localization (SBFL) – which uses code coverage information of test cases and their execution outcomes to calculate the ranks –, the developer has to investigate several locations before finding the faulty code element. Yet, all the knowledge she a priori has or acquires during this process is not reused by the SBFL tool. There are existing approaches in which the developer interacts with the SBFL algorithm by giving feedback on the elements of the prioritized list. We propose a new approach called iFL which extends interactive approaches by exploiting contextual knowledge of the user about the next item in the ranked list (e. g., a statement), with which larger code entities (e. g., a whole function) can be repositioned in their suspiciousness. We implemented a closely related algorithm proposed by Gong et al., called Talk. First, we evaluated iFL using simulated users, and compared the results to SBFL and Talk. Next, we introduced two types of imperfections in the simulation: user’s knowledge and confidence levels. On SIR and Defects4J, results showed notable improvements in fault localization efficiency, even with strong user imperfections. We then empirically evaluated the effectiveness of the approach with real users in two sets of experiments: a quantitative evaluation of the successfulness of using iFL, and a qualitative evaluation of practical uses of the approach with experienced developers in think-aloud sessions.

Real world projects, real faults: evaluating spectrum based fault localization techniques on Python projects

Spectrum Based Fault Localization (SBFL) is a statistical approach to identify faulty code within a program given a program spectra (i.e., records of program elements executed by passing and failing test cases). Several SBFL techniques have been proposed over the years, but most evaluations of those techniques were done only on Java and C programs, and frequently involve artificial faults. Considering the current popularity of Python, indicated by the results of the Stack Overflow survey among developers in 2020, it becomes increasingly important to understand how SBFL techniques perform on Python projects. However, this remains an understudied topic. In this work, our objective is to analyze the effectiveness of popular SBFL techniques in real-world Python projects. We also aim to compare our observed performance on Python to previously-reported performance on Java. Using the recently-built bug benchmark BugsInPy as our fault dataset, we apply five popular SBFL techniques (Tarantula, Ochiai, OP, Barinel, and DStar) and analyze their performances. We subsequently compare our results with results from Java and C projects reported in earlier related works. We find that 1) the real faults in BugsInPy are harder to identify using SBFL techniques compared to the real faults in Defects4J, indicated by the lower performance of the evaluated SBFL techniques on BugsInPy; 2) older techniques such as Tarantula, Barinel, and Ochiai consistently outperform newer techniques (i.e., OP and DStar) in a variety of metrics and debugging scenarios; 3) claims in preceding studies done on artificial faults in C and Java (such as “OP outperforms Tarantula”) do not hold on Python real faults; 4) lower-performing techniques can outperform higher-performing techniques in some cases, emphasizing the potential benefit of combining SBFL techniques. Our results yield insight into how popular SBFL techniques perform in real Python faults and emphasize the importance of conducting SBFL evaluations on real faults.

A Fault Localization Method Based on Metrics Combination

Spectrum-Based Fault Localization (SBFL) is one of the most effective fault localization techniques, and its performance closely depends on the program spectra and the ranking formula. Despite the numerous proposed approaches for fault localization, there are still great demands for fault localization techniques that can help guide developers to the locations of faults. Therefore, this paper defines four metrics from the program spectrum, which can become essential components of ranking formulas to mitigate spectrum-based fault localization problems. These metrics are further combined to propose a new heuristic, Metrics Combination (MECO), which does not require any prior information on program structure or semantics to locate faults effectively. The evaluation experiments are conducted on the Defects4J and SIR datasets, and MECO is compared with the 18 maximal formulas. The experimental result shows that MECO is more efficient in terms of Precision, Accuracy, and Wasted Efforts than the compared formulas. An empirical evaluation also indicates that two of the defined metrics, Assumption Proportion and Fault Assumption, when combined with the existing formulas, improve the localization effectiveness, especially the precision of ER5a-c (77.77%), GP02 (41%), and GP19 (27.22%), respectively.

Fault localization using function call frequencies

In traditional Spectrum-Based Fault Localization (SBFL), hit-based spectrum is used to estimate a program element’s suspiciousness to contain a fault, i.e., only the binary information is used if the code element was executed by the test case or not. Count-based spectra can potentially improve the localization effectiveness due to the number of executions also being available. In this work, we use function-level granularity and define count-based spectra which use function call frequencies. We investigate the naïve approach, which simply counts the function call instances. We also define a novel method which is based on counting the different function call contexts, i.e., the frequency of the investigated function occurring in unique call stack instances during test execution. The basic intuition is that if a function is called in many different contexts during a failing test case, it will be more probable to be accountable for the fault. We empirically evaluated the fault localization capability of different variations of the approach and compared them to 9 traditional SBFL techniques using the Defects4J benchmark. We show that: (i) naïve counts result in worse rank positions than the hit-based approach, but (ii) unique counts produce better rank positions with some of the algorithm variants.

An Innovative Framework for Coincidental Correctness Impacting on Fault Localization

An important research aspect of Spectrum-Based Fault Localization (SBFL) is the influence factors of the effectiveness of suspiciousness formulas from the perspective of symmetry. Coincidental correctness is one of the most important factors impacting the effectiveness of suspiciousness formulas. The influence of fault localization by coincidental correctness has attracted a large amount of research in the perspective of empirical study; however, it can hardly be considered as sufficiently comprehensive when there are a large number of the symmetrical suspiciousness formulas. Therefore, we first develop an innovative theoretical framework with function derivation investigating suspiciousness formulas impacted by coincidental correctness. We define three types of relations between formulas affected by coincidental correctness: namely, improved type, invariant type and uncertain type. We investigated 30 suspiciousness formulas using this framework and group them into three categories. Furthermore, we conduct an empirical study to verify the effectiveness of SBFL affected by coincidental correctness on four relatively large C programs. We proved that coincidental correctness has a positive effect on 23 out of these 30 formulas, no effect on 5 of them, and the effect on the remaining 2 of them depend on certain conditions. The experimental results show that the effectiveness of some suspiciousness formulas can be enhanced and that of some suspiciousness formulas remain unchanged.

A Preliminary Investigation on the Performance of SBFL Techniques and Distance Metrics in Parallel Fault Localization

To locate multiple bugs in parallel, one common practice is to generate fault-focused clusters where failed test cases that are likely caused by the same bug are grouped together. With respect to the fault-focused clustering and fault localization, Spectrum-based fault localization (SBFL) technique and distance metric are two critical impact factors. This article proposes a method to evaluate the parallel fault localization performance of SBFL techniques and distance metrics where the fault-focused information for each failed test case is already given. Case studies are conducted using the proposed method to evaluate five SBFL techniques and five distance metrics on three programs with multiple bugs.