Mutation-based Fault Localization Research Articles

A Systematic Exploration of Mutation‐Based Fault Localization Formulae

ABSTRACTFault localization (FL) aims to automatically find the location of bugs in a software program. In the family of FL approaches, spectrum‐based fault localization (SBFL) is the most widely used and has been extensively studied, which computes the suspicious scores of a code element to be buggy via test coverage information. Mutation‐based fault localization (MBFL) further analyses the relationship between mutants and test results. Although MBFL involves more information, it also faces the following limitations. (1) To precisely evaluate suspicious sources, SBFL approaches proposed dozens of formulae based on coverage, while only a few of them have been adopted by MBFL. (2) The current MBFL approaches are based on the assumption that the capability in localizing bugs of each mutant is the same, so they assign the same weight to the mutants that change the test outputs and consider the mutant from the same code element equivalent. In this paper, we intend to enrich the MBFL family by the following two approaches. First, we collect 25 typical SBFL formulae and transform them into MBFL versions. Second, we propose new assumptions that the mutants should have different weights in computing suspicious sources and propose BLMu, a novel MBFL approach that treats mutants differently. We also propose novel metrics for MBFL by considering the high cost of mutation analysis. We perform large‐scale experiments by evaluating all the MBFL approaches against 395 real‐world Java bugs of the Defects4J benchmark. Our evaluation results reveal that BLMu demonstrates a substantial improvement over both MUSE and Metallaxis, the most popular MBFL approaches, at both the statement level and the method level. Specifically, in terms of Top‐1, BLMu improves by 106% at the statement level and 69% at the method level. When compared with other types of FL approaches, MBFL outperforms typical SBFL approaches while still far behind the state‐of‐the‐art learning‐based FL approaches.

Optimizing Mutation-based Fault Localization through Contribution-based Test Case Reduction

Optimizing Mutation-based Fault Localization through Contribution-based Test Case Reduction

Delta4Ms: Improving mutation‐based fault localization by eliminating mutant bias

AbstractFault localization is a complex, costly and time‐consuming task in software debugging. Numerous automated techniques have been developed to expedite this process. Mutation‐based fault localization (MBFL) is one of the most widely studied techniques which uses mutation analysis to generate mutants for revealing potential faults in the program. However, our theoretical analysis exposes an inherent conflict between the fundamental assumption and the essential meaning of existing MBFL suspiciousness. This conflict is caused by mutant bias. Intuitively, the suspiciousness can be corrected by eliminating the mutant bias for more accurately measuring the faulty probability of the corresponding mutant statement. In this paper, we introduce Delta4Ms, a fault localization approach designed to eliminate mutant bias. Delta4Ms integrates the principles of signal theory, modelling the actual suspiciousness and mutant bias as the desired and false signal components, respectively. Based on theoretical derivation, the average suspiciousness of mutants serves as an estimate of mutant bias. Delta4Ms effectively mitigates mutant bias, extracting the desired signal and yielding corrected suspiciousness for fault localization. To precisely estimate mutant bias, higher order mutants (HOMs) are incorporated. We conduct an extensive experimental evaluation of Delta4Ms on 320 real‐fault programs from Codeflaws. The results indicate that our model significantly outperforms existing SBFL and MBFL techniques, showing a considerable improvement in fault localization effectiveness. We further assessed the robustness of Delta4Ms by examining different HOM ratios and HOM generation strategies. Moreover, Delta4Ms achieves a substantial reduction in mutation execution cost and minimal accuracy loss through the implementation of test case reduction. Finally, we perform preliminary experiments on 15 real‐fault programs from the Defects4J benchmark to assess the generalization of the model's fault localization effectiveness.

An effective fault localization approach based on PageRank and mutation analysis

Mutation-based fault localization (MBFL) is a popular method based on mutation testing. MBFL applies a variety of operators to generate mutants and calculates the statement’s suspiciousness by counting the execution results of the test cases on the mutants. However, the tie problem of MBFL creates obstacles to accurate fault localization. The tie problem refers to that many statements have the same suspiciousness. To solve the tie problem, we propose a fault localization approach based on the PageRank algorithm and mutation analysis (PRMA). We first apply the PageRank algorithm to calculate the faultiness scores of the statements. Then, we weight the suspicious value of the statements with faultiness scores to solve the tie problem. Finally, the weighted suspicious values are sorted in descending order to generate a list, which is provided to developers for fault localization. To evaluate our approach, we conduct experiments on the real fault benchmark Defects4J and the artificial fault dataset Siemens. We compare PRMA with the traditional MBFL techniques (Metallaxis and MUSE) and recently proposed MBFL methods (MCBFL-hybrid-avg, SMFL and SMBFL). The experimental results show that our approach outperforms above comparison methods and improves the effect of fault localization in both quantity and accuracy.

Can Higher-Order Mutants Improve the Performance of Mutation-Based Fault Localization?

First-order mutants (FOMs) have been widely used in mutation-based fault localization (MBFL) approaches and have achieved promising results in single-fault localization scenarios (SFL-scenario). Higher-order mutants (HOMs) are proposed to simulate complex faults and can be applied in MBFL theoretically for multiple-fault localization scenarios (MFL-scenario). However, whether HOMs can improve MBFL’s performance is not investigated and the effectiveness is not thoroughly evaluated. In this empirical study, we investigate the impact of HOMs on the performance of MBFL in SFL-scenario and MFL-scenario. The experiments on two real-world benchmarks reveal that 1) 2-HOMs can help improve the MBFL performance in SFL-scenarios; 2) in MFL-scenarios, both 2-HOMs and 3-HOMs can achieve better performance than FOMs; and 3) huge computational cost cannot be ignored in the practice of HOMs. Therefore, effective methods to reduce the number of HOMs for future MBFL studies should be considered.

An Empirical Study on Higher-Order Mutation-Based Fault Localization

Fault localization is one of the most expensive activities in software debugging. Mutation-based fault localization (MBFL) is a commonly studied technique that applied mutation analysis to find the location of faults in the programs. Previous studies showed that MBFL adopted First-Order-Mutants (FOMs) that could achieve promising results in single-fault localization, but it did not perform well in multiple-fault localization. Recently, Higher-Order-Mutants (HOMs) were proposed for modeling complex faults but whether HOMs can help in fault localization is still unknown. In this paper, we investigate the performance of MBFL with FOMs and HOMs on single- and multiple-fault localization. Moreover, to study the characteristics of HOMs, we divide HOMs into three groups (i.e. Accurate HOMs, Partially accurate HOMs, and Inaccurate HOMs) by considering different mutation locations. Based on the empirical results on 186 versions of six real-world programs, we find that (1) In single-fault localization, FOMs can achieve better performance than HOMs. (2) However, in multiple-fault localization, HOMs (2-HOMs) localize more faults than FOMs. (3) Furthermore, different types of HOMs have different fault localization effectiveness, where Accurate HOMs outperform the other two HOMs categories. Therefore, the researchers should propose methods to find HOMs more useful for fault localization.

HOTFUZ: Cost‐effective higher‐order mutation‐based fault localization

AbstractFault localization techniques are used to deduce the exact source of a failure from a set of failure indications while debugging software and play a crucial role in improving software quality. Mutation‐based fault localization (MBFL) techniques are proposed to localize faults at a finer granularity and with higher accuracy than traditional fault localization techniques. Despite the technique's effectiveness, the immense cost of mutation analysis hinders MBFL's practical application in the industry. Various mutation alternative strategies are utilized to lower the cost of MBFL, but they sacrifice the accuracy of localization results. Higher‐order mutation testing was proposed to search for valuable mutants that drive testing harder and reduce the overall test effort. However, higher‐order mutants (HOMs) never have been used to address the cost problem of MBFL to the extent of our knowledge. This paper proposes a novel, cost‐effective MBFL technique called HOTFUZ, Higher‐Order muTation‐based FaUlt localiZation, that employs HOMs to reduce the cost while minimizing the accuracy degradation. HOTFUZ combines mutants of a program under test into HOMs to decrease the number of mutants by more than half, depending on the order of HOMs. An experimental study is conducted using 65 real‐world faults of CoREBench to assess the proposed approach's cost‐effectiveness. The experimental results show that HOTFUZ outperforms the extant mutation alternative strategies by localizing faults more accurately using the same number of mutants executed. HOTFUZ has three main benefits over existing mutant reduction techniques for MBFL: (a) It keeps the advantage of using the whole set of mutation operators; (b) it does not discard generated mutants randomly for the sake of efficiency; and, finally, (c) it significantly decreases the proportion of equivalent mutants.

MuSim: Mutation-based Fault Localization Using Test Case Proximity

Fault localization techniques aim to localize faulty statements using the information gathered from both passed and failed test cases. We present a mutation-based fault localization technique called MuSim. MuSim identifies the faulty statement based on its computed proximity to different mutants. We study the performance of MuSim by using four different similarity metrics. To satisfactorily measure the effectiveness of our proposed approach, we present a new evaluation metric called Mut_Score. Based on this metric, on an average, MuSim is 33.21% more effective than existing fault localization techniques such as DStar, Tarantula, Crosstab, Ochiai.

커버리지 변화 정보를 활용한 코드 누락 결함에 대한 뮤테이션 기반 결함 위치 식별 기법의 성능 향상

많은 소프트웨어 결함이 코드 누락 결함(omission fault)의 형태를 가지는 데 반하여, 기존의 뮤테이션 기반 결함 위치 식별(mutation-based fault localization) 기법은 코드 누락 결함에 대해 낮은 정확도를 보여 효용성이 낮은 문제가 있다. 본 논문에서는 뮤테이션 기반 결함 위치 식별 기법인 MUSEUM과 Metallaxis에 뮤테이션에 따른 커버리지 변화와 테스트 결과 또는 출력을 연계하는 요소를 추가함으로써 코드 누락 결함을 대상으로 위치 식별 성능을 향상하는 MUSEUM+과 Metallaxis+ 기법을 제안한다. 10개의 코드 누락 결함과 6개의 일반 결함을 포함하는 총 16개 Defects4J 결함사례를 대상으로 실험한 결과, 제안한 MUSEUM+과 Metallaxis+기법은 총 10개의 코드 누락 결함 중 6개에 대해 효과적으로 정확도를 향상시켰으며, 16개 전반에 있어 정확도 향상을 확인할 수 있었다.

SMBFL: slice-based cost reduction of mutation-based fault localization

Fault localization is one of the most important and difficult tasks in the software debugging process. Therefore, several methods have been proposed to automate and improve this process. Mutation-based fault localization is one of the states of the art techniques that try to locate faults by executing different mutants of the faulty program. In addition to favorable results, it is along with a massive increase in mutation execution cost. In this paper, we propose a new mutation-based fault localization approach called SMBFL, that aim to reduce the execution cost by reducing the number of statements to be mutated. As fewer mutants execute with SMBFL, the whole process will become faster and the cost will decrease. SMBFL only examines the statements in the dynamic slice of the program under test. The statements that present in the dynamic slice have a direct effect on the execution of the program with the specified test case. In the SMBFL method, the suspiciousness score of program statements is measured based on the entropy of their mutants. The proposed formula, MuEn, determines the suspiciousness score based on the result of executing mutants of each statement of the program. SMBFL is evaluated during a series of tests. The results show a relative increase in the accuracy of fault localization, by an average of 14.2%, and a decrease in the execution time of the fault localization process, by an average of 24.3%. Finally, the MuEn formula applies the least execution overhead to the fault localization process.

HMER: A Hybrid Mutation Execution Reduction approach for Mutation-based Fault Localization

Identifying the location of faults in programs has been recognized as one of the most manually and time cost activities during software debugging process. Fault localization techniques, which seek to identify faulty program statements as quickly as possible, can assist developers in alleviating the time and manual cost of software debugging. Mutation-based fault localization(MBFL) has a promising fault localization accuracy, but suffered from huge mutation execution cost. To reduce the cost of MBFL, we propose a Hybrid Mutation Execution Reduction(HMER) approach in this paper. HMER consists of two steps: Weighted Statement-Oriented Mutant Sampling(WSOME) and Dynamic Mutation Execution Strategy(DMES). In the first step, we employ Spectrum-Based Fault Localization(SBFL) techniques to calculate the suspiciousness value of statements, and guarantee that the mutants generated from statements with higher suspiciousness value will have more chance to be remained in the sampling process. Next, a dynamic mutation execution strategy is used to execute the reduced mutant set on test suite to avoid worthless execution. Empirical results on 130 versions from 9 subject programs show that HMER can reduce 74.5%-93.4% mutation execution cost while keeping almost the same fault localization accuracy with the original MBFL. A further Wilcoxonsigned−ranktest indicates that when employing HMER strategy in MBFL, the fault localization accuracy has no statistically significant difference in most cases compared with the original MBFL without any reduction techniques.

IETCR: An Information Entropy Based Test Case Reduction Strategy for Mutation-Based Fault Localization

Mutation-based fault localization (MBFL) is a recently proposed technique with the advantage of high fault localization accuracy. However, such a mutation analysis based technique is difficult to be accepted by industry due to its huge computational cost on mutation analysis. There are three ways to improve MBFL's efficiency, which are reducing the number of mutants, optimizing the mutants' execution process, and reducing the number of test cases. The former two ways have been mainly studied and shown promising results, but for the latter way, the related studies are limited since this kind of method will reduce the precision of MBFL. In this paper, we mainly focus on the latter way and propose an information entropy based test case reduction (IETCR) strategy for MBFL. In particular, we first calculate the entropy change of test cases and select a proportion of them according to their value. Then we use a reduced test suite to execute mutants. To show the effectiveness of the IETCR strategy, we choose six real-world programs with 112 faulty versions. In terms of mutation reduction rate, we find MBFL with the IETCR strategy can reduce 56.3%~88.3% cost while keeping almost the same fault localization accuracy when compared to the original MBFL without test case reduction. Moreover, we use Wilcoxon signed-rank test for statistical analysis, which shows that there is no statistically significant difference between MBFL with IETCR strategy and the original MBFL.

Improving Software Fault Localization by Combining Spectrum and Mutation

The performance of software fault localization techniques is critical to software debugging and the reliability of software. Spectrum-based fault localization (SBFL) and mutation-based fault localization (MBFL) are the two most popular fault localization methods. However, the accuracies of the two methods are still limited. For example, only 10.63% of faults can be detected by inspecting the top 3 suspicious elements reported by Ochiai, which is a famous SBFL technique. Unfortunately, programmers only examine the first few suspicious elements before losing patience. Since the information used in SBFL and MBFL are quite different and complementary, this paper proposes a novel approach by combining spectrum and mutation to improve the fault localization accuracy. First, the faulty program is evaluated by using SBFL, and the potential faulty statements are ranked according to their suspiciousness. Then, mutants of the program are generated and executed by MBFL. Finally, the statements that are ranked in the top tied ${n}$ by SBFL are evaluated and reranked according to their mutation scores. Experiments are carried on the Defects4J benchmark and the results reveal that the accuracy of the proposed approach outperforms those of the SBFL and MBFL techniques. In terms of the faults located by inspecting the top 1 suspicious elements, the SMFL techniques detect at least 2.36 times more faults than two SBFL techniques (DStar and Ochiai) and detect at least 1.86 times more faults than two MBFL techniques (MUSE and Metallaxis).

Precise Learn-to-Rank Fault Localization Using Dynamic and Static Features of Target Programs

Finding the root cause of a bug requires a significant effort from developers. Automated fault localization techniques seek to reduce this cost by computing the suspiciousness scores (i.e., the likelihood of program entities being faulty). Existing techniques have been developed by utilizing input features of specific types for the computation of suspiciousness scores, such as program spectrum or mutation analysis results. This article presents a novel learn-to-rank fault localization technique called PRecise machINe-learning-based fault loCalization tEchnique (PRINCE). PRINCE uses genetic programming (GP) to combine multiple sets of localization input features that have been studied separately until now. For dynamic features, PRINCE encompasses both Spectrum Based Fault Localization (SBFL) and Mutation Based Fault Localization (MBFL) techniques. It also uses static features, such as dependency information and structural complexity of program entities. All such information is used by GP to train a ranking model for fault localization. The empirical evaluation on 65 real-world faults from CoREBench, 84 artificial faults from SIR, and 310 real-world faults from Defects4J shows that PRINCE outperforms the state-of-the-art SBFL, MBFL, and learn-to-rank techniques significantly. PRINCE localizes a fault after reviewing 2.4% of the executed statements on average (4.2 and 3.0 times more precise than the best of the compared SBFL and MBFL techniques, respectively). Also, PRINCE ranks 52.9% of the target faults within the top ten suspicious statements.

Automatic debugging of operator errors based on efficient mutation analysis

It takes a lot of time and effort to manually locate and fix software bugs. This paper proposes a method for automatically debugging operator related bugs. Testing, fault localization, and bug-fixing are closely linked based on mutation analysis. However, in the process of mutation analysis, the generation of a large number of mutants and the execution of test cases on mutants, is fairly time-consuming. To solve this problem, optimization methods for selection of mutants and test cases have been proposed. Experiment results has shown that it can improve the efficiency of mutation analysis, so that the cost of fault-localization and bug-fixing can be reduced. We also implemented the exhaustive mutation method and the random mutation method and compared these three methods. These three method have different application scenarios. As the mutation based fault localization can rank statements by suspiciousness, the method integrated with fault localization is more stable and has batter performance. Also, it is more suitable for analyzing program with multi-bugs.

Transforming programs and tests in tandem for fault localization

Localizing failure-inducing code is essential for software debugging. Manual fault localization can be quite tedious, error-prone, and time-consuming. Therefore, a huge body of research e orts have been dedicated to automated fault localization. Spectrum-based fault localization, the most intensively studied fault localization approach based on test execution information, may have limited effectiveness, since a code element executed by a failed tests may not necessarily have impact on the test outcome and cause the test failure. To bridge the gap, mutation-based fault localization has been proposed to transform the programs under test to check the impact of each code element for better fault localization. However, there are limited studies on the effectiveness of mutation-based fault localization on sufficient number of real bugs. In this paper, we perform an extensive study to compare mutation-based fault localization techniques with various state-of-the-art spectrum-based fault localization techniques on 357 real bugs from the Defects4J benchmark suite. The study results firstly demonstrate the effectiveness of mutation-based fault localization, as well as revealing a number of guidelines for further improving mutation-based fault localization. Based on the learnt guidelines, we further transform test outputs/messages and test code to obtain various mutation information. Then, we propose TraPT, an automated Learning-to-Rank technique to fully explore the obtained mutation information for effective fault localization. The experimental results show that TraPT localizes 65.12% and 94.52% more bugs within Top-1 than state-of-the-art mutation and spectrum based techniques when using the default setting of LIBSVM.

An optimal mutation execution strategy for cost reduction of mutation-based fault localization

Identifying faulty program entities has been recognized as one of the most expensive, tedious and time-consuming processes in software debugging activity. Fault localization techniques are designed to assist developers in locating faults by giving a ranking of the probability that program entities incur failures. Mutation-based fault localization (MBFL) is a recently proposed fault localization approach via mutation analysis, which uses the location of mutants to identify the faulty statements. With improved effectiveness, the MBFL also brings huge execution cost. To reduce the execution cost of MBFL technique, this paper proposes a dynamic mutation execution strategy (DMES) to prioritize the execution on both mutants and test cases. As fewer mutants and test cases are executed with DMES, the whole process will become faster and the cost will be decreased. At the same time, it is proved that the fault localization accuracy of MBFL with DMES is the same as that of the original MBFL. Furthermore, this paper discusses the lowest mutation execution cost of MBFL in theory and gives a quick solution to compute the theoretical minimal mutation execution cost of MBFL in the case of keeping fault localization accuracy non-decreasing. The empirical studies show that MBFL with the dynamic strategy proposed in this paper can reduce mutant-test execution cost by 32.4–87% and is close to the theoretical optimal cost. Furthermore, the additional run time required by utilizing our strategy is minimum and can be ignored.

MUSEUM: Debugging real-world multilingual programs using mutation analysis

Context: The programming language ecosystem has diversified over the last few decades. Non-trivial programs are likely to be written in more than a single language to take advantage of various control/data abstractions and legacy libraries.Objective: Debugging multilingual bugs is challenging because language interfaces are difficult to use correctly and the scope of fault localization goes beyond language boundaries. To locate the causes of real-world multilingual bugs, this article proposes a mutation-based fault localization technique (MUSEUM).Method: MUSEUM modifies a buggy program systematically with our new mutation operators as well as conventional mutation operators, observes the dynamic behavioral changes in a test suite, and reports suspicious statements. To reduce the analysis cost, MUSEUM selects a subset of mutated programs and test cases.Results: Our empirical evaluation shows that MUSEUM is (i) effective: it identifies the buggy statements as the most suspicious statements for both resolved and unresolved non-trivial bugs in real-world multilingual programming projects; and (ii) efficient: it locates the buggy statements in modest amount of time using multiple machines in parallel. Also, by applying selective mutation analysis (i.e., selecting subsets of mutants and test cases to use), MUSEUM achieves significant speedup with marginal accuracy loss compared to the full mutation analysis.Conclusion:It is concluded that MUSEUM locates real-world multilingual bugs accurately. This result shows that mutation analysis can provide an effective, efficient, and language semantics agnostic analysis on multilingual code. Our light-weight analysis approach would play important roles as programmers write and debug large and complex programs in diverse programming languages.

FBD 프로그램 뮤테이션 기반 오류 위치 추정 기법 적용 사례연구

Finding the exact location of faults in a program requires enormous time and effort. Several fault localization methods based on control flows of a program have been studied for decades. Unfortunately, these methods are not applicable to programs based on data-flow languages. A recently proposed mutation-based fault localization method is applicable to data-flow languages, as well as control-flow languages. However, there are no studies on the effectiveness of the mutation-based fault localization method for data-flow based programs. In this paper, we provided an experimental case study to evaluate the effectiveness of mutation-based fault localization on programs implemented in Function Block Diagram (FBD), a widely used data-flow based language in safety-critical systems implementation. We analyzed several real faults in the implementation of FBD programs of a nuclear reactor protection system, and evaluated the mutation-based fault localization effectiveness for each fault.