Effectiveness Of Test Suite Research Articles

Subsumption, correctness and relative correctness: Implications for software testing

Context. Several Research areas emerged and have been proceeding independently when in fact they have much in common. These include: mutant subsumption and mutant set minimization; relative correctness and the semantic definition of faults; differentiator sets and their application to test diversity; generate-and–validate methods of program repair; test suite coverage metrics.Objective. Highlight their analogies, commonalities and overlaps; explore their potential for synergy and shared research goals; unify several disparate concepts around a minimal set of artifacts.Method. Introduce and analyze a minimal set of concepts that enable us to model these disparate research efforts, and explore how these models may enable us to share insights between different research directions, and advance their respective goals.Results. Capturing absolute (total and partial) correctness and relative (total and partial) correctness with a single concept: detector sets. Using the same concept to quantify the effectiveness of test suites, and prove that the proposed measure satisfies appealing monotonicity properties. Using the measure of test suite effectiveness to model mutant set minimization as an optimization problem, characterized by an objective function and a constraint.Generalizing the concept of mutant subsumption using the concept of differentiator sets. Identifying analogies between detector sets and differentiator sets, and inferring relationships between subsumption and relative correctness.Conclusion. This paper does not aim to answer any pressing research question as much as it aims to raise research questions that use the insights gained from one research venue to gain a fresh perspective on a related research issue.

EFSM Model-Based Testing for Android Applications

Model-based testing provides an effective means for ensuring the quality of Android apps. Nevertheless, existing models that focus on event sequences and abstract them into Finite State Machines (FSMs) may lack precision and determinism because of the different data values of events that can result in various states of Android applications. To address this issue, a novel model based on Extended Finite State Machines (EFSMs) for Android apps is proposed in this paper. The approach leverages machine learning to infer data constraints on events and annotates them on state transitions, leading to a more precise and deterministic model. Additionally, a state abstraction strategy is presented to further refine the model. Besides, test diversity plays a vital role in enhancing test suite effectiveness. To achieve high coverage and fault detection, test cases are generated from the EFSM model with the help of a Genetic Algorithm (GA), guided by test diversity. To evaluate the effectiveness of our approach, this paper carries out experiments on 93 open-source apps. The results show that our approach performs better in code coverage and crash detection than the existing open-source model-based testing tools. Particularly, the 19 unique crashes that involve complex data constraints are detected by our approach.

Assessing effectiveness of test suites: what do we know and what should we do?

Background. Software testing is a critical activity for ensuring the quality and reliability of software systems. To evaluate the effectiveness of different test suites, researchers have developed a variety of metrics. Problem. However, comparing these metrics is challenging due to the lack of a standardized evaluation framework including comprehensive factors. As a result, researchers often focus on single factors (e.g., size), which finally leads to different or even contradictory conclusions. After comparing dozens of pieces of work in detail, we have found two main problems most troubling to our community: (1) researchers tend to oversimplify the description of the ground truth they use, and (2) data involving real defects is not suitable for analysis using traditional statistical indicators. Objective. We aim at scrutinizing the whole process of comparing test suites for our community. Method. To hit this aim, we propose a framework ASSENT (ev A luating te S t S uite E ffective N ess me T rics) to guide the follow-up research for evaluating a test suite effectiveness metric. ASSENT consists of three fundamental components: ground truth, benchmark test suites, and agreement indicator. Its functioning is as follows: first, users clarify the ground truth for determining the real order in effectiveness among test suites. Second, users generate a set of benchmark test suites and derive their ground truth order in effectiveness. Third, users use the metric to derive the order in effectiveness for the same test suites. Finally, users calculate the agreement indicator between the two orders derived by two metrics. Result. With ASSENT, we are able to compare the accuracy of different test suite effectiveness metrics. We apply ASSENT to evaluate representative test suite effectiveness metrics, including mutation score and code coverage metrics. Our results show that, based on the real faults, mutation score and subsuming mutation score are the best metrics to quantify test suite effectiveness. Meanwhile, by using mutants instead of real faults, test effectiveness will be overestimated by more than 20% in values. Conclusion. We recommend that the standardized evaluation framework ASSENT should be used for evaluating and comparing test effectiveness metrics in the future work.

Route : Roads Not Taken in UI Testing

Core features (functionalities) of an app can often be accessed and invoked in several ways, i.e., through alternative sequences of user-interface (UI) interactions. Given the manual effort of writing tests, developers often only consider the typical way of invoking features when creating the tests (i.e., the “sunny day scenario”). However, the alternative ways of invoking a feature are as likely to be faulty. These faults would go undetected without proper tests. To reduce the manual effort of creating UI tests and help developers more thoroughly examine the features of apps, we present Route , an automated tool for feature-based UI test augmentation for Android apps. Route first takes a UI test and the app under test as input. It then applies novel heuristics to find additional high-quality UI tests, consisting of both inputs and assertions, that verify the same feature as the original test in alternative ways. Application of Route on several dozen tests for popular apps on Google Play shows that for 96% of the existing tests, Route was able to generate at least one alternative test. Moreover, the fault detection effectiveness of augmented test suites in our experiments showed substantial improvements of up to 39% over the original test suites.

Software Refactoring Prediction Using SVM and Optimization Algorithms

Test suite code coverage is often used as an indicator for test suite capability in detecting faults. However, earlier studies that have explored the correlation between code coverage and test suite effectiveness have not addressed this correlation evolutionally. Moreover, some of these works have only addressed small sized systems, or systems from the same domain, which makes the result generalization process unclear for other domain systems. Software refactoring promotes a positive consequence in terms of software maintainability and understandability. It aims to enhance the software quality by modifying the internal structure of systems without affecting their external behavior. However, identifying the refactoring needs and which level should be executed is still a big challenge to software developers. In this paper, the authors explore the effectiveness of employing a support vector machine along with two optimization algorithms to predict software refactoring at the class level. In particular, the SVM was trained in genetic and whale algorithms. A well-known dataset belonging to open-source software systems (i.e., ANTLR4, JUnit, MapDB, and McMMO) was used in this study. All experiments achieved a promising accuracy rate range of between 84% for the SVM–Junit system and 93% for McMMO − GA + Whale + SVM. It was clear that added value was gained from merging the SVM with two optimization algorithms. All experiments achieved a promising F-measure range between the SVM–Antlr4 system’s result of 86% and that of the McMMO − GA + Whale + SVM system at 96%. Moreover, the results of the proposed approach were compared with the results from four well known ML algorithms (NB-Naïve, IBK-Instance, RT-Random Tree, and RF-Random Forest). The results from the proposed approach outperformed the prediction performances of the studied MLs.

Mutant Reduction Evaluation: What is There and What is Missing?

Background. Mutation testing is a commonly used defect injection technique for evaluating the effectiveness of a test suite. However, it is usually computationally expensive. Therefore, many mutation reduction strategies, which aim to reduce the number of mutants, have been proposed. Problem. It is important to measure the ability of a mutation reduction strategy to maintain test suite effectiveness evaluation. However, existing evaluation indicators are unable to measure the “order-preserving ability”, i.e., to what extent the mutation score order among test suites is maintained before and after mutation reduction. As a result, misleading conclusions can be achieved when using existing indicators to evaluate the reduction effectiveness. Objective. We aim to propose evaluation indicators to measure the “order-preserving ability” of a mutation reduction strategy, which is important but missing in our community. Method. Given a test suite on a Software Under Test (SUT) with a set of original mutants, we leverage the test suite to generate a group of test suites that have a partial order relationship in defect detecting ability. When evaluating a reduction strategy, we first construct two partial order relationships among the generated test suites in terms of mutation score, one with the original mutants and another with the reduced mutants. Then, we measure the extent to which the partial order under the original mutants remains unchanged in the partial order under the reduced mutants. The more partial order is unchanged, the stronger the Order Preservation ( OP ) of the mutation reduction strategy is, and the more effective the reduction strategy is. Furthermore, we propose Effort-aware Relative Order Preservation ( EROP ) to measure how much gain a mutation reduction strategy can provide compared with a random reduction strategy. Result. The experimental results show that OP and EROP are able to efficiently measure the “order-preserving ability” of a mutation reduction strategy. As a result, they have a better ability to distinguish various mutation reduction strategies compared with the existing evaluation indicators. In addition, we find that Subsuming Mutant Selection (SMS) and Clustering Mutant Selection (CMS) are more effective than the other strategies under OP and EROP. Conclusion. We suggest, for the researchers, that OP and EROP should be used to measure the effectiveness of a mutant reduction strategy, and for the practitioners, that SMS and CMS should be given priority in practice.

Can we trust tests to automate dependency updates? A case study of Java Projects

Developers are increasingly using services such as Dependabot to automate dependency updates. However, recent research has shown that developers perceive such services as unreliable, as they heavily rely on test coverage to detect conflicts in updates. To understand the prevalence of tests exercising dependencies, we calculate the test coverage of direct and indirect uses of dependencies in 521 well-tested Java projects. We find that tests only cover 58% of direct and 21% of transitive dependency calls. By creating 1,122,420 artificial updates with simple faults covering all dependency usages in 262 projects, we measure the effectiveness of test suites in detecting semantic faults in dependencies; we find that tests can only detect 47% of direct and 35% of indirect artificial faults on average. To increase reliability, we investigate the use of change impact analysis as a means of reducing false negatives; on average, our tool can uncover 74% of injected faults in direct dependencies and 64% for transitive dependencies, nearly two times more than test suites. We then apply our tool in 22 real-world dependency updates, where it identifies three semantically conflicting cases and three cases of unused dependencies that tests were unable to detect. Our findings indicate that the combination of static and dynamic analysis should be a requirement for future dependency updating systems.

Statement frequency coverage: A code coverage criterion for assessing test suite effectiveness

Abstract Context: Software testing is a pivotal activity in the development of high-quality software. As software is evolving through its life cycle, the need for a fault-revealing criterion assessing the effectiveness of the test suite grows. Over the years, researchers have proposed coverage-based criteria, including statement and branch coverage, to solve this issue. In literature, the effectiveness of such criteria is attested in terms of their correlations with the mutation score. Objective: In this paper, we aim at proposing a coverage-based criterion named statement frequency coverage, which outperforms statement and branch coverage in terms of correlation with mutation score. Method: To this end, we incorporated the frequency of executed statements into statement coverage and created a coverage-based criterion for assessing test suite effectiveness. Statement frequency coverage assigns a continuous value to a statement whose value is proportional to the number of times executed during test execution. We evaluated our approach on 22 real-world Python projects with more than 118 000 source lines of code (without blank lines, comments, and test cases) and 21 000 test cases through measuring the correlation between statement frequency coverage and corresponding mutation score. Results: The results show that statement frequency coverage outperforms statement and branch coverage criteria. The correlation between it and the corresponding mutation score is higher than the correlation of statement and branch coverage with their mutation score. The results also show that unlike statement and branch coverage, there is no statistical difference between statement frequency coverage and mutation score. Conclusion: Statement frequency coverage is a better choice compared to statement and branch coverage in assessing test suite effectiveness in the real-world setting. Furthermore, we demonstrate that although statement frequency coverage subsumes statement coverage, it is incomparable to branch coverage under the adequate test suite condition.

Test suites effectiveness evolution in open source systems: empirical study

<span>Test suite code coverage is usually used to indicate the capability of a test suite in detecting faults. Earlier research studies, which explored the relationship among test suite effectiveness and code coverage, have not addressed this relationship evolutionally. Moreover, some of these works were studied small or identical domain systems, which make the result generalization process unclear for other systems. Finally, some of these studies were conducted with automatically generated test suites, which might not present the real situation for studied systems, so the results cannot be generalized to real test suites. In this paper, the authors empirically explore three open-source software systems along with their 11 versions. These versions are evolved over time and might have more sources of code and test suites. This work objective is to study the correlation between test suite effectiveness, the size of the test suite, and coverage for three Java programs during their evolution. In this work, the code coverage, test suite LOC and mutation testing coverage are measured to assess the correlation between the effectiveness of fault detection, code coverage, and test suite size. Based on the result we cannot generalize the assumption that test size is always revealing a positive correlation with its effectiveness, but still weak to the high correlation between test effectiveness, test size, and coverage.</span><p> </p>

CBUA: A Probabilistic, Predictive, and Practical Approach for Evaluating Test Suite Effectiveness

Knowing the effectiveness of a test suite is essential for many activities such as guiding the generation of new test cases and assessing the test adequacy of code. Mutation testing is a commonly used defect injection technique for evaluating the effectiveness of a test suite. However, it is usually computationally expensive, as a large number of mutants (buggy versions) are needed to be generated from a production code under test and executed against the test suite. In order to reduce the expensive testing cost, recent studies proposed to use supervised models to predict the effectiveness of a test suite without executing the test suite against the mutants. Nonetheless, the training of such a supervised model requires labeled data, which still depends on the costly mutant execution. Furthermore, existing models are based on traditional supervised learning techniques, which assumes that the training and testing data come from the same distribution. But, in practice, software systems are subject to considerable concept drifts, i.e. the same distribution assumption usually does not hold. This can lead to inaccurate predictions of a learned supervised model on the target code as time progresses. To tackle these problems, in this paper, we propose a Coverage-Based Unsupervised Approach (CBUA) for evaluating the effectiveness of a test suite. The whole process only requires a one-time execution of the test suite against the target production code, without involving any mutant execution and any training data. CBUA can ensure the score monotonicity property (i.e. adding test cases to a test suite does not decrease its mutation score), which may be violated by a supervised approach. The experimental results show that CBUA is very competitive to the state-of-the-art supervised approaches in terms of the prediction accuracy. Since CBUA is an easy-to-implement model with a low cost, we suggest that it should be used as a baseline approach for comparison when any novel prediction approach is proposed in future studies.

Diversity-Oriented Test Suite Generation for EFSM Model

In this article, test diversity has been suggested to be a valid way to improve test suite effectiveness. Extended finite state machine (EFSM) is a widely used formal model, but little attention is paid on the test suite generation with more diversity. EFSM test suite generation involves test paths generation and test data generation. Considering the discrepancy between test paths has a more crucial impact on the diversity of test suite, compared with the difference between test data, this article, therefore, mainly concerns the test paths generation with more diversity for EFSM models. Hence, the factors that influence the discrepancy between test paths are investigated. Then based on these factors, an integrated distance metric is designed to evaluate the dissimilarity between test paths, and a diversity measurement for EFSM test suite is presented. Furthermore, a diversity-oriented test suite generation (DOTSG) method is proposed where a dissimilarity-based fitness function and diversity-oriented update strategy are adopted in traditional coverage-oriented EFSM test suite generation (COTSG) by genetic algorithm. The experimental results show that, compared to COTSG, our DOTSG can not only generate more diverse test suite to satisfy a certain coverage criteria, improving the fault detection capability of the test suite, but also decrease the evolution time cost and the size of test suite generated.

Empirical Assessment of Multimorphic Testing

The performance of software systems such as speed, memory usage, correct identification rate, tends to be an evermore important concern, often nowadays on par with functional correctness for critical systems. Systematically testing these performance concerns is however extremely difficult, in particular because there exists no theory underpinning the evaluation of a performance test suite, i.e., to tell the software developer whether such a test suite is ”good enough” or even whether a test suite is better than another one. This paper proposes to apply Multimorphic testing and empirically assess the effectiveness of performance test suites of software systems coming from various domains. By analogy with mutation testing, our core idea is to leverage the typical configurability of these systems, and to check whether it makes any difference in the outcome of the tests: i.e., are some tests able to “kill” underperforming system configurations? More precisely, we propose a framework for defining and evaluating the coverage of a test suite with respect to a quantitative property of interest. Such properties can be the execution time, the memory usage or the success rate in tasks performed by a software system. This framework can be used to assess whether a new test case is worth adding to a test suite or to select an optimal test suite with respect to a property of interest. We evaluate several aspects of our proposal through 3 empirical studies carried out in different fields: object tracking in videos, object recognition in images, and code generators.

An Efficient Heuristic Based Test Suite Minimization Approach

Objectives: Development of an efficient test suite minimization approach in order to reduce the size of a previously acquired test suite and produce a new representative suite which will guarantee the same requirement coverage that was achieved before minimization for an effective and efficient regression testing. Method: Test suite minimizations techniques try to reduce the size and redundancy of test suite by removing certain test cases since requirement covered by them are already covered by other test cases. But, it has been found that the acquired test cases after minimization severely lacks ability to achieve the desirable code coverage because the minimization was done based on a single test adequacy criteria. In this paper, we propose an efficient heuristic based test suite minimization algorithm which will reduce the size of the test suites with respective to multiple test adequacy criterions in order to preserve the fault detection effectiveness and code coverage characteristics of the final test suite. Findings: Our experimental results indicate that a significant percentage of reduction in the test suite size is achieved when the minimization is performed with respect to multiple test adequacy criterions. Our approach is unique compared to the existing approaches in the sense that, we carried out minimization based on multiple test adequacy criterions while most of the existing approaches usually take one or two criterions into consideration. The proposed approach is evaluated based on two well known software testing metrics; one indicate the percentage of reduction in test suite size and the second one indicate the percentage of code coverage achieved by the minimized test suite. Our experimental results indicate that a significant percentage of reduction in the size as well as significant code coverage characteristics is achieved when the minimization is done according to the proposed approach. Improvements: The important contribution of this study is that, it presents a novel and efficient test suite minimization technique that optimizes the test suite size based on multiple adequacy criterions.

Memory mutation testing

ContextThree decades of mutation testing development have given software testers a rich set of mutation operators, yet relatively few operators can target memory faults (as we demonstrate in this paper).ObjectiveTo address this shortcoming, we introduce Memory Mutation Testing, proposing 9 Memory Mutation Operators each of which targets common forms of memory fault. We compare Memory Mutation Operators with traditional Mutation Operators, while handling equivalent and duplicate mutants.MethodWe extend our previous workshop paper, which introduced Memory Mutation Testing, with a more extensive and precise analysis of 18 open source programs, including 2 large real-world programs, all of which come with well-designed unit test suites. Specifically, our empirical study makes use of recent results on Trivial Compiler Equivalence (TCE) to identify both equivalent and duplicate mutants. Though the literature on mutation testing has previously deployed various techniques to cater for equivalent mutants, no previous study has catered for duplicate mutants.ResultsCatering for such extraneous mutants improves the precision with which claims about mutation scores can be interpreted. We also report the results of a new empirical study that compares Memory Mutation Testing with traditional Mutation Testing, providing evidence to support the claim that traditional mutation testing inadequately captures memory faults; 2% of the memory mutants are TCE-duplicates of traditional mutants and average test suite effectiveness drops by 44% when the target shifts from traditional mutants to memory mutants.ConclusionsIntroducing Memory Mutation Operators will cost only a small portion of the overall testing effort, yet generate higher quality mutants compared with traditional operators. Moreover, TCE technique does not only help with reducing testing effort, but also improves the precision of assessment on test quality, therefore should be considered in other Mutation Testing studies.

Achieving Effective Test Suites for Reactive Systems using Specification Mining and Test Suite Reduction Techniques

Failures in reactive embedded systems are often unacceptable. Moreover, effective testing of such systems to detect potential critical failures is a difficult task. We present an automated black box test suite generation technique for reactive systems. The technique is based on dynamic mining of specifications, in form of a finite state machine (FSM), from initial runs. The set of test cases thus produced contain several redundant test cases, many of which are eliminated by a simple greedy test suite reduction algorithm to give the final test suite. The effectiveness of tests generated by our technique was evaluated using five case studies from the reactive embedded domain. Results indicate that a test suite generated by our technique is promising in terms of effectiveness and scalability. While the test suite reduction algorithm removes redundant test cases, the change in effectiveness of test suites due to this reduction is examined in the experimentation. We present our specification mining based test suite generation technique, the test suite reduction technique and results on industrial case studies.

Improving the Effectiveness of Testing Pervasive Software via Context Diversity

Context-aware pervasive software is responsive to various contexts and their changes. A faulty implementation of the context-aware features may lead to unpredictable behavior with adverse effects. In software testing, one of the most important research issues is to determine the sufficiency of a test suite to verify the software under test. Existing adequacy criteria for testing traditional software, however, have not explored the dimension of serial test inputs and have not considered context changes when constructing test suites. In this article, we define the concept of context diversity to capture the extent of context changes in serial inputs and propose three strategies to study how context diversity may improve the effectiveness of the data-flow testing criteria. Our case study shows that the strategy that uses test cases with higher context diversity can significantly improve the effectiveness of existing data-flow testing criteria for context-aware pervasive software. In addition, test suites with higher context diversity are found to execute significantly longer paths, which may provide a clue that reveals why context diversity can contribute to the improvement of effectiveness of test suites.

Critical components testing using hybrid genetic algorithm

As quality of software plays a vital role in real time systems, it is essential to identify the crucial parts in the system and to test them effectively. In the proposed approach, the critical components are identified by means of mutation based impact analysis. The next task is to test the critical components using the Hybrid Genetic Algorithm (HGA) based test case generation and optimization approach. The mutants are automatically generated by seeding faults into each method of all the components in the Software Under Test (SUT). The initial set of test cases is generated using randomized test data. The generated test cases are executed over the original and the mutant to identify whether the test case detects the error or not. Based on the results, the Mutation Score (MS) is calculated, which always lies between 0 and 1. The best test cases are chosen based on having higher mutation scores and are executed on mutants to analyze how each component affects the other components in the SUT. Based on the analysis, the critical components are identified and they need rigorous testing using the test cases generated by the HGA. The algorithm uses the RemoveTop and LocalBest improvement heuristics to achieve near optimal solutions. In unit testing, the test cases are executed against the original and the mutant. The test case optimization is done by evaluating the effectiveness of test suites using the Mutation Score and the Branch Coverage Value (BCV). In pair-wise testing, the effective test cases are selected based on the higher mutation scores and branch coverage values. The components are executed against these test cases and the execution traces are recorded. The traced results are compared against the expected outputs which were previously stored in the repository and the statuses are updated. Based on the statuses, the faulty methods are revealed. The efficiency of the proposed approach is compared with Genetic Algorithm (GA) and we concluded that the final test suite size and the total execution time are reduced in the proposed approach. Finally various graphs and PDF reports are generated for visualization purposes.

On generating mutants for AspectJ programs

ContextMutation analysis has been widely used in research studies to evaluate the effectiveness of test suites and testing techniques. Faulty versions (i.e., mutants) of a program are generated such that each mutant contains one seeded fault. The mutation score provides a measure of effectiveness. ObjectiveWe study three problems with the use of mutation analysis for testing AspectJ programs:•The manual identification and removal of equivalent mutants is difficult and time consuming. We calculate the percentage of equivalent mutants generated for benchmark AspectJ programs using available mutation tools.•The generated mutants need to cover the various fault types described in the literature on fault models for AspectJ programs. We measure the distribution of the mutants generated using available mutation tools with respect to the AspectJ fault types.•We measure the difficulty of killing the generated mutants.We propose the use of simple analysis of the subject programs to prevent the generation of some equivalent mutants. MethodWe revised existing AspectJ fault models and presented a fault model that removes the problems in existing fault models, such as overlapping between fault types and missing fault types. We also defined three new fault types that occur due to incorrect data-flow interactions occurring in AspectJ programs. We used three mutation tools: AjMutator, Proteum/AJ, and MuJava on three AspectJ programs. To measure the difficulty of killing the mutants created using a mutation operator, we compared the average number of the mutants killed by 10 test suites that satisfy block coverage criterion. ResultsA high percentage of the mutants are equivalent. The mutation tools do not cover all the fault types. Only 4 out of 27 operators generated mutants that were easy to kill. ConclusionsOur analysis approach removed about 80% of the equivalent mutants. Higher order mutation is needed to cover all the fault types.

ASSESSING TEST SUITES FOR BUFFER OVERFLOW VULNERABILITIES

Over the last few years, numerous vulnerabilities have been reported in software, and successful exploitations of these vulnerabilities have resulted in severe consequences such as denial of services and application state corruptions. Researches have shown that effective quality assurance methods can prevent such consequences when applied during software (or applications) development processes. Software security testing is a popular assurance method in this direction. However, effective testing involves obtaining an effective test suite (or collection of test cases) that can reveal specific faults. Over the last few years, different testing approaches have been applied for revealing vulnerabilities in software. However, only few works have assessed the effectiveness of test suites for revealing vulnerabilities. We believe that bringing the idea of mutation-based assessment of test adequacy for vulnerabilities can help in detecting and removing vulnerabilities proactively. In this work, we apply mutation-based adequate testing for one of the worst vulnerabilities namely buffer overflow (BOF). We propose 16 mutation operators to force the generation of adequate test suites for BOF vulnerabilities. A prototype tool is developed to automatically generate mutants and perform mutation analysis with input test cases. The effectiveness of the operators is evaluated by using several benchmark programs having BOF vulnerabilities, and the results indicate that the proposed operators are effective for testing BOF vulnerabilities. Moreover, we present an analysis to find selective mutation operators for reducing the cost of mutation-based testing of BOF vulnerabilities.

Judy – a mutation testing tool for Java

Popular code coverage measures, such as branch coverage, are indicators of the thoroughness rather than the fault detection capability of test suites. Mutation testing is a fault-based technique that measures the effectiveness of test suites for fault localisation. Unfortunately, use of mutation testing in the software industry is rare because generating and running vast numbers of mutants against the test cases is time-consuming and difficult to do without an automated, fast and reliable tool. Our objective is to present an innovative approach to mutation testing that takes advantage of a novel aspect-oriented programming mechanism, called ‘pointcut and advice’, to avoid multiple compilation of mutants and, therefore, to speed up mutation testing. An empirical comparison of the performance of the developed tool, called Judy, with the MuJava mutation testing tool on 24 open-source projects demonstrates the value of the presented approach. The results show that there is a statistically significant (t(23)=−12.28, p < 0.0005, effect size d=3.43) difference in the number of mutants generated per second between MuJava (M=4.15, SD=1.42) and Judy (M=52.05, SD=19.69). Apart from being statistically significant, this effect is considered very large and, therefore, represents a substantive finding. This therefore allows us to estimate the fault detection effectiveness of test suites of much larger systems.