Mutation Testing Tool Research Articles

A Review on Tools, Mechanics, Benefits, and Challenges of Gamified Software Testing

Gamification is an established practice in Software Engineering to increase effectiveness and engagement in many practices. This manuscript provides a characterization of the application of gamification to the Software Testing area. Such practice in fact reportedly suffers from low engagement by both personnel in industrial contexts and learners in educational contexts. Our goal is to identify the application areas and utilized gamified techniques and mechanics, the provided benefits and drawbacks, as well as the open challenges in the field. To this purpose, we conducted a Multivocal Literature Review to identify white and grey literature sources addressing gamified software testing. We analyzed 73 contributions and summarized the most common gamified mechanics, concepts, tools, and domains where they are mostly applied. We conclude that gamification in software testing is mostly applied to the test creation phase with simple white-box unit or mutation testing tools and is mostly used to foster good behaviors by promoting the testers’ accomplishment. Key research areas and main challenges in the field are: careful design of tailored gamified mechanics for specific testing techniques; the need for technological improvements to enable crowdsourcing, cooperation, and concurrency; the necessity for empirical and large-scale evaluation of the benefits delivered by gamification mechanics.

Syntactic Versus Semantic Similarity of Artificial and Real Faults in Mutation Testing Studies

Fault seeding is typically used in empirical studies to evaluate and compare test techniques. Central to these techniques lies the hypothesis that artificially seeded faults involve some form of realistic properties and thus provide realistic experimental results. In an attempt to strengthen realism, a recent line of research uses machine learning techniques, such as deep learning and Natural Language Processing, to seed faults that look like (syntactically) real ones, implying that fault realism is related to syntactic similarity. This raises the question of whether seeding syntactically similar faults indeed results in semantically similar faults and, more generally whether syntactically dissimilar faults are far away (semantically) from the real ones. We answer this question by employing 4 state-of-the-art fault-seeding techniques (PiTest - a popular mutation testing tool, IBIR - a tool with manually crafted fault patterns, DeepMutation - a learning-based fault seeded framework and μBERT - a mutation testing tool based on the pre-trained language model CodeBERT) that operate in a fundamentally different way, and demonstrate that syntactic similarity does not reflect semantic similarity. We also show that 65.11%, 76.44%, 61.39% and 9.76% of the real faults of Defects4J V2 are semantically resembled by PiTest, IBIR, μBERT and DeepMutation faults, respectively.

ReSuMo: a regression strategy and tool for mutation testing of solidity smart contracts

ReSuMo: a regression strategy and tool for mutation testing of solidity smart contracts

Mutta: a novel tool for E2E web mutation testing

Mutation testing is an important technique able to evaluate the bug-detection effectiveness of existing software test suites. Mutation testing tools exist for several languages, e.g., Java and JavaScript, but no solutions are available for managing the mutation testing process for entire web applications, in the context of end-to-end (E2E) web testing. In this paper, we propose Mutta, a novel tool able to automate the entire mutation testing process. Mutta mutates the various server source files of the target web application, runs the E2E test suite against the mutated web applications, and finally collects the test outcomes. To evaluate Mutta, we designed a case study using the mutated versions of the target web application with the aim of comparing the effectiveness of two different approaches to E2E web testing: (1) test cases based on classical assertions and (2) test cases relying on differential testing. In detail, Mutta has been executed on two web applications, each equipped with different test suites to compare assertions with differential testing. In this scenario, Mutta generated a large number of mutants (more than 15k overall), took into account the coverage information to consider only the mutants actually executed, deployed the mutated web app, ran the entire E2E test suites (about 87k tests runs overall), and finally, it correctly saved the test suite results. Thus, results of the case study show that Mutta can be successfully employed to automate the entire mutation testing process of E2E web test suites and, therefore, can be used in practice to evaluate the effectiveness of different test suites (e.g., based on different techniques, E2E frameworks, or composed by a different number of test scripts).

Application of the Law of Minimum and Dissimilarity Analysis to Regression Test Case Prioritization

Regression testing is one of the most expensive processes in testing. Prioritizing test cases in regression testing is critical for the goal of detecting the faults sooner within a large set of test cases. We propose a test case prioritization (TCP) technique for regression testing called <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">LoM-Score</i> inspired by the Law of Minimum (LoM) from biology. This technique calculates the impact probabilities of methods calculated by change impact analysis with forward slicing and orders test cases according to LoM. However, this ordering doesn’t consider the possibility that consecutive test cases may be covering the same methods repeatedly. Thereby, such ordering can delay the time of revealing faults that exist in other methods. To solve this problem, we enhance the <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">LoM-Score</i> TCP technique with an adaptive approach, namely with a dissimilarity-based coordinate analysis approach. The dissimilarity-based coordinate analysis uses Jaccard Similarity for calculating the similarity coefficients between test cases in terms of covered methods and the enhanced technique called Dissimilarity-LoM-Score ( <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">Dis-LoM-Score</i> ) applies a penalty with respective on the ordered test cases. We performed our case study on 10 open-source Java projects from Defects4J, which is a dataset of real bugs and an infrastructure for controlled experiments provided for software engineering researchers. Then, we hand-seeded multiple mutants generated by Major, which is a mutation testing tool. Then we compared our TCP techniques <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">LoM-Score</i> and <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">Dis-LoM-Score</i> with the four traditional TCP techniques based on their Average Percentage of Faults Detected (APFD) results.

Investigation on Java Mutation Testing Tools

Software Testing is one of the most significant phases within the software development life cycle since software bugs can be costly and traumatic. However, the traditional software testing process is not enough on its own as some undiscovered faults might still exist due to the test cases’ inability to detect all underlying faults. Amidst the various proposed techniques of test suites’ efficiency detection comes mutation testing, one of the most effective approaches as declared by many researchers. Nevertheless, there is not enough research on how well the mutation testing tools adhere to the theory of mutation or how well their mutation operators are performing the tasks they were developed for. This research paper presents an investigative study on two different mutation testing tools for Java programming language, namely PIT and µJava. The study aims to point out the weaknesses and strengths of each tool involved through performing mutation testing on four different open-source Java programs to identify the best mutation tool among them. The study aims to further identify and compare the mutation operators of each tool by calculating the mutation score. That is, the operators’ performance is evaluated with the mutation score, with the presumption that the more prominent the number of killed mutants is, the higher the mutation score, thus the more effective the mutation operator and the affiliated tool.Â

SuMo: A mutation testing approach and tool for the Ethereum blockchain

Blockchain technologies have had a rather disruptive impact on many sectors of the contemporary society. The establishment of virtual currencies is probably the most representative case. Nonetheless, the inherent support to trustworthy electronic interactions has widened the possible adoption contexts. In the last years, the introduction of Smart Contracts has further increased the potential impact of such technologies. These self-enforcing programs have interesting peculiarities (e.g., code immutability) that require innovative testing strategies. This paper presents a mutation testing approach for assessing the quality of test suites accompanying Smart Contracts written in Solidity, the language used by the Ethereum Blockchain. Specifically, we propose a novel suite of mutation operators capable of simulating a wide variety of traditional programming errors and Solidity-specific faults. The operators come in two flavors: Optimized, for faster mutation testing campaigns, and Non-Optimized, for performing a more thorough adequacy assessment. We implemented our approach in a proof-of-concept work, SuMo (SOlidity MUtator), and we evaluated its effectiveness on a set of real-world Solidity projects. The experiments highlighted a recurrent low Mutation Score for the test suites shipped with the selected applications. Moreover, analyzing the surviving mutants of a selected project helped us to identify faulty test cases and Smart Contract code. These results suggest that SuMo can concretely improve the fault-detection capabilities of a test suite, and help to deliver more reliable Solidity code.

Mutation testing in the wild: findings from GitHub

Mutation testing exploits artificial faults to measure the adequacy of test suites and guide their improvement. It has become an extremely popular testing technique as evidenced by the vast literature, numerous tools, and research events on the topic. Previous survey papers have successfully compiled the state of research, its evolution, problems, and challenges. However, the use of mutation testing in practice is still largely unexplored. In this paper, we report the results of a thorough study on the use of mutation testing in GitHub projects. Specifically, we first performed a search for mutation testing tools, 127 in total, and we automatically searched the GitHub repositories including evidence of their use. Then, we focused on the top ten most widely used tools, based on the previous results, and manually revised and classified over 3.5K GitHub active repositories importing them. Among other findings, we observed a recent upturn in interest and activity, with Infection (PHP), PIT (Java) and Humbug (PHP) being the most widely used mutation tools in recent years. The predominant use of mutation testing is development, followed by teaching and learning, and research projects, although with significant differences among mutation tools found in the literature—less adopted and largely used in teaching and research—and those found in GitHub only—more popular and more widely used in development. Our work provides a new and encouraging perspective on the state of practice of mutation testing.

Mutation testing of PL/SQL programs

Mutation testing is a prominent technique for evaluating the effectiveness of a test suite. Existing tools developed for supporting this technique are applicable for mainstream programming languages like C and Java. Mutation testing tools and mutation operators used by these tools are inherently language-specific. Moreover, there is a lack of industrial case studies for evaluating mutation testing tools and techniques in practice. In this article, we introduce muPLSQL, a tool for applying mutation testing on PL/SQL programs, facilitating automation for both mutant generation and test execution. We utilized existing mutation operators that are applicable for PL/SQL. In addition, we introduced some operators specifically for this language. We conducted an industrial case study for evaluating the applicability and usefulness of our tool and mutation testing in general. We applied mutation testing on a business support software system. muPLSQL generated a total of 5,939 mutants. The number of live mutants was 680. Manual inspection of live mutants led to improvements of the existing test suite. In addition, we found 8 faults in source code during the inspection process. Test execution against the mutants required around 40 h. The overall effort was almost one person month.

Mutation testing with hyperproperties

We present a new method for model-based mutation-driven test case generation. Mutants are generated by making small syntactical modifications to the model or source code of the system under test. A test case kills a mutant if the behavior of the mutant deviates from the original system when running the test. In this work, we use hyperproperties—which allow to express relations between multiple executions—to formalize different notions of killing for both deterministic as well as non-deterministic models. The resulting hyperproperties are universal in the sense that they apply to arbitrary reactive models and mutants. Moreover, an off-the-shelf model checking tool for hyperproperties can be used to generate test cases. Furthermore, we propose solutions to overcome the limitations of current model checking tools via a model transformation and a bounded SMT encoding. We evaluate our approach on a number of models expressed in two different modeling languages by generating tests using a state-of-the-art mutation testing tool.

LCR based quick detection of hotspot G1896A mutation in patients with different spectrum of hepatitis B

G1896A switch is one of the hotspots in subjects affected with hepatitis B. This hotspot mutation is observed in all the different spectrum of hepatitis B, and it has a very dangerous and a long lasting effect. The major purpose of the study was to screen G1986A mutations at a large scale and also to establish ligase chain reaction as a mutation testing tool. Polymerase chain reaction (PCR) and Nucleotide Sequencing was done to identify the G1896A mutation in the precore region of the genome. All the 331 HBV positive patients were screened. Almost 29% (24/82) of the cases remarkably had the presence of G1896A mutation confirmed by LCR and direct sequencing.The precore G1896A mutation is responsible for one third of the patients suffering from precore stop codon mutation. It clearly exhibits that LCR is 100% in sync with direct sequencing and is extremely reliable and the results are highly reproducible.

Wodel-Test: a model-based framework for language-independent mutation testing

Mutation testing (MT) targets the assessment of test cases by measuring their efficiency to detect faults. This technique involves modifying the program under test to emulate programming faults, and assessing whether the existing test cases detect such mutations. MT has been extensively studied since the 70’s, and many tools have been proposed for widely used languages like C, Java, Fortran, Ada and SQL; and for notations like Petri-nets. However, building MT tools is costly and error-prone, which may prevent their development for new programming and domain-specific (modelling) languages. In this paper, we propose a framework called Wodel-Test to reduce the effort to create MT tools. For this purpose, it follows a model-driven approach by which MT tools are synthesized from a high-level description. This description makes use of the domain-specific language Wodel to define and execute model mutations. Wodel is language-independent, as it allows the creation of mutation operators for any language defined by a meta-model. Starting from the definition of the mutation operators, Wodel-Test generates a MT environment which parses the program under test into a model, applies the mutation operators, and evaluates the test-suite against the generated mutants, offering a rich collection of MT metrics. We report on an evaluation of the approach based on the creation of MT tools for Java and the Atlas transformation language.

Mutating code annotations: An empirical evaluation on Java and C# programs

• 100 new bugs regarding C# code annotations listed. • Evaluation on mutation operators validity for C# code annotations. • A mutation engine tool capable of targeting Java annotations. • A discussion over relationship between bugs found and code annotation use/design. • An evaluation of mutation operators applicability for Java and C# code annotations. • A GitHub repository with 200 bugs related to code annotations in Java and C#. • A mutation engine tool capable of targeting Java annotations. • A discussion over relationship between bugs found and code annotation use/design. Mutation testing injects code changes to check whether tests can detect them. Mutation testing tools use mutation operators that modify program elements such as operators, names, and entire statements. Most existing mutation operators focus on imperative and object-oriented language constructs. However, many current projects use meta-programming through code annotations. In a previous work, we have proposed nine mutation operators for code annotations focused on the Java programming language. In this article, we extend our previous work by mapping the operators to the C# language. Moreover, we enlarge the empirical evaluation. In particular, we mine Java and C# projects that make heavy use of annotations to identify annotation-related faults. We analyzed 200 faults and categorized them as “misuse,” when the developer did not appear to know how to use the code annotations properly, and “wrong annotation parsing” when the developer incorrectly parsed annotation code (by using reflection, for example). Our operators mimic 95% of the 200 mined faults. In particular, three operators can mimic 82% of the faults in Java projects and 84% of the faults in C# projects. In addition, we provide an extended and improved repository hosted on GitHub with the 200 code annotation faults we analyzed. We organize the repository according to the type of errors made by the programmers while dealing with code annotations, and to the mutation operator that can mimic the faults. Last but not least, we also provide a mutation engine, based on these operators, which is publicly available and can be incorporated into existing or new mutation tools. The engine works for Java and C#. As implications for practice, our operators can help developers to improve test suites and parsers of annotated code.

RegularMutator: A Mutation Testing Tool for Solidity Smart Contracts

Abstract With the growing popularity of smart contracts, the problem of validation of their correctness becomes more and more relevant, but at the moment there are no universally recognized tools for their testing. In this paper, we propose to apply the mutational analysis to improve reliability of Solidity smart contracts. We identified wide spread errors in the source code of existing contracts and developed a RegularMutator tool for mutation analysis. It has shown its effectiveness in testing a large smart contracts project. As a result of our analysis, we managed to improve the test suites of the project to find the discovered defects and, increase the quality of the test suite.

MUTWEB- A Testing Tool for performing Mutation Testing of Java and Servlet Based Web Applications

Mutation testing is one of the oldest and unique testing techniques to perform white box testing of software applications. Code coverage becoming an increasing concern in the testing cycle of software, mutation testing technique aids in achieving higher code coverage and unearthing more number of errors at the testing site itself. The parameters like the database connectivity, session management, cookie management, are the beginning point of web application testing failures given the heterogeneity aspects associated with the development of a web application. A detailed account on list of available testing tools for performing mutation testing are presented here. A big bundle of mutation testing tools are still available, however they are not focussing on some of the crucial web vulnerabilities like session and cookie management in web apps. In the current work, a tool to perform mutation testing of web applications is developed and tested to see if desired results are occurring. An architecture of the tool is designed is discussed and presented. A brief analysis on results is presented.

Abstract 3007: Validation of an NGS panel assay for detection of hotspot cancer somatic mutations

Abstract Background The gene tests to predict whether cancers respond to specific targeted therapies or prediction of prognosis has quickly increased over the last decade. Next-generation sequencing (NGS) has become a powerful tool for high-throughput and sensitive gene mutation testing, which is required for development of targeted therapies, clinical trials, and further research. The aim of this study was to validate the AmpliSeq for Illumina Cancer Hotspot Panel v2 on the Illumina MiSeq platform and simultaneously analyze 2800 somatic mutations across the hotspot regions of 50 genes, including SNP and small INDEL, with known associations to cancer, as identified in the Catalogue of Somatic Mutations in Cancer (COSMIC) database. This panel enables the study of the genes associated with different cancer types, including the cancers of the lung, colon, breast, thyroid, oral, bladder, prostate, liver, kidney, ovarian and cervix, etc. Methods and findings We validated this assay using the genomic DNA extracted from human paraffin-embedded (FFPE) samples and peripheral blood with negative control DNA (NA12878), positive control DNA (HD701), blank control as well as the global proficiency testing samples. Following two lab scientists parallel NGS library preparation, sequenced on Miseq platform (2x150 PE) and analyzed using a pipeline centered on Illumina Basespace, the somatic mutations were called at an average number of reads coverage &amp;gt;2500X, variant frequency cutoff 1%. An integrated test report can be generated within five working days. Panel sequencing analysis showed that the mutations are 100% concordance with known mutations and consistent in four runs, between lab personal performances, different input DNA concentrations (1ng-100ng), etc. Briefly, 25 mutations were detected in the FFPE samples and 20 mutations detected in proficiency testing samples. The variant types, HGVS ID and frequency are 100% match with its standard results which are a prove of precision and sensitivity. Conclusions A sensitive, high-throughput, pan-cancer mutation panel for NGS analysis of cancer hot-spot mutations in 50 genes was highly repeatable and reproducible and validated for routine application in a gene testing lab. Note: This abstract was not presented at the meeting. Citation Format: Wenzhi Li, Henry She, Xiaohua Tian, Margaret Neja, Adrian Harris, Yuhua Li. Validation of an NGS panel assay for detection of hotspot cancer somatic mutations [abstract]. In: Proceedings of the American Association for Cancer Research Annual Meeting 2019; 2019 Mar 29-Apr 3; Atlanta, GA. Philadelphia (PA): AACR; Cancer Res 2019;79(13 Suppl):Abstract nr 3007.

Pre-clinical validation of a next generation sequencing testing panel

ObjectiveNext Generation Sequencing (NGS) has become a useful tool for gene mutation testing which is required for targeted therapies. The aim of this study was to validate the GeneRead QIAact Actionable Insights Tumor Panel (Qiagen) on the GeneReader System in a diagnostic laboratory setting. MethodsThe GeneRead QIAact Actionable Insights Tumor Panel allows the analysis of 773 variant positions in 12 genes (ALK, BRAF, EGFR, ERBB2, ERBB3, ESR1, KIT, KRAS, NRAS, PDGFRA, PIK3CA and RAF1). For the validation of the panel we used a commercial available multiplex reference standard carrying 11 mutations in defined positions, samples from interlaboratory tests, and FFPE tumor samples from patients which were tested previously for mutations in KRAS, NRAS, BRAF, EGFR, KIT, and/or PDGFRA with pyrosequencing. ResultsAmong the 122 tested samples, 121 samples (99.2%) were successfully sequenced. The sensitivity and specificity for detecting variants was 100% and results proved to be reproducible and precise. 119 (98.3%) results were concordant to the expected results. The differences between NGS and pyrosequencing observed in two samples were due to a wrong analysis by the pyrosequencing software which did not cover the present mutations. ConclusionOverall, the GeneRead QIAact Actionable Insights Tumor Panel was specific and sensitive for mutation analysis for targeted therapies and can be incorporated into laboratory diagnostics' daily practice.

How effective are mutation testing tools? An empirical analysis of Java mutation testing tools with manual analysis and real faults

Mutation analysis is a well-studied, fault-based testing technique. It requires testers to design tests based on a set of artificial defects. The defects help in performing testing activities by measuring the ratio that is revealed by the candidate tests. Unfortunately, applying mutation to real-world programs requires automated tools due to the vast number of defects involved. In such a case, the effectiveness of the method strongly depends on the peculiarities of the employed tools. Thus, when using automated tools, their implementation inadequacies can lead to inaccurate results. To deal with this issue, we cross-evaluate four mutation testing tools for Java, namely PIT, muJava, Major and the research version of PIT, PITRV, with respect to their fault-detection capabilities. We investigate the strengths of the tools based on: a) a set of real faults and b) manual analysis of the mutants they introduce. We find that there are large differences between the tools' effectiveness and demonstrate that no tool is able to subsume the others. We also provide results indicating the application cost of the method. Overall, we find that PITRV achieves the best results. In particular, PITRV outperforms the other tools by finding 6% more faults than the other tools combined.

Avoiding useless mutants

Mutation testing is a program-transformation technique that injects artificial bugs to check whether the existing test suite can detect them. However, the costs of using mutation testing are usually high, hindering its use in industry. Useless mutants (equivalent and duplicated) contribute to increase costs. Previous research has focused mainly on detecting useless mutants only after they are generated and compiled. In this paper, we introduce a strategy to help developers with deriving rules to avoid the generation of useless mutants. To use our strategy, we pass as input a set of programs. For each program, we also need a passing test suite and a set of mutants. As output, our strategy yields a set of useless mutants candidates. After manually confirming that the mutants classified by our strategy as "useless" are indeed useless, we derive rules that can avoid their generation and thus decrease costs. To the best of our knowledge, we introduce 37 new rules that can avoid useless mutants right before their generation. We then implement a subset of these rules in the MUJAVA mutation testing tool. Since our rules have been derived based on artificial and small Java programs, we take our MUJAVA version embedded with our rules and execute it in industrial-scale projects. Our rules reduced the number of mutants by almost 13% on average. Our results are promising because (i) we avoid useless mutants generation; (ii) our strategy can help with identifying more rules in case we set it to use more complex Java programs; and (iii) our MUJAVA version has only a subset of the rules we derived.

A Study and Review on the Development of Mutation Testing Tools for Java and Aspect-J Programs

Mutation analysis in software testing is observed as the most effective way to validate the software under inspection. In last decade, number of researchers developed various methods and tools to apply mutation testing on Aspect Oriented Programs. In this paper, authors analyzed numerous mutation testing based tools available to test the Java and AspectJ programs. All effective and popular Aspect-J testing tools have been considered and analyzed in this paper, based on essential requirements in this context, considered to be fulfilled by testing tools decided by testing professional and researchers for such tools. This paper analyzed the work progress in the field of mutation testing techniques and tools specific to Java and AspectJ. This work considered essential parameters on which the analysis of analyzed tools is carried out. In case of addition parameters considered for evaluation, some of the resultant metrics may vary slightly under modification in basic requirements. Based on the numeric value estimated, it is finally suggested the merits of a mutation tool under different circumstances. This is the extension of the work carried by us in previous review for aspect based mutation testing techniques.