Higher Order Mutation Testing Research Articles

Hybridizing genetic algorithm and harmony search for higher-order mutation testing

Mutation testing can evaluate the quality of the test inputs, generate test data, and simulate any test coverage criterion. Genetic algorithms and harmony search have been applied to reduce the cost of generating test inputs. Although hybridizing search algorithms enhances the efficiency of searching the solution domain, there is a shortage of applying the hybrid search techniques in mutation testing. This paper merges the genetic and harmony search algorithms to effectively generate test data to kill higher-order mutants. In addition, the performance of the proposed technique will be evaluated and compared with a stand-alone genetic algorithm and a stand-alone harmony search algorithm through an empirical study using a set of benchmark programs. The experimental study shows that the proposed technique outperformed the compared algorithms, reaching a higher killing ratio, where the proposed approach kills 92.8% of higher-order mutants for all tested programs. In comparison, GA kills 88.7%, and HA kills 86.6%. Besides, the proposed algorithm overcame the compared algorithm in reaching a targeted killing ratio faster than the compared algorithms. HGA reduced the execution time for each program with a reduction ratio ranging from 58.9% to 89.8%.

Empirical Comparison of Higher-Order Mutation Testing and Data-Flow Testing of C# with the Aid of Genetic Algorithm

Data-Flow and Higher-Order Mutation are white-box testing techniques. To our knowledge, no work has been proposed to compare data flow and Higher-Order Mutation. This paper compares all def-uses Data-Flow and second-order mutation criteria. The comparison will support the testing decision-making, especially when choosing a suitable criterion. This compassion investigates the subsumption relation between these two criteria and evaluates the effectiveness of test data developed for each. To compare the two criteria, a set of test data satisfying each criterion is generated using genetic algorithms; the set is then used to explore whether one criterion subsumes the other criterion and assess the effectiveness of the test set that was developed for one methodology in terms of the other. The results showed that the mean mutation coverage ratio of the all du-pairs adequate test cover is 80.9%, and the mean data flow coverage ratio of the second-order mutant adequate test cover is 98.7%. Consequently, second-order mutation “ProbSubsumes” the all du-pairs data flow. The failure detection efficiency of the mutation (98%) is significantly better than the failure detection efficiency of data flow (86%). Consequently, second-order mutation testing is “ProbBetter” than all du-pairs data flow testing. In contrast, the size of the test suite of second-order mutation is more significant than the size of the test suite of all du-pairs.

Toward Improving the Quality of Mutation Operator and Test Case Effectiveness in Higher-Order Mutation Testing

Currently, there are many research studies that apply and improve mutation testing techniques including traditional mutation testing or first-order mutation testing, and higher-order mutation testing (HOMT) for evaluating the quality of the set of test data in particular, and the quality of test suites in general. The results of those studies have proven the effectiveness of mutation testing in the field of software testing. Mutation testing allows the quality of test cases to be automatically evaluated, thereby helping the testers to improve the quality in the design and execution of the software testing. Besides, these studies have also pointed out the main barriers in applying mutation testing techniques in practice. However, we are the first to introduce a method that can reduce the cost, but keep the quality of testing activity based on evaluating the quality of the mutation operator as well as the quality of the test cases. In this paper, we concentrate on two problems regarding higher-order mutation testing: Evaluating the quality of mutation operators as well as generated mutants and prioritizing test cases based upon its capability of killing mutants. This may help developers allocate suitably their resources during testing phase. The study of this paper is an extended version of our previous study titled “Evaluating Mutation Operator and Test Case Effectiveness by Means of Mutation Testing”, which is published in the proceedings of the 13th Asian Conference on Intelligent Information and Database Systems, ACIIDS 2021 (V. N. Do, Q. V. Nguyen and T. B. Nguyen, Evaluating Mutation Operator and Test Case Effectiveness by Means of Mutation Testing, in Intelligent Information and Database Systems. ACIIDS 2021, eds. N. T. Nguyen, S. Chittayasothorn, D. Niyato and B. Trawiński. Lecture Notes in Computer Science, Vol. 12672 (Springer, Cham), https://doi.org/10.1007/978-3-030-73280-6_66 ) to confirm the usefulness of our proposed method.

Mutants as Patches: Towards a formal approach to Mutation Testing

Abstract Background: Mutation testing is a widely explored technique used to evaluate the quality of software tests, but little attention has been given to its mathematical foundations. Aim: We provide a formal description of the core concepts in mutation testing, relations between them and conclusions that can be drawn from the presented model. Method: We introduce concepts of mutant space and patch space, and refer to patch merging procedure from the patch theory. We explicitly present constraints, such as location-dependence, that affect mutation operators. We also present a way to use introduced formalism with traditional operators proposed in other papers. Results: The proposed formalism allows to describe interactions between separate mutations using well-known abstract algebra notation. Conclusion: The presented formalism may substantially decrease the number of tested weak mutants and increase the number of valuable ones, while giving tools to partially address the problem of equivalent mutants, particularly for higher-order mutation testing. However, additional empirical evaluation is still needed.

A systematic mapping study on higher order mutation testing

Context: Higher Order Mutants (HOMs) present some advantages concerning the First-Order Mutants (FOMs). HOMs can better simulate real and subtle faults, reduce the number of generated mutants and test cases, and so on.Objective: In order to characterize the Higher Order Mutation Testing (HOMT) field, this paper presents results of a mapping study, by synthesizing characteristics of the HOMT approaches, HOM generation strategies, evaluation aspects, trends and research opportunities.Method: We followed a research plan to locate, assess, extract and group the outcomes from relevant studies. We found 69 primary studies, which were classified based on dimensions related to aspects of the conducted evaluation, purpose and use of HOMs.Results: Java is the preferred language. Most approaches use Second-Order Mutants (SOMs). We found 50 different techniques used to generate/select HOMs. We observed that from 39 primary studies which apply a strategy, ≈49% use search-based techniques.Conclusions: HOMT has been arising interest in the last years. The results herein presented provide researchers the start-of-the-art on HOMT, allowing them to understand existing approaches, and how the HOMs have been used and evaluated. Furthermore, this paper points out open issues and not addressed topics, which require more investigation, discussing trends and research opportunities in the field.

Employing Dynamic Symbolic Execution for Equivalent Mutant Detection

Equivalent mutants (EM) issue is a key challenge in mutation testing. Many methods were applied for detecting and reducing the equivalent mutants. These methods are classified into four classes: equivalent mutant detection, avoiding the generation of equivalent mutants, higher-order equivalent mutants, and suggesting equivalent mutants. Higher-order mutation testing (HOMT) is considered the strongest employed technique in avoiding the generation of equivalent mutants and reduction of their number. In this paper, a combination of HOMT especially second-order mutation testing (SOMT) and dynamic symbolic execution (DSE) techniques are applied for the automatic detection and reduction of the equivalent second-order mutants. First, SOMT is used to reduce the number of equivalent mutants. Second, DSE technique is applied to classify the SOMs and detect EM. To assess the efficiency of the proposed technique, it is applied to some subject programs and the results of this technique are compared to the manual results and those of related works. The results showed that the proposed algorithm is more effective in detecting and reducing the number of EM. It detects 94% from the equivalent mutants that have manually analyzed. This percentage is a high percentage comparing with previous studies. Besides, the DEM-DSE technique detects 100% of equivalent mutants for 9 of the 14 subject programs.

Reducing the Cost of Higher-Order Mutation Testing

Constructing mutants of order higher than first order is a key step in higher-order mutation testing. The majority of higher-order mutant generation techniques merge two (or more) first-order mutants (FOMs) to build a higher-order mutant. Unfortunately, these techniques suffer from the high cost due to the explosion in the number of higher-order mutants (HOMs). Consequently, developing techniques to find the minimum adequate and effective number of mutants are desired. Earlier work reduced the number of mutants by considering only a subset of mutants, a subset of operators, or selecting specific locations in the program (to be mutated) instead of the whole program. In this paper, we present three new techniques (SCWR, 2E2O, and 2E2OWR) to generate and reduce the overall number of HOMs and the equivalent ones as well. Each technique merges FOMs in more effective and simpler way than the previous techniques to find more effective HOMs. These techniques have been applied on a benchmark of programs, and the results have been compared to the results of some related work such as DiffOp, JudyDiffOp, and Last2First techniques. The results showed that SCWR, 2E2O, and 2E2OWR outperformed the related work and reduced the total number of mutants by 67.9, 16.0, and 60.1% comparing to approximately 50% for the related work and the number of equivalent mutants by 66.9, 79.6, and 65.0% comparing to 25.8 and 36.4% for the related work, respectively.

Empirical Evaluation of Multiobjective Optimization Algorithms Searching for Higher Order Mutants

ABSTRACTFirst order mutation testing is used to evaluate the quality of a given set of test cases by inserting single changes into the program under test to produce first order mutants (FOMs) of the original program, and then checking whether tests are good enough to detect the artificially injected defects. However, mutation testing is not yet widely used due to the problems of a large number of generated mutants and limited realism of introduced changes that do not necessarily reflect real software defects. Furthermore, many of the generated mutants are equivalent, i.e., they keep the program semantics unchanged and, thus, cannot be detected by any test suite. Higher order mutation testing has been coined as a promising solution for overcoming these limitations of FOM testing. In particular, finding strongly subsuming higher order mutants (SSHOMs), which are able to replace all of their constituent FOMs without scarifying test effectiveness while being able to reflect complex, real defects that require more than one change to correct them, is considered an important research challenge and is the focus of this work. The contribution of this article is a new, extended classification of higher order mutants (HOMs) to cover all cases of generated HOMs. Fitness functions and empirical comparison of four different multiobjective optimization algorithms are used to generate and evaluate HOMs as well as search for valuable high-quality and reasonable HOMs (strongly subsuming and coupled HOMs) and ten other types of HOMs. The main goal of this study is to assert the effect of applying multiobjective optimization algorithms in the area of higher order-mutation testing, while asserting the correctness of the proposed HOMs classification, objectives, and fitness functions. Our experimental results show that the total number of generated HOMs is smaller (about 70%) in comparison to FOMs, while the mean ratio of reasonable HOMs (subsuming HOMs) to all found HOMs is over 56%, and the mean ratio of high-quality and reasonable HOMs (strongly subsuming and coupled HOMs) to all found reasonable HOMs (subsuming HOMs) is fairly high (around 8.74%).