Test Suite Generation Research Articles

Wingsuit Flying Search Optimization Algorithm Strategy for Combinatorial T-Way Test Suite Generation

Advancement in software development has resulted in complex software applications that encompass various functional and non-functional requirements. Such a complex system usually consists of many inputs either directly from users or from other connected systems or devices. Here, there is a potential for the system to go wrong due to certain combinations of inputs. Combinatorial Testing (or T-Way Testing) is effective in tackling the issue. Numerous studies have proposed strategies in generating T-Way test suite, and current trend indicates that researchers often incorporate metaheuristic algorithms in their proposed strategies. Many recent studies employ parameter optimization algorithms such as (Whale Optimization Algorithm, Particle Swarm Optimization, Gravitational Optimization Algorithm) in generating an optimized T-Way test suite. Often, researchers need to tune the parameters involved in the algorithm before the algorithm can be used for test suite generation. Since the system under test (SUT) can come in various numbers of input, it is impossible to find a single best value for every algorithm parameter. As a result, this paper proposed a T-Way Test suite generator utilizing Wingsuit Optimization Algorithm (a parameter free optimization algorithm) for combinatorial test suite generation. The algorithm learnt by itself as the optimization process progresses and hence eliminates the need for control parameters. Statistical analysis shows that WFS produces a smaller test suite compares to most T-Way strategies and in some cases, the difference between test suite size produce by WFS and other T-Way strategies are insignificant.

Автоматизований метод перевірки та налагодження тестових сценаріїв на основі формальної моделі

Introduction. Model-based test cases generation is a popular strategy for test automation. It helps to reduce time spent on the development of a test suite and can improve level of coverage. However, many reports show shortage of such test cases in poor quality and doubtable efficiency. Purpose. The main goal of the proposed method is cost-effective validation, assessment, debugging and concretization of generated test cases. The method helps improve quality and efficiency of the test cases, make their scenario meaningful and goal-oriented. The method also develops debugging facilities and simplifies data dependency analysis and test scenario editing. Methods. We propose an automated post-processing method which allows to evaluate path that is examined by the test case, and to make safe changes to the path which will eliminate the shortcomings while leaving the coverage targets of the test case unharmed. The method is based on visualization of the path along the control flow graph of the model with additional information about factual evaluation history of all variables and possible alternative variants of behavior. For consistent substitution of certain values in the signal parameters, which would determine the artifacts of the test environment (such as, for example, files, databases, etc.) and check boundary cases (in predicates of conditions, indexing of arrays, etc.), a method of interactive specification of symbolic traces has been developed. Results. The role of the user in deciding whether to add a test case to the project test suite and make changes to it remains crucial, but to reduce labor intensity, the following processes are automated: evaluation of test scenarios according to certain objective characteristics (level of coverage, ability to detect defects, data cohesion, etc.); highlighting of possible alternatives for making corrections; consistent updating of computations for the corresponding corrections. A prototype was developed based on the proposed methods. The empirical results demonstrated a positive impact on the overall efficiency (ability to detect defects and reduce resource consumption) and quality (meaningfulness, readability, maintenance, usefulness for debugging, etc) of the generated test suites. The method allows to make automatically generated test cases trustable and usable. Conclusion. The proposed toolkit significantly reduces the time spent on researching the results of test generation and validation of the obtained tests and their editing. Unlike existing simulation methods, the proposed method not only informs about the values of variables, but also explores the history of their computations and additionally provides information about admissible alternatives. Further we plan to improve the process of localizing the causes of test failure at execution phase to speed up the search for defects.

Enhancing logic-based testing with EvoDomain: A search-based domain-oriented test suite generation approach

ContextEffective software testing requires test adequacy criteria. MC/DC, a widely used logic-based testing criterion, struggles to detect domain errors caused by incorrect arithmetic operations. Domain errors occur when test requirement boundaries shift or tilt, causing unpredictable behavior and system crashes. ObjectiveTo address the inadequacy of MC/DC in detecting domain errors, we present EvoDomain, a search-based testing technique. MethodEvoDomain uses a memetic algorithm combining genetic and hill-climbing algorithms, along with the DBSCAN clustering algorithm to select diversified boundary test data. The memetic algorithm is designed to efficiently enhance the search process for covering boundary test data. We compared EvoDomain with two logic-based testing approaches, a domain-oriented test suite generation approach, and random testing. ResultsEvaluations on 30 case studies show EvoDomain increases fault detection by 74.44% over MC/DC and 65.06% over RoRG. Additionally, EvoDomain improves support for different fault types by up to 68.89% for MC/DC and 66.33% for RoRG. Compared to COSMOS, which uses static analysis, EvoDomain improves the convergence effectiveness of identifying feasible subdomains by 32%. It offers high accuracy (0.99-1) and F1-score (0.99-1). EvoDomain finds the subdomains in less than 1/3 the time of Random search. ConclusionEvoDomain effectively generates domain-oriented test suites, enhancing the accuracy and effectiveness of fault detection.

Solving the t -wise Coverage Maximum Problem via Effective and Efficient Local Search-based Sampling

To meet the increasing demand for customized software, highly configurable systems become essential in practice. Such systems offer many options to configure, and ensuring the reliability of these systems is critical. A widely-used evaluation metric for testing these systems is \(t\) -wise coverage, where \(t\) represents testing strength, and its value typically ranges from 2 to 6. It is crucial to design effective and efficient methods for generating test suites that achieve high \(t\) -wise coverage. However, current state-of-the-art methods need to generate large test suites for achieving high \(t\) -wise coverage. In this work, we propose a novel method called LS-Sampling-Plus that can efficiently generate test suites with high \(t\) -wise coverage for \(2\leq t\leq 6\) while being smaller in size compared to existing state-of-the-art methods. LS-Sampling-Plus incorporates many core algorithmic techniques, including two novel scoring functions, a dynamic mechanism for updating sampling probabilities, and a validity-guaranteed systematic search method. Our experiments on various practical benchmarks show that LS-Sampling-Plus can achieve higher \(t\) -wise coverage than current state-of-the-art methods, through building a test suite of the same size. Moreover, our evaluations indicate the effectiveness of all core algorithmic techniques of LS-Sampling-Plus . Further, LS-Sampling-Plus exhibits better scalability and fault detection capability than existing state-of-the-art methods.

Code-Aware Prompting: A Study of Coverage-Guided Test Generation in Regression Setting using LLM

Testing plays a pivotal role in ensuring software quality, yet conventional Search Based Software Testing (SBST) methods often struggle with complex software units, achieving suboptimal test coverage. Recent work using large language models (LLMs) for test generation have focused on improving generation quality through optimizing the test generation context and correcting errors in model outputs, but use fixed prompting strategies that prompt the model to generate tests without additional guidance. As a result LLM-generated testsuites still suffer from low coverage. In this paper, we present SymPrompt, a code-aware prompting strategy for LLMs in test generation. SymPrompt’s approach is based on recent work that demonstrates LLMs can solve more complex logical problems when prompted to reason about the problem in a multi-step fashion. We apply this methodology to test generation by deconstructing the testsuite generation process into a multi-stage sequence, each of which is driven by a specific prompt aligned with the execution paths of the method under test, and exposing relevant type and dependency focal context to the model. Our approach enables pretrained LLMs to generate more complete test cases without any additional training. We implement SymPrompt using the TreeSitter parsing framework and evaluate on a benchmark challenging methods from open source Python projects. SymPrompt enhances correct test generations by a factor of 5 and bolsters relative coverage by 26% for CodeGen2. Notably, when applied to GPT-4, SymPrompt improves coverage by over 2x compared to baseline prompting strategies.

Automated Test Suite Generation for Software Product Lines Based on Quality-Diversity Optimization

A Software Product Line (SPL) is a set of software products that are built from a variability model. Real-world SPLs typically involve a vast number of valid products, making it impossible to individually test each of them. This arises the need for automated test suite generation, which was previously modeled as either a single-objective or a multi-objective optimization problem considering only objective functions. This article provides a completely different mathematical model by exploiting the benefits of Quality-Diversity (QD) optimization that is composed of not only an objective function (e.g., t -wise coverage or test suite diversity) but also a user-defined behavior space (e.g., the space with test suite size as its dimension). We argue that the new model is more suitable and generic than the two alternatives because it provides at a time a large set of diverse (measured in the behavior space) and high-performing solutions that can ease the decision-making process. We apply MAP-Elites, one of the most popular QD algorithms, to solve the model. The results of the evaluation, on both realistic and artificial SPLs, are promising, with MAP-Elites significantly and substantially outperforming both single- and multi-objective approaches, and also several state-of-the-art SPL testing tools. In summary, this article provides a new and promising perspective on the test suite generation for SPLs.

CMBMeTest: Generation of Test Suites Using Model-Based Testing Plus Constraint Programming and Metamorphic Testing

Various software testing techniques have been shown to be successful in producing high-quality test suites for software where the code is not accessible (black-box approach). Nevertheless, no method has been found to guide combining some of these in a general way. In this study, a test suite generation method for black-box software called CMBMeTest was created to respond to these challenges. It employs several coupled software testing techniques, namely, model-based testing (MBT), constraint programming (CP), and metamorphic testing (MT). CMBMeTest provides step-by-step instructions for using the information available (such as program specifications, inputs and outputs) to create an initial test suite that covers the model obtained, using a combination of MBT and CP (referred to as MBT+CP). Furthermore, using the metamorphic relations (MRs) of MT, a better test suite was produced from that initial test suite. The method allows particular stages to be iterated to improve the results by building new models and new MRs. A comprehensive case study was conducted, employing CMBMeTest to produce encouraging results. Mutation testing was used to evaluate the test suite, and the first round produced a high mutation score. A more detailed model was used to repeat the process, with similar outcomes.

GUI testing of Android applications: Investigating the impact of the number of testers on different exploratory testing strategies

AbstractGraphical user interface (GUI) testing plays a pivotal role in ensuring the quality and functionality of mobile apps. In this context, exploratory testing (ET), a distinctive methodology in which individual testers pursue a creative, and experience‐based approach to test design, is often used as an alternative or in addition to traditional scripted testing. Managing the exploratory testing process is a challenging task that can easily result either in wasteful spending or in inadequate software quality, due to the relative unpredictability of exploratory testing activities, which depend on the skills and abilities of individual testers. A number of works have investigated the diversity of testers' performance when using ET strategies, often in a crowdtesting setting. These works, however, investigated ET effectiveness in detecting bugs, and not in scenarios in which the goal is to generate a re‐executable test suite, as well. Moreover, less work has been conducted on evaluating the impact of adopting different exploratory testing strategies. As a first step toward filling this gap in the literature, in this work, we conduct an empirical evaluation involving four open‐source Android apps and 20 masters students that we believe can be representative of practitioners partaking in exploratory testing activities. The students were asked to generate test suites for the apps using a capture and replay tool and different exploratory testing strategies. We then compare the effectiveness, in terms of aggregate code coverage that different‐sized groups of students using different exploratory testing strategies may achieve. Results provide deeper insights into code coverage dynamics to project managers interested in using exploratory approaches to test simple Android apps, on which they can make more informed decisions.

Q-learning whale optimization algorithm for test suite generation with constraints support

Q-learning whale optimization algorithm for test suite generation with constraints support

Variable-strength combinatorial testing of exported activities based on misexposure prediction

Exported Activity (EA), a kind of activities in Android apps that can be launched by external components, is one of the most important inter-component communication (ICC) mechanisms. In combinatorial testing of EAs, although exhaustive testing of all possible combinations of input elements is ideal, it is often not feasible due to the combinatorial explosion of test cases. This paper presents ExaDroid, a novel variable-strength combinatorial testing framework for generating test suites for exported activities. ExaDroid is based on two observations: many activities are unintentionally exposed, and the complexity of input interactions in activities can be very limited. ExaDroid uses misexposure prediction and complexity analysis to decide the (default) testing strength of an EA. It also leverages input interactions to focus testing resources on important combinations by setting stronger (variable) test strengths on certain attributes. Our experiments have confirmed that ExaDroid is capable of trigger many unique crashes using a dozen or so test cases. The tool successfully found 100 unique crashes across 135 EAs in 30 apps, at an average cost of 14.2 test cases per EA.

Mutation-Based Minimal Test Suite Generation for Boolean Expressions

Boolean expressions are highly involved in control flows of programs and software specifications. Coverage criteria for Boolean expressions aim at producing minimal test suites to detect software faults. There exist various testing criteria, efficiency of which is usually evaluated through mutation analysis. This paper proposes an integer programming-based minimal test suite generation technique relying on mutation analysis. The proposed technique also takes into account the cost of fault detection. The technique is optimal such that the resulting test suite guarantees to detect all the mutants under given fault assumptions, while maximizing the average percentage of fault detection of a test suite. Therefore, the approach presented can also be considered as a reference method to check the efficiency of any common technique. The method is evaluated using four well-known real benchmark sets of Boolean expressions and is also exemplary compared with MCDC criterion. The results show that the test suites generated by the proposed method provide better fault coverage values and faster fault detection.

Finite State Testing of Graphical User Interface using Genetic Algorithm

Graphical user interfaces are the key components of any software. Nowadays, the popularity of the software depends upon how easily the user can interact with the system. However, as the system becomes complex, this interaction is also complicated with many states. The testing of graphical user interfaces is an important phase of modern software. The testing of GUI is possible only by interacting with the system, which may be a time-consuming process and is generally automated based on the test suite. The test suite generation proposed in this paper is based on the genetic algorithm in which various test cases are generated heuristically. For performance validation of the proposed approach, the same has been compared with a variant of PSO, and it found that GA is slightly better in comparison to the PSO.

Cryptographic protocol conformance testing based on domain-specific state machine

We present a method of conformance testing of the cryptographic protocol implementations that ensures a high degree of automation of the testing process and provides criteria for the quality assessment of the generated test suites. Our method is based on a kind of extended (non-finite) state machine which is specifically designed for the domain of cryptographic protocols. The key concept of the proposed machine is a built-in mechanism of sequences, which, firstly, sets an effective way to deal with the nondeterminism inherent in cryptographic protocols and, secondly, provides the means to express "inconvenient" (for the high-level declarative languages) requirements, such as e.g. "take the concatenation of the specified parts of all the previous messages which meet specified condition". Basing on this machine, we propose a high-level declarative language (embedded in Haskell) intended for development of interoperable protocol specifications used as input data for our testing tool. The protocol specification consists of two parts: a description of the structure of the protocol messages (this part, by virtue of its conciseness and declarativeness, can be directly incorporated in the primary specifications such as RFC) and a set of state machine transition functions binded to the sequences. In the paper, we present the principles of building of our machine and introduce the notion of sequences fuzzing. The proposed approach was successfully approved for the industrial cryptographic protocol TLS.

Generating Class-Level Integration Tests Using Call Site Information

Search-based approaches have been used in the literature to automate the process of creating unit test cases. However, related work has shown that generated unit-tests with high code coverage could be ineffective, i.e., they may not detect all faults or kill all injected mutants. In this paper, we propose CLING, an integration-level test case generation approach that exploits how a pair of classes, the caller and the callee, interact with each other through method calls. In particular, CLING generates integration-level test cases that maximize the Coupled Branches Criterion (CBC). Coupled branches are pairs of branches containing a branch of the caller and a branch of the callee such that an integration test that exercises the former also exercises the latter. CBC is a novel integration-level coverage criterion, measuring the degree to which a test suite exercises the interactions between a caller and its callee classes. We implemented CLING and evaluated the approach on 140 pairs of classes from five different open-source Java projects. Our results show that (1) CLING generates test suites with high CBC coverage, thanks to the definition of the test suite generation as a many-objectives problem where each couple of branches is an independent objective; (2) such generated suites trigger different class interactions and can kill on average 7.7% (with a maximum of 50%) of mutants that are not detected by tests generated at the unit level; (3) CLING can detect integration faults coming from wrong assumptions about the usage of the callee class (32 for our subject systems) that remain undetected when using automatically generated unit-level test suites.

InterEvo-TR: Interactive Evolutionary Test Generation With Readability Assessment

Automated test case generation has proven to be useful to reduce the usually high expenses of software testing. However, several studies have also noted the skepticism of testers regarding the comprehension of generated test suites when compared to manually designed ones. This fact suggests that involving testers in the test generation process could be helpful to increase their acceptance of automatically-produced test suites. In this paper, we propose incorporating interactive readability assessments made by a tester into EvoSuite, a widely-known evolutionary test generation tool. Our approach, <sc xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">InterEvo-TR</small> , interacts with the tester at different moments during the search and shows different test cases covering the same coverage target for their subjective evaluation. The design of such an interactive approach involves a schedule of interaction, a method to diversify the selected targets, a plan to save and handle the readability values, and some mechanisms to customize the level of engagement in the revision, among other aspects. To analyze the potential and practicability of our proposal, we conduct a controlled experiment in which 39 participants, including academics, professional developers, and student collaborators, interact with <sc xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">InterEvo-TR</small> . Our results show that the strategy to select and present intermediate results is effective for the purpose of readability assessment. Furthermore, the participants' actions and responses to a questionnaire allowed us to analyze the aspects influencing test code readability and the benefits and limitations of an interactive approach in the context of test case generation, paving the way for future developments based on interactivity.

On integrating multiple restriction domains to automatically generate test cases of model transformations

Testing model transformations poses several challenges, one of which is how to automatically generate effective test suites. Existing approaches in the literature often focus on exploiting either structural aspects of models captured by a metamodel or restrictions as contracts within the specification of model transformation for a partition analysis to generate test suites. However, test cases for model transformation, i.e., test models, often need to be defined not only from one restriction source as within the current methods but also from many different sources. This paper introduces a method to facilitate defining test models on such multiple restriction domains. This method also allows the tester to flexibly select and combine constraints as a restriction on test models for different strategies and objectives of the model transformation testing. We develop a support tool based on the OCL tool USE and perform experiments on several different transformations to point out the effectiveness of our method.

Conformance Checking and Pushdown Reactive Systems

Due to their asynchronous interactions, testing reactive systems is a laborious activity present in any software development project.&#x0D; In this setting, the finite memory formalism of Labeled Transition Systems has been used to generate test suites that can be applied to check ioco conformance of implementations to a given specification.&#x0D; In this work we turn to a more complex scenario where a stronger formalism is considered, the Visibly Pushdown Labeled Transition System (VPTS), which allows access to a potentially infinite pushdown memory.&#x0D; We study an extension of the ioco conformance relation to VPTS models and develop polynomial time algorithms to verify conformance for VPTS models in a white-box testing scenario.

Multi-Objective Optimization Model and Hierarchical Attention Networks for Mutation Testing

Mutation testing is devised for measuring test suite adequacy by identifying the artificially induced faults in software. This paper presents a novel approach by considering multiobjectives-based optimization. Here, the optimal test suite generation is performed using the proposed water cycle water wave optimization (WCWWO). The best test suites are generated by satisfying the multi-objective factors, such as time of execution, test suite size, mutant score, and mutant reduction rate. The WCWWO is devised by a combination of the water cycle algorithm (WCA) and water wave optimization (WWO). The hierarchical attention network (HAN) is used for classifying the equivalent mutants by utilizing the MutPy tool. Furthermore, the performance of the developed WCWWO+HAN is evaluated in terms of three metrics—mutant score (MS), mutant reduction rate (MRR), and fitness—with the maximal MS of 0.585, higher MRR of 0.397, and maximum fitness of 0.652.

Fragment-Based Test Generation for Web Apps

Automated model-based test generation presents a viable alternative to the costly manual test creation currently employed for regression testing of web apps. However, existing model inference techniques rely on threshold-based whole-page comparison to establish state equivalence, which cannot reliably identify near-duplicate web pages in modern web apps. Consequently, existing techniques produce inadequate models for dynamic web apps, and fragile test oracles, rendering the generated regression test suites ineffective. We propose a model-based test generation technique, FRAGGEN, that eliminates the need for thresholds, by employing a novel state abstraction based on page fragmentation to establish state equivalence. FRAGGEN also uses fine-grained page fragment analysis to diversify state exploration and generate reliable test oracles. Our evaluation shows that FRAGGEN outperforms existing whole-page techniques by detecting more near-duplicates, inferring better web app models and generating test suites that are better suited for regression testing. On a dataset of 86,165 state-pairs, FRAGGEN detected 123% more near-duplicates on average compared to whole-page techniques. The crawl models inferred by FRAGGEN have 62% more precision and 70% more recall on average. FRAGGEN also generates reliable regression test suites with test actions that have nearly 100% success rate on the same version of the web app even if the execution environment is varied. The test oracles generated by FRAGGEN can detect 98.7% of the visible changes in web pages while being highly robust, making them suitable for regression testing.

COSMOS: A comprehensive framework for automatically generating domain-oriented test suite

Coverage criteria are satisfied by at least one examination of the test target, while many faults are not revealed by one execution. However, despite executing the faulty statement, the test result is correct in certain circumstances. Such coincidentally passing test cases execute the faulty statement but do not cause failures. This paper introduces the new concept of domain solver. Domain solvers attempt to detect the domain of inputs rather than a single input satisfying a path constraint. Domain coverage is a new metric to evaluate the relative accuracy of the detected domains. The promising point is that the proposed approach similarly treats nonlinear and linear constraints. Domain solver splits a path constraint into conjunctions of simple conditions comparing two expressions. Such a splitting simplifies the constraint-solving task to detect regions of the input space satisfying a comparison between two expressions. After finding a region, an improved version of an algorithm, update, is used to determine the domain of variables involved in the comparing expressions. Our proposed approach, COSMOS, is implemented using the Roslyn compiler. We compared COSMOS with well-known constraint solvers using various linear/nonlinear constraints. The results show that COSMOS improves the number of supported data types involved in a constraint and solves100% of the instances in which the other solvers fail. Besides, COSMOS achieves the best relative accuracy of 84% compared to the existing domain-oriented test suite generation approaches. Moreover, our experiment results illustrate that COSMOS improves the fault-finding capability of other existing test coverage criteria by detecting coincidentally correct test cases. Combining domain coverage and compiler as a service makes a powerful constraint satisfaction method outperforming the existing constraint solvers regarding the number of solved linear/nonlinear constraints. Augmenting other structural test coverage criteria with domain coverage reveals the coincidentally correct test cases.