Search-based Software Testing Research Articles

Code-Aware Prompting: A Study of Coverage-Guided Test Generation in Regression Setting using LLM

Testing plays a pivotal role in ensuring software quality, yet conventional Search Based Software Testing (SBST) methods often struggle with complex software units, achieving suboptimal test coverage. Recent work using large language models (LLMs) for test generation have focused on improving generation quality through optimizing the test generation context and correcting errors in model outputs, but use fixed prompting strategies that prompt the model to generate tests without additional guidance. As a result LLM-generated testsuites still suffer from low coverage. In this paper, we present SymPrompt, a code-aware prompting strategy for LLMs in test generation. SymPrompt’s approach is based on recent work that demonstrates LLMs can solve more complex logical problems when prompted to reason about the problem in a multi-step fashion. We apply this methodology to test generation by deconstructing the testsuite generation process into a multi-stage sequence, each of which is driven by a specific prompt aligned with the execution paths of the method under test, and exposing relevant type and dependency focal context to the model. Our approach enables pretrained LLMs to generate more complete test cases without any additional training. We implement SymPrompt using the TreeSitter parsing framework and evaluate on a benchmark challenging methods from open source Python projects. SymPrompt enhances correct test generations by a factor of 5 and bolsters relative coverage by 26% for CodeGen2. Notably, when applied to GPT-4, SymPrompt improves coverage by over 2x compared to baseline prompting strategies.

On the Impact of Lower Recall and Precision in Defect Prediction for Guiding Search-based Software Testing

Defect predictors, static bug detectors, and humans inspecting the code can propose locations in the program that are more likely to be buggy before they are discovered through testing. Automated test generators such as search-based software testing (SBST) techniques can use this information to direct their search for test cases to likely buggy code, thus speeding up the process of detecting existing bugs in those locations. Often the predictions given by these tools or humans are imprecise, which can misguide the SBST technique and may deteriorate its performance. In this article, we study the impact of imprecision in defect prediction on the bug detection effectiveness of SBST. Our study finds that the recall of the defect predictor, i.e., the proportion of correctly identified buggy code, has a significant impact on bug detection effectiveness of SBST with a large effect size. More precisely, the SBST technique detects 7.5 fewer bugs on average (out of 420 bugs) for every 5% decrements of the recall. However, the effect of precision, a measure for false alarms, is not of meaningful practical significance, as indicated by a very small effect size. In the context of combining defect prediction and SBST, our recommendation is to increase the recall of defect predictors as a primary objective and precision as a secondary objective. In our experiments, we find that 75% precision is as good as 100% precision. To account for the imprecision of defect predictors, in particular low recall values, SBST techniques should be designed to search for test cases that also cover the predicted non-buggy parts of the program, while prioritising the parts that have been predicted as buggy.

Search-Based Software Testing Driven by Automatically Generated and Manually Defined Fitness Functions

Search-based software testing (SBST) typically relies on fitness functions to guide the search exploration toward software failures. There are two main techniques to define fitness functions: (a) automated fitness function computation from the specification of the system requirements, and (b) manual fitness function design. Both techniques have advantages. The former uses information from the system requirements to guide the search toward portions of the input domain more likely to contain failures. The latter uses the engineers’ domain knowledge. We propose ATheNA , a novel SBST framework that combines fitness functions automatically generated from requirements specifications and those manually defined by engineers. We design and implement ATheNA-S , an instance of ATheNA that targets Simulink ® models. We evaluate ATheNA-S by considering a large set of models from different domains. Our results show that ATheNA-S generates more failure-revealing test cases than existing baseline tools and that the difference between the runtime performance of ATheNA-S and the baseline tools is not statistically significant. We also assess whether ATheNA-S could generate failure-revealing test cases when applied to two representative case studies: one from the automotive domain and one from the medical domain. Our results show that ATheNA-S successfully revealed a requirement violation in our case studies.

.NET/C# instrumentation for search-based software testing

AbstractC# is one of the most widely used programming languages. However, to the best of our knowledge, there has been no work in the literature aimed at enabling search-based software testing techniques for applications running on the .NET platform, like the ones written in C#. In this paper, we propose a search-based approach and an open source tool to enable white-box testing for C# applications. The approach is integrated with a .NET bytecode instrumentation, in order to collect code coverage at runtime during the search. In addition, by taking advantage of Branch Distance, we define heuristics to better guide the search, e.g., how heuristically close it is to cover a branch in the source code. To empirically evaluate our technique, we integrated our tool into the EvoMaster test generation tool and conducted experiments on three .NET RESTful APIs as case studies. Results show that our technique significantly outperforms gray-box testing tools in terms of code coverage.

Instance Space Analysis of Search-Based Software Testing

Search-based software testing (SBST) is now a mature area, with numerous techniques developed to tackle the challenging task of software testing. SBST techniques have shown promising results and have been successfully applied in the industry to automatically generate test cases for large and complex software systems. Their effectiveness, however, has been shown to be problem dependent. In this paper, we revisit the problem of objective performance evaluation of SBST techniques in light of recent methodological advances - in the form of Instance Space Analysis (ISA) - enabling the strengths and weaknesses of SBST techniques to be visualised and assessed across the broadest possible space of problem instances (software classes) from common benchmark datasets. We identify features of SBST problems that explain why a particular instance is hard for an SBST technique, reveal areas of hard and easy problems in the instance space of existing benchmark datasets, and identify the strengths and weaknesses of state-of-the-art SBST techniques. In addition, we examine the diversity and quality of common benchmark datasets used in experimental evaluations.

Search-Based Software Test Data Generation for Path Coverage Based on a Feedback-Directed Mechanism

Automatically generating test cases by evolutionary algorithms to satisfy the path coverage criterion has attracted much research attention in software testing. In the context of generating test cases to cover many target paths, the efficiency of existing methods needs to be further improved when infeasible or difficult paths exist in the program under test. This is because a significant amount of the search budget (i.e., time allocated for the search to run) is consumed when computing fitness evaluations of individuals on infeasible or difficult paths. In this work, we present a feedback-directed mechanism that temporarily removes groups of paths from the target paths when no improvement is observed for these paths in subsequent generations. To fulfill this task, our strategy first organizes paths into groups. Then, in each generation, the objective scores of each individual for all paths in each group are summed up. For each group, the lowest value of the summed up objective scores among all individuals is assigned as the best aggregated score for a group. A group is removed when no improvement is observed in its best aggregated score over the last two generations. The experimental results show that the proposed approach can significantly improve path coverage rates for programs under test with infeasible or difficult paths in case of a limited search budget. In particular, the feedback-directed mechanism reduces wasting the search budget on infeasible paths or on difficult target paths that require many fitness evaluations before getting an improvement.

Fuzzing vs SBST

Search-Based Software Testing (SBST) is the application of SBSE for solving hard software testing problems. SBST has been the subject of discussion of our International SBST Workshop, colocated with the International Conference on Software Engineering (ICSE). In 2022 we hosted the 15th edition of our SBSTWorkshop, which brought together researchers and industrial practitioners to encourage the use of search-based and fuzz testing techniques and tools for addressing software engineering-specific challenges. In this 2022 edition, SBST held, among other exciting events, a discussion panel on the similarities and differences between SBST and Fuzzing. As it implies, the goal of the panel was to have a common ground for discussion on the main similarities and differences between the Fuzzing and SBST fields, focusing on how both communities can collaborate to advance the stateof- the-art on automated testing. This strong panel composed of researchers from both academia and industry was the highlight of SBST'22 and allowed the chairs of the workshop to make substantial changes for 2023. In this paper, we present the main take away messages from that seminal panel and highlight exciting new challenges in the field.

An Experimental Assessment of Using Theoretical Defect Predictors to Guide Search-Based Software Testing

Automated test generators, such as search-based software testing (SBST) techniques are primarily guided by coverage information. As a result, they are very effective at achieving high code coverage. However, is high code coverage alone sufficient to detect bugs effectively? In this paper, we propose a new SBST technique, predictive many objective sorting algorithm (PreMOSA), which augments coverage information with defect prediction information to decide where to increase the test coverage in the class under test (CUT). Through an experimental evaluation using 420 labelled bugs on the Defects4J benchmark and using theoretical defect predictors, we demonstrate the improved effectiveness and efficiency of PreMOSA in detecting bugs when using any acceptable defect predictor, i.e., a defect predictor with recall and precision <inline-formula><tex-math notation="LaTeX">$\geq$</tex-math></inline-formula> 75%, compared to the state-of-the-art dynamic many objective sorting algorithm (DynaMOSA). PreMOSA detects up to 8.3% more labelled bugs on average than DynaMOSA when given a time budget of 2 minutes for test generation per CUT.

DiverGet: a Search-Based Software Testing approach for Deep Neural Network Quantization assessment

Quantization is one of the most applied Deep Neural Network (DNN) compression strategies, when deploying a trained DNN model on an embedded system or a cell phone. This is owing to its simplicity and adaptability to a wide range of applications and circumstances, as opposed to specific Artificial Intelligence (AI) accelerators and compilers that are often designed only for certain specific hardware (e.g., Google Coral Edge TPU). With the growing demand for quantization, ensuring the reliability of this strategy is becoming a critical challenge. Traditional testing methods, which gather more and more genuine data for better assessment, are often not practical because of the large size of the input space and the high similarity between the original DNN and its quantized counterpart. As a result, advanced assessment strategies have become of paramount importance. In this paper, we present DiverGet, a search-based testing framework for quantization assessment. DiverGet defines a space of metamorphic relations that simulate naturally-occurring distortions on the inputs. Then, it optimally explores these relations to reveal the disagreements among DNNs of different arithmetic precision. We evaluate the performance of DiverGet on state-of-the-art DNNs applied to hyperspectral remote sensing images. We chose the remote sensing DNNs as they're being increasingly deployed at the edge (e.g., high-lift drones) in critical domains like climate change research and astronomy. Our results show that DiverGet successfully challenges the robustness of established quantization techniques against naturally-occurring shifted data, and outperforms its most recent concurrent, DiffChaser, with a success rate that is (on average) four times higher.

On the Faults Found in REST APIs by Automated Test Generation

RESTful web services are often used for building a wide variety of enterprise applications. The diversity and increased number of applications using RESTful APIs means that increasing amounts of resources are spent developing and testing these systems. Automation in test data generation provides a useful way of generating test data in a fast and efficient manner. However, automated test generation often results in large test suites that are hard to evaluate and investigate manually. This article proposes a taxonomy of the faults we have found using search-based software testing techniques applied on RESTful APIs. The taxonomy is a first step in understanding, analyzing, and ultimately fixing software faults in web services and enterprise applications. We propose to apply a density-based clustering algorithm to the test cases evolved during the search to allow a better separation between different groups of faults. This is needed to enable engineers to highlight and focus on the most serious faults. Tests were automatically generated for a set of eight case studies, seven open-source and one industrial. The test cases generated during the search are clustered based on the reported last executed line and based on the error messages returned, when such error messages were available. The tests were manually evaluated to determine their root causes and to obtain additional information. The article presents a taxonomy of the faults found based on the manual analysis of 415 faults in the eight case studies and proposes a method to support the classification using clustering of the resulting test cases.

Enhancing Search-based Testing with Testability Transformations for Existing APIs

Search-based software testing (SBST) has been shown to be an effective technique to generate test cases automatically. Its effectiveness strongly depends on the guidance of the fitness function. Unfortunately, a common issue in SBST is the so-called flag problem , where the fitness landscape presents a plateau that provides no guidance to the search. In this article, we provide a series of novel testability transformations aimed at providing guidance in the context of commonly used API calls (e.g., strings that need to be converted into valid date/time objects). We also provide specific transformations aimed at helping the testing of REST Web Services. We implemented our novel techniques as an extension to EvoMaster , an SBST tool that generates system-level test cases. Experiments on nine open-source REST web services, as well as an industrial web service, show that our novel techniques improve performance significantly.

Seeding Grammars in Grammatical Evolution to Improve Search-Based Software Testing

Heuristic-based optimization techniques have been increasingly used to automate different types of code coverage analysis. Several studies suggest that interdependencies (in the form of comparisons) may exist between the condition constructs, of variables and constant values, in the branching conditions of real-world programs, e.g. (i le 100) or (i==j), etc. In this work, by interdependencies we refer to the situations where, to satisfy a branching condition, there must be a certain relationship between the values of some specific condition constructs (which may or may not be a part of the respective condition predicates). For example, the values of variables i and j must be equal to satisfy the condition of (i==j), and the value of variable k must be equal to 100 for the satisfaction of the condition of (k==100). To date, only the Ariadne, a Grammatical Evolution (GE)-based system, exploits these interdependencies between input variables (e.g. of the form (i le j) or (i==j), etc.) to efficiently generate test data. Ariadne employs a simple attribute grammar to exploit these dependencies, which enables it to evolve complex test data, and has been compared favourably to other well-known techniques in the literature. However, Ariadne does not benefit from interdependencies involving constants, e.g. (i le 100) or (j==500), etc., due to the difficulty in evolving precise values, and these are equally important constructs of condition predicates. Furthermore, constant creation in GE can be difficult, particularly with high precision. We propose to seed the grammar with constants extracted from the source code of the program under test to enhance and extend Ariadne’s capability to exploit richer types of dependencies (involving all combinations of both variables and constant values). We compared our results with the original system of Ariadne against a large set of benchmark problems which include 10 numeric programs in addition to the ones originally used for Ariadne. Our results demonstrate that the seeding strategy not only dramatically improves the generality of the system, as it improves the code coverage (effectiveness) by impressive margins, but it also reduces the search budgets (efficiency) often up to an order of magnitude. Moreover, we also performed a rigorous analysis to investigate the scalability of our improved Ariadne, showing that it stays highly scalable when compared to both the original system of Ariadne and GA-based test data generation approach.

Diversifying Focused Testing for Unit Testing

Software changes constantly, because developers add new features or modifications. This directly affects the effectiveness of the test suite associated with that software, especially when these new modifications are in a specific area that no test case covers. This article tackles the problem of generating a high-quality test suite to cover repeatedly a given point in a program, with the ultimate goal of exposing faults possibly affecting the given program point. Both search-based software testing and constraint solving offer ready, but low-quality, solutions to this: Ideally, a maximally diverse covering test set is required, whereas search and constraint solving tend to generate test sets with biased distributions. Our approach, Diversified Focused Testing (DFT), uses a search strategy inspired by GödelTest. We artificially inject parameters into the code branching conditions and use a bi-objective search algorithm to find diverse inputs by perturbing the injected parameters, while keeping the path conditions still satisfiable. Our results demonstrate that our technique, DFT, is able to cover a desired point in the code at least 90% of the time. Moreover, adding diversity improves the bug detection and the mutation killing abilities of the test suites. We show that DFT achieves better results than focused testing, symbolic execution, and random testing by achieving from 3% to 70% improvement in mutation score and up to 100% improvement in fault detection across 105 software subjects.

Archive-based multi-criteria Artificial Bee Colony algorithm for whole test suite generation

Testing an object-oriented software is harder than testing a structural program due to inheritance, states, behaviour of different objects, association, and polymorphism properties of the object-oriented software, which is established based on classes. The classes should be carefully tested using predefined test cases to optimize coverage goals. Because generating test cases manually is labour-intensive, search-based software testing methods that maximize coverage criteria by a search algorithm can be used to generate test cases automatically. Artificial Bee Colony (ABC) algorithm is a powerful search tool that performs balanced exploration and exploitation. In this study, two new multi-criteria and combinatorial ABC algorithms using mutation and crossover operators are proposed to generate a test suite that maximizes a fitness function combining various goals for object-oriented software. First, an archive-based ABC algorithm is proposed to keep covered targets in the archive to use available search resource efficiently. Second, the archive-based ABC algorithm is improved to bring more diversity in the population. To investigate the effect of the archive, the basic ABC algorithm and the archive-based ABC algorithms are compared on a set of classes from SF110 corpus. To validate efficiency of the proposed methods, they are compared to state-of-the-art evolutionary test suite generators and random testing methodology on the classes divided into difficulty levels based on their weight method class values. The experimental results reveal that introducing an archive into the ABC algorithm provides fast convergence compared to the basic ABC and the improved archive-based ABC produces higher coverage for the problems with both low and high difficulty levels.

Handling SQL Databases in Automated System Test Generation

Automated system test generation for web/enterprise systems requires either a sequence of actions on a GUI (e.g., clicking on HTML links and form buttons) or direct HTTP calls when dealing with web services (e.g., REST and SOAP). When doing white-box testing of such systems, their code can be analyzed, and the same type of heuristics (e.g., the branch distance ) used in search-based unit testing can be employed to improve performance. However, web/enterprise systems do often interact with a database. To obtain higher coverage and find new faults, the state of the databases needs to be taken into account when generating white-box tests. In this work, we present a novel heuristic to enhance search-based software testing of web/enterprise systems, which takes into account the state of the accessed databases. Furthermore, we enable the generation of SQL data directly from the test cases. This is useful when it is too difficult or time consuming to generate the right sequence of events to put the database in the right state. Also, it is useful when dealing with databases that are “read-only” for the system under test, and the actual data are generated by other services. We implemented our technique as an extension of E VO M ASTER , where system tests are generated in the JUnit format. Experiments on six RESTful APIs (five open-source and one industrial) show that our novel techniques improve coverage significantly (up to +16.5%), finding seven new faults in those systems.

Big data driven genetic improvement for maintenance of legacy software systems

Big data driven genetic improvement for maintenance of legacy software systems

Test Suit Generation for Object Oriented Programs: A Hybrid Firefly and Differential Evolution Approach

In model-based testing, the test suites are derived from design models of system specification documents instead of actual program codes to reduce cost and time of testing. In search-based software testing approach, the nature inspired meta-heuristic search algorithms are used for automating and optimizing the test suite generation process of software testing. This paper proposes a concrete model-based testing framework; using UML behavioral state chart model along with the hybrid version of the two most popular nature inspired algorithms, Firefly algorithm (FA) and Differential Algorithm (DE). The hybrid algorithm is adopted to generate optimized test suits for the benchmark triangle classification problem. Experimental results evidently show that the hybrid FA-DE search algorithm outperforms the individual model-based Firefly and Differential Evolution algorithm's performances in terms of time complexity, better exploration and exploitation as well as variations in test case generation process. The framework generates optimized test data for complete transition path coverage of the available feasible paths of the example problem.

A systematic mapping addressing Hyper-Heuristics within Search-based Software Testing

Search-based Software Testing (SBST) is a research field where testing a software product is formulated as an optimization problem . It is an active sub-area of Search-based Software Engineering (SBSE) where many studies have been published and some reviews have been carried out. The majority of studies in SBST has been adopted meta-heuristics while hyper-heuristics have a long way to go. Moreover, there is still a lack of studies to perceive the state-of-the-art of the use of hyper-heuristics within SBST. The objective of this work is to investigate the adoption of hyper-heuristics for Software Testing highlighting the current efforts and identifying new research directions. A Systematic mapping study was carried out with 5 research questions considering papers published up to may/2019, and 4 different bases. The research questions aims to find out, among other things, what are the hyper-heuristics used in the context of Software Testing, for what problems hyper-heuristics have been applied, and what are the objective functions in the scope of Software Testing. A total of 734 studies were found via the search strings and 164 articles were related to Software Testing. However, from these, only 26 papers were actually in accordance with the scope of this research and 3 more papers were considered due to snowballing or expert’s suggestion, totalizing 29 selected papers. Few different problems and application domains where hyper-heuristics have been considered were identified. Differently from other communities (Operational Research, Artificial Intelligence), SBST has little explored the benefits of hyper-heuristics which include generalization and less difficulty in parameterization. Hence, it is important to further investigate this area in order to alleviate the effort of practitioners to use such an approach in their testing activities.

Genetic algorithm based test data generation for MPI parallel programs with blocking communication

Parallel computing is one of mainstream techniques for high-performance computation in which MPI parallel programs have gained more and more attention. Genetic algorithms (GAs) have been widely employed in automated test data generation, leading to a major family of search-based software testing techniques. However, previous GA-based methods have limitations when testing MPI parallel programs with blocking communication. In this paper, we focus on the path coverage problem for MPI parallel programs with blocking communication, and formulate the problem as an optimization problem with its decision variable being the program input and the execution order of sending nodes. In addition, we develop target amending strategies for candidates when solving the problem using genetic algorithms. The proposed method is evaluated and compared with several state-of-the-art methods through a series of controlled experiments on five typical programs. The experimental results show that the proposed method can effectively and efficiently generate test data for path coverage.

A Bat-Inspired Testing Strategy for Generating Constraints Pairwise Test Suite

Indubitably, exhaustive testing is impractical in real world. Several sampling strategies have been introduced to minimize test data systematically. Complementing existing interaction based sampling strategies, pairwise testing has been extensively exploited resulting into many prototypes strategy implementations. Recently, the growing interests of adopting optimization algorithms as a backbone search engine for the new and upcoming search based software testing is increasing for newly developed test suite generation methods. This paper proposes an enhancement design and implementation of BTS strategy for constraints pairwise test generation based on the bat-inspired algorithm (BA). The benchmarking results of BTS show that it outperforms the generated test suite of the existing tools and strategies even in the presence of constraints.