Search-based Test Case Generation Research Articles

Knowledge transfer based many-objective approach for finding bugs in multi-path loops

AbstractGenerating test cases is essential for discovering software bugs. However, finding bugs in multi-path loops is challenging, especially when bugs can only be exposed after a specific interleaving execution order. Furthermore, the number of paths increases exponentially when the iteration count increases. It results in running out of resources (e.g., allocated search budget) in existing algorithms for test case generation before finding program bugs. To overcome these challenges and improve the effectiveness of existing algorithms in finding bugs among multi-path loops, we propose a path-oriented test case generation framework to explore loop paths of bounded depth. To fulfill this task, we first present a method that groups loop paths based on their latent similarities. Then, we formulate the problem of generating test cases to cover loop paths as a many-objective optimization one. The many-objective optimization problem is designed to optimize one group of loop paths at a time. Finally, a many-objective test case generation framework is used to generate test cases for covering loop paths. In this framework, a knowledge transfer scheme is incorporated to reuse suitable archived test cases at the start of the search process for a similar group of paths. We have implemented the proposed knowledge transfer scheme with generalized differential evolution 3 (KT-GDE3) and evaluated it with benchmarks from the competition on software verification (SV-COMP16 and SV-COMP21). The experimental results show that KT-GDE3 outperforms alternative search-based test case generation algorithms in bug finding capability for multi-path loops. To evaluate the effectiveness of the proposed knowledge transfer scheme, our approach is compared with the variant without this scheme (called KT-GDE3-unarchived) in the benchmarks. The corresponding experimental results verify our assumption that the knowledge transfer scheme can significantly enhance the optimization process of multi-objective algorithms such as generalized differential evolution when finding bugs in multi-path loops.

DTester: Diversity-Driven Test Case Generation for Web Applications

Search-based Test Case Generation (TCG) for web applications suffers from unstable performance and suboptimal test suite problems due to diversity loss. However, previous diversity metrics mainly only focus on client-side models or server-side code, which are prone to low robustness and poor generalization in practical applications. We propose a diversity-driven TCG method DTester, which can maximize behavior exploration and minimize the test suite size while covering more server-side vulnerable paths. Three diversity metrics (i.e. phenotypic coupling, intent coupling and competitiveness) are proposed to measure the underlying relationship between test cases from user behavior, code logic and test execution history. Moreover, a 3-dimensional weight graph is designed to model association among metrics, which provides fine-grained guidance for the genetic algorithm to generate diverse test cases from the client-side behavior model. Our empirical evaluation on five web applications shows that DTester can efficiently and robustly generate better test suites than the state-of-the-art TCG method. The maximum improvement is [Formula: see text], [Formula: see text], [Formula: see text] and [Formula: see text] in efficiency, test suite size, diversity and robustness.

A pragmatic approach for hyper-parameter tuning in search-based test case generation

Search-based test case generation, which is the application of meta-heuristic search for generating test cases, has been studied a lot in the literature, lately. Since, in theory, the performance of meta-heuristic search methods is highly dependent on their hyper-parameters, there is a need to study hyper-parameter tuning in this domain. In this paper, we propose a new metric (“Tuning Gain”), which estimates how cost-effective tuning a particular class is. We then predict “Tuning Gain” using static features of source code classes. Finally, we prioritize classes for tuning, based on the estimated “Tuning Gains” and spend the tuning budget only on the highly-ranked classes. To evaluate our approach, we exhaustively analyze 1,200 hyper-parameter configurations of a well-known search-based test generation tool (EvoSuite) for 250 classes of 19 projects from benchmarks such as SF110 and SBST2018 tool competition. We used a tuning approach called Meta-GA and compared the tuning results with and without the proposed class prioritization. The results show that for a low tuning budget, prioritizing classes outperforms the alternatives in terms of extra covered branches (10 times more than a traditional global tuning). In addition, we report the impact of different features of our approach such as search space size, tuning budgets, tuning algorithms, and the number of classes to tune, on the final results.

Enabledness-based Testing of Object Protocols

A significant proportion of classes in modern software introduce or use object protocols, prescriptions on the temporal orderings of method calls on objects. This article studies search-based test generation techniques that aim to exploit a particular abstraction of object protocols (enabledness preserving abstractions (EPAs)) to find failures. We define coverage criteria over an extension of EPAs that includes abnormal method termination and define a search-based test case generation technique aimed at achieving high coverage. Results suggest that the proposed case generation technique with a fitness function that aims at combined structural and extended EPA coverage can provide better failure-detection capabilities not only for protocol failures but also for general failures when compared to random testing and search-based test generation for standard structural coverage.

Simulink model testing method based on program mutation

In order to solve the current problems( expensive testing cost and low efficiency of test case generation) in mutation test for Simulink models,a mutation testing process and an optimized set of mutation operators were proposed for Simulink models based on program mutation according to the current mutation operators for the Simulink models. Experiments show that this set of mutation operators can effectively reduce the generation number of mutation models without prejudice to the mutation score of testing case set,thus it will effectively save the testing cost. Then a search-based test case generation method for Simulink models mutation testing was described. The test case generation problem was transformed into the objective function minimization problem,and the test cases which can kill the mutation models were ultimately obtained through the optimization of objective function by algorithm of simulated annealing. Finally,the application of a typical case for the method verified the correctness and effectiveness.

A Systematic Review of the Application and Empirical Investigation of Search-Based Test Case Generation

Metaheuristic search techniques have been extensively used to automate the process of generating test cases, and thus providing solutions for a more cost-effective testing process. This approach to test automation, often coined “Search-based Software Testing” (SBST), has been used for a wide variety of test case generation purposes. Since SBST techniques are heuristic by nature, they must be empirically investigated in terms of how costly and effective they are at reaching their test objectives and whether they scale up to realistic development artifacts. However, approaches to empirically study SBST techniques have shown wide variation in the literature. This paper presents the results of a systematic, comprehensive review that aims at characterizing how empirical studies have been designed to investigate SBST cost-effectiveness and what empirical evidence is available in the literature regarding SBST cost-effectiveness and scalability. We also provide a framework that drives the data collection process of this systematic review and can be the starting point of guidelines on how SBST techniques can be empirically assessed. The intent is to aid future researchers doing empirical studies in SBST by providing an unbiased view of the body of empirical evidence and by guiding them in performing well-designed and executed empirical studies.