Search-based Test Data Generation Research Articles

Using Static Information of Programs to Partition the Input Domain in Search-based Test Data Generation

The quality of test data has an important effect on the fault-revealing ability of software testing. Search-based test data generation reformulates testing goals as fitness functions, thus, test data generation can be automated by meta-heuristic algorithms. Meta-heuristic algorithms search the domain of input variables in order to find input data that cover the targets. The domain of input variables is very large, even for simple programs, while this size has a major influence on the efficiency and effectiveness of all search-based methods. Despite the large volume of works on search-based test data generation, the literature contains few approaches that concern the impact of search space reduction. In order to partition the input domain, this study defines a relationship between the structure of the program and the input domain. Based on this relationship, we propose a method for partitioning the input domain. Then, to search in the partitioned search space, we select ant colony optimization as one of the important and prosperous meta-heuristic algorithms. To evaluate the performance of the proposed approach in comparison with the previous work, we selected a number of different benchmark programs. The experimental results show that our approach has 14.40% better average coverage versus the competitive approach

Test data generation using genetic programming

Typically, search-based test data generation methods search on a population of program input values. Program input values can be regarded as solutions to underlying path constraints over program input parameters. One way to discover these path constraints is to use the symbolic execution method. Search-based methods attempt to find input values which are solutions to these path constraints, without knowing the actual constraints. In this paper, we show that we can search for the underlying path constraints using search-based methods, without resorting to symbolic execution. Trying to discover the exact or a good enough approximation of the underlying constraints may lead to a more targeted search, compared to directly searching for program input values. Besides, the construction of approximate constraints by searching may help to avoid some problems of symbolic execution. The proposed method uses genetic programming for learning constraints on program input parameters. To evaluate the performance of the proposed approach, a series of experiments have been conducted on a number of different benchmark programs. For 91.8% of benchmark programs, the proposed method achieved the best efficiency among the competitive algorithms. The results show that, if constraint solving can be provided for some or all parameter types of the methods of programs under test, our approach can improve the efficiency and effectiveness of search-based test data generation.

Using likely invariants for test data generation

Various approaches have been developed to tackle the problem of automatic test data generation. Among them search-based methods use metaheuristic algorithms to guide search in the input space of the program under test. This paper presents a new approach for improving search-based test data generation methods. This approach is based on learning the relationships between program input values and program parts covered by those values. The learned relationships are used to accelerate achieving test coverage goals. We introduce the concepts of branch likely invariant and path likely invariant as the basis for the learning method. In addition, we utilize simple predicates (based on some predefined templates) over program input variables to generate better initial candidate solutions, and use the mutation of the mentioned predicates to cover unexplored program parts. The current version of the proposed approach only considers numeric and string input parameters. To evaluate the performance of the proposed approach, a series of experiments have been carried out on a number of different benchmark programs. Through experiments and analysis, we show that the proposed approach enhances the effectiveness of common search-based test data generation methods, in terms of the coverage percentage.

A Search-Based Test Data Generation Method for Concurrent Programs

Concurrent programs are being widely adopted in development of multi-core and many-core processors. However, these types of programs present some features such as concurrency, communication and synchronization which make their testing more challenging than sequential programs. Search-based techniques, which use meta-heuristic search algorithms, have frequently been used for testing sequential programs, especially in the test data generation activity. However, application of search-based techniques in test data generation for concurrent programs has seldom been covered in the literature. The first contribution of this paper is to presenta search-based test data generation framework for concurrent programs. Additionally, a hybrid meta-heuristic algorithm, called SFLA-VND, is proposed, which could be used in the mentioned framework as well as other meta-heuristic algorithms. SFLA-VND is a combination of the shuffled frog leaping algorithm (SFLA) and the variable neighborhood descent (VND). The proposed framework has been experimented on five concurrent benchmark programs by applying genetic algorithm (GA), ant colony optimization (ACO), particle swarm optimization (PSO), SFLA and SFLA-VND. Experimental results demonstrate the effectiveness and efficiency of this framework. Also, the results confirm the superiority of SFLA-VND in comparison with some popular meta-heuristic algorithms, when they are used for test data generation.

Augmenting ant colony optimization with adaptive random testing to cover prime paths

Test data generation has a notable impact on the performance of software testing. A well-known approach to automate this activity is search-based test data generation. Most studies in this area use branch coverage as the test criterion. Since the prime path coverage criterion includes branch coverage, it has higher probability to detect software failures than the branch coverage criterion. This paper customizes and improves ant colony optimization (ACO) to provide a test data generation approach for covering prime paths. The proposed approach incorporates the notion of input space partitioning to maintain pheromone values in the search space. In addition, it employs the idea of adaptive random testing in the local search. At last, it uses the information of program predicates in order to make a relation between the logic of the program and pheromone values in the search space. The experimental results confirm the positive effects of the mentioned contributions, especially for programs with complex predicates. Furthermore, they represent that, on average, test suites generated by the proposed approach has 9% better mutation score in comparison to test suites produced by EvoSuite, a well-known test data generation tool.

Particle Swarm and Genetic Algorithm applied to mutation testing for test data generation: A comparative evaluation

Search based test data generation has gained popularity in recent times. Mutation testing can assist in improving the effectiveness of the generated test data. We recently proposed PSO-MT (Particle Swarm Optimization along with Mutation Testing) for generation of test data. In this paper, we fortify our proposal by applying the proposed approach on larger programs from Software-artifact Infrastructure Repository (SIR). PSO exhibits similar working characteristics with those of Genetic Algorithm (GA) which has extensively been applied for evolution of test data with mutation testing. The results are evaluated against comparison with GA used with mutation testing (GA-MT) for generation of test data which is already implemented in the literature of Search based Mutation Testing. The results depict that PSO-MT exhibits better computational efficiency than GA-MT for most of the benchmark programs. Statistical test (MannWhitney U-test) has been conducted to statistically analyze the presented results.

Optimizing Cost Function in Imperialist Competitive Algorithm for Path Coverage Problem in Software Testing

Search-based optimization methods have been used for software engineering activities such as software testing. In the field of software testing, search-based test data generation refers to application of meta-heuristic optimization methods to generate test data that cover the code space of a program. Automatic test data generation that can cover all the paths of software is known as a major challenge. The paper establishes a new cost function for automatic test data generation, which can traverse the non-iterative paths of software control flow graphs. This function is later compared with similar cost functions proposed in other articles. The results indicate the superior performance of the proposed function. Still another innovation in this paper is the application of the Imperialist Competitive Algorithm in automatic test data generation along with the proposed cost function. Automatic test data generation is implemented through the Imperialist Competitive Algorithm as well as the Genetic and Particle Swarm Optimization Algorithms for three software programs with different search space sizes. The algorithms are compared with each other in terms of convergence speed, computational time, and local search. Test data generated by the proposed method has achieved better results than other algorithms in finding the number of non-iterative paths, the convergence speed and computational time with growing the searching space of the software's control flow graph.

Generating Test Data for Software Structural Testing using Particle Swarm Optimization

Search-based test data generation is a very popular domain in the field of automatic test data generation. However, existing search-based test data generators suffer from some problems. By combining static program analysis and search-based testing, our proposed approach overcomes one of these problems. Considering the automatic ability and the path coverage as the test adequacy criterion, this paper proposes using Particle Swarm Optimization, an alternative search technique, for automating the generation of test data for evolutionary structural testing. Experimental results demonstrate that our test data generator can generate suitable test data has higher path coverage than the previous one.

An extensive evaluation of search-based software testing: a review

In recent years, search-based software testing (SBST) is the active research topic in software testing. SBST is the process of generating test cases that use metaheuristics for optimization of a task in the framework of software testing to solve difficult NP-hard problems. The best fitness results must be found with the heuristic search among many possibilities for a more cost-effective testing process and automate the process of generating test cases. Although search-based test data generation is a field of interest, some challenges remain unknown. The main objective of this survey is to find the main topics and trends in this emerging field of search-based software testing by examining the methods and the literature of software testing. A review of earlier studies of search-based software testing from the year 1996 to 2016 is discussed with the application of metaheuristics for the optimization of software testing.

Search-Based Junit Test Case Generation of Code Using Object Instances and Genetic Algorithm

To ensure software quality software testing is done .Test data generation is one of the more expensive parts of software testing. So, for reducing software cost and development time automation of test data generation is required. The object-oriented paradigm can be challenging for generating test data, due to some aspects of its features like abstract class, encapsulation, inheritance and visibility. To reach high code coverage of object oriented code search based test data generation technique is used. Proposed approach show that how efficiency and effectiveness of search based test data generation using static analysis. System takes input as different java class files then instances are generated for that classes then generate sequence of method call for whole code coverage then using genetic algorithm which is very useful and work efficiently when there is large search space it use to reach test target and finally generate test cases in Junit format which are helpful for all developers to perform unit testing.

Exploiting Binary Floating-Point Representations for Constraint Propagation

Floating-point computations are quickly finding their way in the design of safety- and mission-critical systems, despite the fact that designing floating-point algorithms is significantly more difficult than designing integer algorithms. For this reason, verification and validation of floating-point computations are hot research topics. An important verification technique, especially in some industrial sectors, is testing. However, generating test data for floating-point intensive programs proved to be a challenging problem. Existing approaches usually resort to random or search-based test data generation, but without symbolic reasoning it is almost impossible to generate test inputs that execute complex paths controlled by floating-point computations. Moreover, because constraint solvers over the reals or the rationals do not natively support the handling of rounding errors, the need arises for efficient constraint solvers over floating-point domains. In this paper, we present and fully justify improved algorithms for the propagation of arithmetic IEEE 754 binary floating-point constraints. The key point of these algorithms is a generalization of an idea by B. Marre and C. Michel that exploits a property of the representation of floating-point numbers.

Genetic Algorithm Based Test Data Generation for Structured Software Testing

Conventional software testing is time consuming and expensive task. The techniques for automated test case generation of software is very important as it can reduce the time and cost of this process. Search based test data generation received lot of attention in last two decades for solving this problem. The well known metaheuristic search technique genetic algorithm is used to obtain a specific coverage in software testing. This technique automatically generates test data in order to obtain branch coverage in software testing. The authors first use a technique to identify the unique paths from a control flow graph and then apply genetic algorithm for test case generation. This paper defines the basic research issues of genetic algorithm like chromosome, fitness function, crossing over, mutation in this problem domain and uses a case study to illustrate it.

Adapting ant colony optimization to generate test data for software structural testing

In general, software testing has been viewed as an effective way to improve software quality and reliability. However, the quality of test data has a significant impact on the fault-revealing ability of software testing activity. Recently, search-based test data generation has been treated as an operational approach to settle this difficulty. In the paper, the basic ACO algorithm is reformed into discrete version so as to generate test data for structural testing. First, the technical roadmap of combining the adapted ACO algorithm and test process together is introduced. In order to improve algorithm׳s searching ability and generate more diverse test inputs, some strategies such as local transfer, global transfer and pheromone update are defined and applied. The coverage for program elements is a special optimization objective, so the customized fitness function is constructed in our approach through comprehensively considering the nesting level and predicate type of branch. To validate the effectiveness of our ACO-based test data generation method, eight well-known programs are utilized to perform the comparative analysis. The experimental results show that our approach outperforms the existing simulated annealing and genetic algorithm in the quality of test data and stability, and is comparable to particle swarm optimization-based method. In addition, the sensitivity analysis on algorithm parameters is also employed to recommend the reasonable parameter settings for practical applications.

An approach for test data generation using program slicing and particle swarm optimization

Heuristic search-based test data generation has a potential higher efficiency in software testing with path covering. However, these approaches are suffered in covering the long and complex path. In this paper, we propose a method for generating test data based on program slicing and particle swarm optimization. With the interest points selected from a target path, we perform a program slicing to remove the statements which are irrelevant to the interest points. Our method simplifies the target path and the actual path to get a better fitness value. After program slices obtained, the population is evolved using particle swarm optimization to improve the efficiency of test data generation.

Generating Test Data for Software Structural Testing Based on Particle Swarm Optimization

Testing is an important way to ensure and improve the quality of software system. However, it is a time-consuming and labor-intensive activity. In the paper, our main concern is software structural testing, and propose a search-based test data generation solution. In our framework, particle swarm optimization (PSO) technique is adopted due to its simplicity and fast convergence speed. For test data generation problem, the inputs of program under test are encoded into particles. Once a set of test inputs is produced, coverage information can be collected by test driver. Meanwhile, the fitness value of branch coverage can be calculated based on such information. Then, the fitness is used for PSO to adjust the search direction. Finally, the test data set with the highest coverage rate is yielded. In addition, eight well-known programs are used for experimental analysis. The results show that PSO-based approach has a distinct advantage compared to the traditional evolutionary algorithms such as generic algorithm and simulated annealing, and also outperforms comprehensive learning-PSO both in covering effect and in evolution speed.

Dynamic stopping criteria for search-based test data generation for path testing

ContextEvolutionary algorithms have proved to be successful for generating test data for path coverage testing. However in this approach, the set of target paths to be covered may include some that are infeasible. It is impossible to find test data to cover those paths. Rather than searching indefinitely, or until a fixed limit of generations is reached, it would be desirable to stop searching as soon it seems likely that feasible paths have been covered and all remaining un-covered target paths are infeasible. ObjectiveThe objective is to develop criteria to halt the evolutionary test data generation process as soon as it seems not worth continuing, without compromising testing confidence level. MethodDrawing on software reliability growth models as an analogy, this paper proposes and evaluates a method for determining when it is no longer worthwhile to continue searching for test data to cover un-covered target paths. We outline the method, its key parameters, and how it can be used as the basis for different decision rules for early termination of a search. Twenty-one test programs from the SBSE path testing literature are used to evaluate the method. ResultsCompared to searching for a standard number of generations, an average of 30–75% of total computation was avoided in test programs with infeasible paths, and no feasible paths were missed due to early termination. The extra computation in programs with no infeasible paths was negligible. ConclusionsThe method is effective and efficient. It avoids the need to specify a limit on the number of generations for searching. It can help to overcome problems caused by infeasible paths in search-based test data generation for path testing.

Harmony search-based test data generation for branch coverage in software structural testing

Test data generation is always a key task in the field of software testing. In recent years, meta-heuristic search techniques have been considered as an effective way to assist test data generation in software structural testing. In this way, some representative test cases with high-coverage capability can be picked out from program input space. Harmony search (HS) is a recently developed algorithm and has been vigorously applied to various optimization problems. In the paper, we attempt to apply harmony search algorithm to generate test data satisfying branch coverage. At the preprocessing stage, the probes used for gathering coverage information are inserted into all branches via program static analysis. At the same time, the encoding and decoding styles between a test case and a harmony are also determined in advance. At the stage of test data searching, the subset of test data that has much stronger covering ability is stored in harmony memory. During the evolution process, one part of test suite is selected and adjusted from the harmony memory, and the other part is randomly generated from input space. Once a test suite is yielded after one-round search, its coverage can be measured by fitness function in our search algorithm. In our work, a new fitness function for branch coverage is constructed by comprehensively considering branch distance and branch weight. Here, the branch weight is determined by branch information in program, that is, the nesting level of a specific branch and the predicate types in it. Subsequently, the computed coverage metric is used for updating the test suite in the next round of searching. In order to validate the effectiveness of our proposed method, eight well-known programs are used for experimental evaluation. Experimental results show that the coverage of HS-based method is usually higher than those of other search algorithms, such as simulated annealing (SA) and genetic algorithm (GA). Meanwhile, HS demonstrates greater stability than SA and GA when varying the population size or performing repeated trials. That is to say, music-inspired HS algorithm is more suitable to generate test data for branch coverage in software structural testing.

TEST DATA GENERATION FOR SOFTWARE TESTING BASED ON QUANTUM-INSPIRED GENETIC ALGORITHM

The quality of test data has an important impact on the effect of software testing, so test data generation has always been a key task for finding the potential faults in program code. In structural testing, the primary goal is to cover some kinds of structure elements with some specific inputs. Search-based test data generation provides a rational way to handle this difficult problem. In the past, some well-known meta-heuristic search algorithms have been successfully utilized to solve this issue. In this paper, we introduce a variant of genetic algorithm (GA), called quantum-inspired genetic algorithm (QIGA), to generate the test data with stronger coverage ability. In this new algorithm, the traditional binary bit is replaced by a quantum bit (Q-bit) to enlarge the search space so as to avoid falling into local optimal solution. On the other hand, some other strategies such as quantum rotation gate and catastrophe operation are also used to improve algorithm efficiency and quality of test data. In addition, experimental analysis on eight real-world programs is performed to validate the effectiveness of our method. The results show that QIGA-based method can generate test data with higher coverage in much smaller convergence generations than GA-based method. More importantly, our proposed method is more robust for algorithm parameter change.

Generating Unit Tests for Floating Point Embedded Software using Compositional Dynamic Symbolic Execution

We present a method that would integrate both: compositional dynamic symbolic execution and search-based test data generation methods to achieve better code coverage for software that largely depends on floating point computations. We have implemented our method as an extension of a well-know symbolic execution engine – PEX. Our extension implements search-based testing as an optimization technique using AVM method. We present coverage comparison for several benchmark functions. Ill. 1, bibl. 17, tabl. 2 (in English; abstracts in English and Lithuanian).DOI: http://dx.doi.org/10.5755/j01.eee.122.6.1814

Input Domain Reduction through Irrelevant Variable Removal and Its Effect on Local, Global, and Hybrid Search-Based Structural Test Data Generation

Search-Based Test Data Generation reformulates testing goals as fitness functions so that test input generation can be automated by some chosen search-based optimization algorithm. The optimization algorithm searches the space of potential inputs, seeking those that are “fit for purpose,” guided by the fitness function. The search space of potential inputs can be very large, even for very small systems under test. Its size is, of course, a key determining factor affecting the performance of any search-based approach. However, despite the large volume of work on Search-Based Software Testing, the literature contains little that concerns the performance impact of search space reduction. This paper proposes a static dependence analysis derived from program slicing that can be used to support search space reduction. The paper presents both a theoretical and empirical analysis of the application of this approach to open source and industrial production code. The results provide evidence to support the claim that input domain reduction has a significant effect on the performance of local, global, and hybrid search, while a purely random search is unaffected.