Software Test Data Generation Research Articles

Search-Based Software Test Data Generation for Path Coverage Based on a Feedback-Directed Mechanism

Automatically generating test cases by evolutionary algorithms to satisfy the path coverage criterion has attracted much research attention in software testing. In the context of generating test cases to cover many target paths, the efficiency of existing methods needs to be further improved when infeasible or difficult paths exist in the program under test. This is because a significant amount of the search budget (i.e., time allocated for the search to run) is consumed when computing fitness evaluations of individuals on infeasible or difficult paths. In this work, we present a feedback-directed mechanism that temporarily removes groups of paths from the target paths when no improvement is observed for these paths in subsequent generations. To fulfill this task, our strategy first organizes paths into groups. Then, in each generation, the objective scores of each individual for all paths in each group are summed up. For each group, the lowest value of the summed up objective scores among all individuals is assigned as the best aggregated score for a group. A group is removed when no improvement is observed in its best aggregated score over the last two generations. The experimental results show that the proposed approach can significantly improve path coverage rates for programs under test with infeasible or difficult paths in case of a limited search budget. In particular, the feedback-directed mechanism reduces wasting the search budget on infeasible paths or on difficult target paths that require many fitness evaluations before getting an improvement.

Automated Software Test Data Generation With Generative Adversarial Networks

With the rapid increase of software scale and complexity, the cost of traditional software testing methods will increase faster than the scale of software. In order to improve test efficiency, it is particularly important to automatically generate high-quality test cases. This paper introduces a framework for automatic test data generation based on the generative adversarial network (GAN). GAN is employed to train a generative model over execution path information to learn the behavior of the software. Then we can use the trained generative model to produce new test data, and select the test data that can improve the branch coverage according to our proposed selection strategy. Compared to prior work, our proposed method is able to handle programs under test with large-scale branches without analyzing branch expressions. In the experiment, we exhibit the performance of our method by using two modules in GNU Scientific Library. In particular, we consider the application of our method in two testing scenarios; unit testing and integration testing, and conduct a series of experiments to compare the performance of three types of GAN models. Results indicate that the WGAN-GP shows the best performance in our framework. Compared with the random testing method, the WGAN-GP based framework improves the test coverage of five functions out of the seven in the unit testing.

Automation of software test data generation using genetic algorithm and reinforcement learning

Software testing is one of the most important methods of analyzing software quality assurance. This process is very time consuming and expensive and accounts for almost 50% of the software production cost. In addition to the cost problem, the nature of the test, which seeks errors in the program, is such that software engineers are not interested in doing the process, so we are looking to use automated methods to reduce the cost and time of the test. In the last decade, various methods have been introduced for the automatic test data generation, the purpose of which is to maximize the detection of errors by generating minimum amount of test data. The main issue in the test data generation process is to determine the input data of the program in such a way that it meets the specified test criterion. In this research, a structural method has been used in order to automate the process of test data generation considering the criterion of covering all finite paths. In structural methods, the problem is converted into a search problem and meta-heuristic algorithms are used to solve it. The proposed method in this paper is a memetic algorithm in which reinforcement learning is used as a local search method within a genetic algorithm. Experimental results have shown that this method is faster for test data generation than many existing evolutionary or meta-heuristic algorithms and can provide better coverage with fewer evaluations. Compared algorithms include: conventional genetic algorithm, a variety of improvements to the genetic algorithm, random search, particle swarm optimization, bees algorithm, ant colony optimization, simulated annealing, hill climbing, and tabu search.

Fitness for Solving SMCP Using Evolutionary Algorithm

Search based software engineering is a paradigm with prime focus to apply search techniques and meta-heuristics for solving various software engineering open NP hard problems. Evolutionary meta-heuristics techniques are already proven to provide optimized solutions to other software engineering problems like automated software test data generation, project estimation, class responsibility assignment to name few. Software module clustering is such an open problem of software engineering that cannot be solved in definite manner. For a given module dependency graph of software system, there exists large number of possible partitions. Identifying a good partition to cluster all modules of software system is an exhaustive search that cannot be carried out in finite manner. With search based techniques applying evolutionary algorithms, an optimized solution can be identified with evaluating goodness of a given partitions. Identifying a fitness function that can direct the search towards optimal solution is very critical. This paper discusses various types of fitness applicable to solve software module clustering problem.

Software test data generation technology based on polymorphic particle swarm evolutionary algorithm

Combinatorial testing is a statute-based software testing method that aims to select a small number of valid test cases from a large combinatorial space of software under test to generate a set of test cases with high coverage and strong error debunking ability. However, combinatorial test case generation is an NP-hard problem that requires solving the combinatorial problem in polynomial time, so a meta-heuristic search algorithm is needed to solve the problem. Compared with other meta-heuristic search algorithms, the particle swarm algorithm is more competitive in terms of coverage table generation scale and execution time. In this paper, we systematically review and summarize the existing research results on generating combinatorial test case sets using particle swarm algorithm, and propose a combinatorial test case generation method that can handle arbitrary coverage strengths by combining the improved one-test-at-a-time strategy and the adaptive particle swarm algorithm for the variable strength combinatorial test problem and the parameter selection problem of the particle swarm algorithm. To address the parameter configuration problem of the particle swarm algorithm, the four parameters of inertia weight, learning factor, population size and iteration number are reasonably set, which makes the particle swarm algorithm more suitable for the generation of coverage tables. For the inertia weights.

Software Test Data Generation based on Path Testing using Genetic Algorithms

Test data generation is the task of constructing test cases for predicting the acceptability of novel or updated software. Test data could be the original test suite taken from previous run or imitation data generated afresh specifically for this purpose. The simplest way of generating test data is done randomly but such test cases may not be competent enough in detecting all defects and bugs. In contrast, test cases can also be generated automatically and this has a number of advantages over the conventional manual method. Genetic Algorithms, one of the automation techniques, are iterative algorithms and apply basic operations repeatedly in greed for optimal solutions or in this case, test data. By finding out the most error-prone path using such test cases one can reduce the software development cost and improve the testing efficiency. During the evolution process such algorithms pass on the better traits to the next generations and when applied to generations of software test data they produce test cases that are closer to optimal solutions. Most of the automated test data generators developed so far work well only for continuous functions. In this study, we have used Genetic Algorithms to develop a tool and named it TG-GA (Test Data Generation using Genetic Algorithms) that searches for test data in a discontinuous space. The goal of the work is to analyze the effectiveness of Genetic Algorithms in automated test data generation and to compare its performance over random sampling particularly for discontinuous spaces.

Enhanced Optimizer Algorithm and its Application to Software Testing

ABSTRACT Optimisation algorithm is currently one of the most applicable techniques to solve real-world problems by finding the best solution from all feasible solutions in the search space. This paper proposes enhanced multiverse optimiser algorithm that is inspired from the physics multiverse theory. The proposed algorithm suggests an enhancement of multiverse optimiser algorithm . It enhances the performance of multiverse optimiser to find the global minimal value among search space and solve the problems in the multiverse optimiser algorithm. In order to confirm the performance of the suggested algorithm, it has been benchmarked with benchmark functions challenging optimisation problems. The proposed algorithm is compared with state-of-the-art optimisation algorithm to confirm its performance; it is being compared with particle swarm optimisation, sine cosine algorithm, grey wolf optimiser, moth-flame optimisation and multiverse optimiser. Also, the algorithm is applied on software testing and test data generation, the results of the benchmarked functions and the test data generation proves that the proposed algorithm is able to provide very competitive results and outperforms other compared algorithms over the tested cases.

Framework for cloud‐based software test data generation service

SummaryThis paper presents the framework of cloud‐based software test data generation service (CSTS) that caters to cost‐effective test data generation service in a cloud environment. In contrast to existing conventional or cloud‐based testing frameworks, CSTS has a number of unique benefits. First, CSTS is designed to facilitate test data generation in minimum time and cost. Second, unlike existing frameworks which mandates clients to opt for resources to test their jobs, CSTS guides customer for selecting best cluster configuration in order to minimize the cost. While the existing models do not provide any solution for trust establishment in cloud computing services, CSTS delivers it by implementing security mechanism with the provision of role based access control. The security mechanism proposed in this paper ensures the protection of data and code of different users. Third, CSTS provides a mathematical pricing model to fulfill the expectations of customers and also to maximize the net profit of service providers. Cloud service request model has also been designed that postulates service level agreements between customers and service providers. We have evaluated, compared, and analyzed our framework and have found that it outperforms other existing cloud‐based frameworks.

Large scale software test data generation based on collective constraint and weighted combination method

Large scale software test data generation based on collective constraint and weighted combination method

An accelerating PSO algorithm based test data generator for data-flow dependencies using dominance concepts

One of the most important and effort intensive activity of the entire software development process is software testing. The effort involved chiefly increases because of the need to obtain optimal test data out of the entire search space of the problem under testing. Software test data generation is one area that has seen tremendous research in terms of automation and optimization. Generating or identifying an optimal test set that satisfies a more robust adequacy criteria, like data flow testing, is still a challenging task. A number of heuristic and meta-heuristics like GA, PSO have been applied to optimize the test data generation problem. GA, although more popular, has its own difficulties such as complex to implement and slow convergence rate. In this paper an accelerating particle swarm optimization algorithm (APSO) is applied to generate test data for data-flow dependencies of a program guided by a new fitness function. APSO is used because of its capability of balancing in exploration and exploitation. A new fitness function is designed based on the concepts of dominance relations, weighted branch distance for APSO to guide the search direction. A set of benchmark programs and four modules of Krishna Institute of Engineering and Technology ERP system were taken for the experimental analysis. The experimental results show that the proposed APSO based approach performed significantly better than random search, genetic algorithm and PSO in enhancing the convergence speed.

An Adaptive PSO Algorithm Based Test Data Generator for Data-Flow Dependencies using Dominance Concepts

One of the most important and effort intensive activity of the entire software development process is software testing. The effort involved chiefly increases because of the need to obtain optimal test data out of the entire search space of the problem under testing. Software test data generation is one area that has seen tremendous research in terms of automation and optimization. Generating or identifying an optimal test set that satisfies a more robust adequacy criteria, like data flow testing, is still a challenging task. A number of heuristic and meta-heuristics like genetic algorithm (GA), Particle Swarm Optimization (PSO) have been applied to optimize the test data generation problem. GA, although more popular, has its own difficulties such as complex to implement and slow convergence rate. In this paper an Adaptive Particle Swarm Optimization (APSO) algorithm is applied to generate test data for data-flow dependencies of a program guided by a novel fitness function. Adaptive PSO is used because of its capability of balancing in exploration and exploitation. A new fitness function is designed based on the concepts of dominance relations, weighted branch distance for APSO to guide the search direction. A set of benchmark programs and four modules of Krishna Institute of Engineering and Technology (KIET), Enterprise resource planning (ERP) system were taken for the experimental analysis. The experimental results show that the proposed adaptive PSO based approach performed significantly better than random search, Genetic Algorithm and PSO in enhancing the convergence speed.

Comparisons of metaheuristic algorithms and fitness functions on software test data generation

Cost of testing activities is a major portion of the total cost of a software. In testing, generating test data is very important because the efficiency of testing is highly dependent on the data used in this phase. In search-based software testing, soft computing algorithms explore test data in order to maximize a coverage metric which can be considered as an optimization problem. In this paper, we employed some meta-heuristics (Artificial Bee Colony, Particle Swarm Optimization, Differential Evolution and Firefly Algorithms) and Random Search algorithm to solve this optimization problem. First, the dependency of the algorithms on the values of the control parameters was analyzed and suitable values for the control parameters were recommended. Algorithms were compared based on various fitness functions (path-based, dissimilarity-based and approximation level+branch distance) because the fitness function affects the behaviour of the algorithms in the search space. Results showed that meta-heuristics can be effectively used for hard problems and when the search space is large. Besides, approximation level+branch distance based fitness function is generally a good fitness function that guides the algorithms accurately.

Evolution or revolution: the critical need in genetic algorithm based testing

Software testing is one of the most inevitable processes in software development. The field of software testing has seen an extensive use of search based techniques in the last decade. Among the search based techniques, it is the metaheuristic techniques such as genetic algorithm that has garnered the major share of attention from researchers. Looking at the large body of work that has happened and is happening in this field, we feel that it is high time someone studied how well genetic algorithm based techniques fare in practical testing process. Method: In this work, we present a roadmap to the future of genetic algorithm based software testing, based on a review of literature. We have mainly reviewed the works which use genetic algorithm for software test data generation. This independent review is designed to direct the attention of future researchers to the deficiencies of genetic algorithm based testing, their possible solutions and the extent to which they are correctable. The observations from the selected primary studies highlight the issues faced when genetic algorithm is applied in software testing. The observations form the review reveal that the type of genetic algorithm used, fitness function design, population initialization and parameter settings does impact the quality of solution obtained in software testing using genetic algorithm. From the review we conclude that, more generalized approaches can make genetic algorithm based software testing one of the strongest methods in practical software testing. We hope that, this review will be a major breakthrough in genetic algorithm based software testing field.

RGA: A lightweight and effective regeneration genetic algorithm for coverage-oriented software test data generation

ContextGenetic algorithm (GA) is an important intelligent method in the area of automatic software test data generation. However, existing GAs tend to get trapped in the local optimal solution, leading to population aging, which can significantly reduce the benefits of GA-based software testing and increase cost and effort. Although much attention has been focused on solving this problem by improving chromosome population, genetic operations, and genetic parameters adjustment, the applicability of most of the algorithms proposed is often narrow because of the complex operations involved and nondeterminism inherited from traditional GAs. ObjectivesThis paper proposes a new algorithm called the regenerate genetic algorithm (RGA), which is based on a new simple, stable, and easy-to-implement regeneration strategy that involves judging the population aging process. MethodsWe propose a new regeneration strategy—called Regenerate Genetic Algorithm (RGA)—that solves these problems easily and effectively. The proposed strategy defines population aging factors and process in order to determine the degree of population aging. Subsequently, when population aging has reached a certain limit, a population regeneration operation is triggered. In contrast to other improved methods, the proposed regeneration strategy for population aging easily achieves a stronger ability to jump out of the local optimal solution, thereby preventing population aging and effectively improving test coverage, without modifying any parameter of the original GA. ResultsThe proposed algorithm is experimentally evaluated by comparing it to the basic GA, Random Testing (RT) and several other methods in terms of both efficiency and effectiveness on the Siemens Suite of test programs and a more complex real program. The results obtained indicate that the proposed algorithm can effectively increase search efficiency, restrain population aging, increase test coverage, and reduce the number of test cases. ConclusionRGA has better optimization ability than the conventional algorithms, especially for large-scale and highly complex programs.

Search-Based Test Data Generator for Data-Flow Dependencies Using Dominance Concepts, Branch Distance and Elitism

Software testing is a complex and expensive activity of the software development life cycle. Software testing includes test data generation according to a test adequacy criterion. The use of search-based techniques has been the focus of researchers to automate the process of software test data generation for structural control-flow criteria. Automating test data generation remains a challenging problem for more robust adequacy criterion such as satisfying data-flow dependencies of a program. This study proposes a search-based approach that generates test data for data-flow dependencies of a program using dominance concepts, branch distance, and elitism. Genetic algorithm is used for the proposed approach and Gray encoding is used to encode test data. A set of subject programs is taken from the research literature to evaluate efficiency and effectiveness of the proposed approach. For the proposed approach, the measures considered are the mean number of generations and mean percentage coverage achieved. The performance of the proposed approach is evaluated by comparing the results with those of random search and earlier studies on data-flow testing. Over several experiments, it is shown that the proposed approach performed significantly better than random search and earlier studies with respect to data-flow test data generation and optimization. There is an increasing performance gap for more complex subject programs.

Instance Generator and Problem Representation to Improve Object Oriented Code Coverage

Search-based approaches have been extensively applied to solve the problem of software test-data generation. Yet, test-data generation for object-oriented programming (OOP) is challenging due to the features of OOP, e.g., abstraction, encapsulation, and visibility that prevent direct access to some parts of the source code. To address this problem we present a new automated search-based software test-data generation approach that achieves high code coverage for unit-class testing. We first describe how we structure the test-data generation problem for unit-class testing to generate relevant sequences of method calls. Through a static analysis, we consider only methods or constructors changing the state of the class-under-test or that may reach a test target. Then we introduce a generator of instances of classes that is based on a family of means-of-instantiation including subclasses and external factory methods. It also uses a seeding strategy and a diversification strategy to increase the likelihood to reach a test target. Using a search heuristic to reach all test targets at the same time, we implement our approach in a tool, JTExpert, that we evaluate on more than a hundred Java classes from different open-source libraries. JTExpert gives better results in terms of search time and code coverage than the state of the art, EvoSuite, which uses traditional techniques.

Automated Software Test Data Generation Using Improved Search Procedure

Automated software test data generation is a complex and one of the most challenging task. Generation of test data manually is tedious and error-prone. The base algorithm, which has been improved in this paper involves redundant and un-necessary execution cycles. In this paper, we have proposed an efficient search procedure that reduces the time complexity of test data generation significantly. Our experiments conducted on varying sizes of source code showing ratio of execution time of automated test data generation using Basic Search Procedure and our Improved Search Procedure conclude that, with the increase of source code size, test data generation takes place in lesser time using our improved search procedure.

Application of meta-heuristic algorithms for automated software test data generation

The quality of a developed software system depends on factors such as reliability, maintainability, scalability etc. To deliver reliable software, it is necessary that it is defect free. In order to achieve this objective, the software system needs to be thoroughly tested. Since exhaustive testing is not possible, it is necessary to identify suitable test data that helps in improving the quality of a software product. Manual generation of test data is laborious and time consuming due to the presence of a huge number of decision nodes in a program. Meta-heuristic algorithms can help in such scenario by generating optimal test data automatically, from a very large search space of candidate solutions. In this paper, an attempt is made to automatically generate test data using five meta-heuristic algorithms. The proposed approach is applied for a case study bank ATM. Results show that meta-heuristics are efficient in generating optimal test data.

Assisting in search heuristics selection through multidimensional supervised classification: A case study on software testing

A fundamental question in the field of approximation algorithms, for a given problem instance, is the selection of the best (or a suitable) algorithm with regard to some performance criteria. A practical strategy for facing this problem is the application of machine learning techniques. However, limited support has been given in the literature to the case of more than one performance criteria, which is the natural scenario for approximation algorithms. We propose multidimensional Bayesian network (mBN) classifiers as a relatively simple, yet well-principled, approach for helping to solve this problem. Precisely, we relax the algorithm selection decision problem into the elucidation of the nondominated subset of algorithms, which contains the best. This formulation can be used in different ways to elucidate the main problem, each of which can be tackled with an mBN classifier. Namely, we deal with two of them: the prediction of the whole nondominated set and whether an algorithm is nondominated or not. We illustrate the feasibility of the approach for real-life scenarios with a case study in the context of Search Based Software Test Data Generation (SBSTDG). A set of five SBSTDG generators is considered and the aim is to assist a hypothetical test engineer in elucidating good generators to fulfil the branch testing of a given programme.

Search based software test data generation for structural testing

Software testing is an important and expensive phase of the software development life cycle. Over the past few decades, there has been an ongoing research to automate the process of software testing but the attempts have been constrained by the size and the complexity of software especially due to the use of dynamic memory allocation which makes the software behavior highly unpredictable. The use of metaheuristic global search techniques for software test data generation has been the focus of researchers in recent years. Many new techniques and hybrid methods have been proposed to tackle the problem more effectively. This study provides an overview of the various techniques that have been applied for structural test data generation. It also presents the open areas, challenges and future directions in the field of search based software testing with an emphasis on test data generation for structural testing.