Automatic Test Data Generation Research Articles

Observations in using parallel and sequential evolutionary algorithms for automatic software testing

In this paper we analyze the application of parallel and sequential evolutionary algorithms (EAs) to the automatic test data generation problem. The problem consists of automatically creating a set of input data to test a program. This is a fundamental step in software development and a time consuming task in existing software companies. Canonical sequential EAs have been used in the past for this task. We explore here the use of parallel EAs. Evidence of greater efficiency, larger diversity maintenance, additional availability of memory/CPU, and multi-solution capabilities of the parallel approach, reinforce the importance of the advances in research with these algorithms. We describe in this work how canonical genetic algorithms (GAs) and evolutionary strategies (ESs) can help in software testing, and what the advantages are (if any) of using decentralized populations in these techniques. In addition, we study the influence of some parameters of the proposed test data generator in the results. For the experiments we use a large benchmark composed of twelve programs that includes fundamental algorithms in computer science.

Goal-oriented test data generation for pointer programs

Automatic test data generation leads to the identification of input values on which a selected path or a selected branch is executed within a program (path-oriented vs. goal-oriented methods). In both cases, several approaches based on constraint solving exist, but in the presence of pointer variables only path-oriented methods have been proposed. Pointers are responsible for the existence of conditional aliasing problems that usually provoke the failure of the goal-oriented test data generation process. In this paper, we propose an overall constraint-based method that exploits the results of an intraprocedural points-to analysis and provides two specific constraint combinators for automatically generating goal-oriented test data. This approach correctly handles multi-levels stack-directed pointers that are mainly used in C programs. The method has been fully implemented in the test data generation tool INKA and first experiences in applying it to a variety of existing programs are presented.

Empirical evaluation of optimization algorithms when used in goal-oriented automated test data generation techniques

Software testing is an essential process in software development. Software testing is very costly, often consuming half the financial resources assigned to a project. The most laborious part of software testing is the generation of test-data. Currently, this process is principally a manual process. Hence, the automation of test-data generation can significantly cut the total cost of software testing and the software development cycle in general. A number of automated test-data generation approaches have already been explored. This paper highlights the goal-oriented approach as a promising approach to devise automated test-data generators. A range of optimization techniques can be used within these goal-oriented test-data generators, and their respective characteristics, when applied to these situations remain relatively unexplored. Therefore, in this paper, a comparative study about the effectiveness of the most commonly used optimization techniques is conducted.

Search-based software test data generation for string data using program-specific search operators

This paper presents a novel approach to automatic software test data generation, where the test data is intended to cover program branches which depend on string predicates such as string equality,...

Search‐based software test data generation for string data using program‐specific search operators

AbstractThis paper presents a novel approach to automatic software test data generation, where the test data is intended to cover program branches which depend on string predicates such as string equality, string ordering and regular expression matching. A search‐based approach is assumed and some potential search operators and corresponding evaluation functions are assembled. Their performance is assessed empirically by using them to generate test data for a number of test programs. A novel approach of using search operators based on programming language string operators and parameterized by string literals from the program under test is introduced. These operators are also assessed empirically in generating test data for the test programs and are shown to provide a significant increase in performance. Copyright © 2006 John Wiley & Sons, Ltd.

Avoiding coincidental correctness in boundary value analysis

In partition analysis we divide the input domain to form subdomains on which the system's behaviour should be uniform. Boundary value analysis produces test inputs near each subdomain's boundaries to find failures caused by incorrect implementation of the boundaries. However, boundary value analysis can be adversely affected by coincidental correctness---the system produces the expected output, but for the wrong reason. This article shows how boundary value analysis can be adapted in order to reduce the likelihood of coincidental correctness. The main contribution is to cases of automated test data generation in which we cannot rely on the expertise of a tester.

A scheme on automated test data generation and its evaluation

By analyzing some existing test data generation methods, a new automated test data generation approach was presented. The linear predicate functions on a given path was directly used to construct a linear constrain system for input variables. Only when the predicate function is nonlinear, does the linear arithmetic representation need to be computed. If the entire predicate functions on the given path are linear, either the desired test data or the guarantee that the path is infeasible can be gotten from the solution of the constrain system. Otherwise, the iterative refining for the input is required to obtain the desired test data. Theoretical analysis and test results show that the approach is simple and effective, and takes less computation. The scheme can also be used to generate path-based test data for the programs with arrays and loops.

A New Script Language Applicable To Symbolic Execution Systems

The problem of test data generation in software testing is well referenced. Of the various methods used in the literature, symbolic execution appears to be a promising approach. It can be used either for software verification or to facilitate the automated test data generation process. A number of symbolic systems that employ symbolic execution to generate test data have already been constructed. However, these systems analyze programs written in specific programming languages. Thus, although some of them use an internal representation for performing symbolic execution, each one can only deal with programs in a single language. In this paper a script language called SYMEXLAN is presented. It can be adapted in order to construct a general symbolic execution testing system independent of the language in which the software under test is written.

Automatic test data generation using genetic algorithm and program dependence graphs

The complexity of software systems has been increasing dramatically in the past decade, and software testing as a labor-intensive component is becoming more and more expensive. Testing costs often account for up to 50% of the total expense of software development; hence any techniques leading to the automatic generation of test data will have great potential to considerably reduce costs. Existing approaches of automatic test data generation have achieved some success by using evolutionary computation algorithms, but they are unable to deal with Boolean variables or enumerated types and they need to be improved in many other aspects. This paper presents a new approach utilizing program dependence analysis techniques and genetic algorithms (GAs) to generate test data. A set of experiments using the new approach is reported to show its effectiveness and efficiency based upon established criterion.

Evolutionary testing in the presence of loop-assigned flags

Evolutionary testing is an effective technique for automatically generating good quality test data. However, for structural testing, the technique degenerates to random testing in the presence of flag variables, which also present problems for other automated test data generation techniques. Previous work on the flag problem does not address flags assigned in loops.This paper introduces a testability transformation that transforms programs with loop--assigned flags so that existing genetic approaches can be successfully applied. It then presents empirical data demonstrating the effectiveness of the transformation. Untransformed, the genetic algorithm flounders and is unable to find a solution. Two transformations are considered. The first allows the search to find a solution. The second reduces the time taken by an order of magnitude and, more importantly, reduces the slope of the cost increase; thus, greatly increasing the complexity of the problem to which the genetic algorithm can be applied. The paper also presents a second empirical study showing that loop--assigned flags are prevalent in real world code. They account for just under 11% of all flags.

Search-based software test data generation: a survey

The use of metaheuristic search techniques for the automatic generation of test data has been a burgeoning interest for many researchers in recent years. Previous attempts to automate the test gene...

Search‐based software test data generation: a survey

AbstractThe use of metaheuristic search techniques for the automatic generation of test data has been a burgeoning interest for many researchers in recent years. Previous attempts to automate the test generation process have been limited, having been constrained by the size and complexity of software, and the basic fact that, in general, test data generation is an undecidable problem. Metaheuristic search techniques offer much promise in regard to these problems. Metaheuristic search techniques are high‐level frameworks, which utilize heuristics to seek solutions for combinatorial problems at a reasonable computational cost. To date, metaheuristic search techniques have been applied to automate test data generation for structural and functional testing; the testing of grey‐box properties, for example safety constraints; and also non‐functional properties, such as worst‐case execution time. This paper surveys some of the work undertaken in this field, discussing possible new future directions of research for each of its different individual areas. Copyright © 2004 John Wiley & Sons, Ltd.

Consistency techniques for interprocedural test data generation

This paper presents a novel approach for automated test data generation of imperative programs containing integer , boolean and/or float variables. It extends our previous work to programs with procedure calls and arrays. A test program (with procedure calls) is represented by an Interprocedural Control Flow Graph (ICFG). The classical testing criteria (statement, branch, and path coverage), widely used in unit testing, are extended to the ICFG. For path coverage, the specified path is transformed into a path constraint . Our previous consistency techniques, the core idea behind the solving of path constraints, have been extended to handle procedural calls and operations with arrays. For statement (and branch) coverage, paths reaching the specified node or branch are dynamically constructed. The search for suitable paths is guided by the interprocedural control dependences of the program. The search is also pruned by a new specialized consistency filter. Finally, test data are generated by the application of the proposed path coverage algorithm. A prototype has been implemented. Experiments show the feasibility of the approach.

Improved method to generate path-wise test data

Gupta et al., proposed a method, which is referred to as the Iterative Relaxation Method, to generate test data for a given path in a program by linearizing the predicate functions. In this paper, a model language is presented and the properties of static and dynamic data dependencies are investigated. The notions in the Iterative Relaxation Method are defined formally. The predicate slice proposed by Gupta et al. is extended to path-wise static slice. The correctness of the constructional algorithm is proved afterward. The improvement shows that the constructions of predicate slice and input dependency set can be omitted. The equivalence of systems of constraints generated by both methods is proved. The prototype of path-wise test data generator is presented in this paper. The experiments show that our method is practical, and fits the pathwise automatic generation of test data for both white-box testing and black-box testing.

AUTOMATED GENERATION OF TEST TRAJECTORIES FOR EMBEDDED FLIGHT CONTROL SYSTEMS

Automated generation of test cases is a prerequisite for fast testing. Whereas the research in automated test data generation addressed the creation of individual test points, test trajectory generation has attracted limited attention. In simple terms, a test trajectory is defined as a series of data points, with each (possibly multidimensional) point relying upon the value(s) of previous point(s). Many embedded systems use data trajectories as inputs, including closed-loop process controllers, robotic manipulators, nuclear monitoring systems, and flight control systems. For these systems, testers can either handcraft test trajectories, use input trajectories from older versions of the system or, perhaps, collect test data in a high fidelity system simulator. While these are valid approaches, they are expensive and time-consuming, especially if the assessment goals require many tests. We developed a framework for expanding a small, conventionally developed set of test trajectories into a large set suitable, for example, for system safety assurance. Statistical regression is the core of this framework. The regression analysis builds a relationship between controllable independent variables and closely correlated dependent variables, which represent test trajectories. By perturbing the independent variables, new test trajectories are generated automatically. Our approach has been applied in the safety assessment of a fault tolerant flight control system. Linear regression, multiple linear regression, and autoregressive techniques are compared. The performance metrics include the speed of test generation and the percentage of "acceptable" trajectories, measured by the domain specific reasonableness checks.

Evolutionary test environment for automatic structural testing

Testing is the most significant analytic quality assurance measure for software. Systematic design of test cases is crucial for the test quality. Structure-oriented test methods, which define test cases on the basis of the internal program structures, are widely used. A promising approach for the automation of structural test case design is evolutionary testing. Evolutionary testing searches test data that fulfil a given structural test criteria by means of evolutionary computation. In this work, an evolutionary test environment has been developed that performs fully automatic test data generation for most structural test methods. The introduction of an approximation level for fitness evaluation of generated test data and the definition of an efficient test strategy for processing test goals, increases the performance of evolutionary testing considerably.

Analysis of the constraint solver in UNA based test data generation

In a series of articles Gupta et al. develop a framework for automatic test data generation for computer programs. In general, their approach consists of a branch predicate collector, which derives a system of linear inequalities representing the branch predicates for a given path in the program. This system is solved using a solving technique of theirs called the Unified Numerical Approach (UNA) [5, 7]. In this paper we show that in contrast to traditional optimization methods the UNA is not bounded by the size of the solved system. Instead it depends on how input is composed. That is, even for very simple systems consisting of one variable we can easily get more than a thousand iterations. We will also give a formal proof that UNA does not always find a mixed integer solution when there is one. Finally, we suggest using some traditional optimization method instead, like the simplex method in combination with branch-and-bound and/or a cutting-plane algorithm as a constraint solver.

ATGen: automatic test data generation using constraint logic programming and symbolic execution†

AbstractThe verification and validation of software through dynamic testing is an area of software engineering where progress towards automation has been slow. In particular the automatic design and generation of test data remains, by and large, a manual activity. This is despite the high promises that the symbolic execution technique engendered when it was first proposed as a method for automatic test data generation. This paper presents an automatic test data generator based on constraint logic programming and symbolic execution. After reviewing the symbolic execution technique, approaches for the resolution of the technical difficulties that have so far prevented symbolic execution from reaching its full potential in the area of automatic test data generation are presented. ATGen, an automatic test data generator, based on symbolic execution and that uses constraint logic programming, is then discussed. Copyright © 2001 John Wiley & Sons, Ltd.

Generating software test data by evolution

This paper discusses the use of genetic algorithms (GAs) for automatic software test data generation. This research extends previous work on dynamic test data generation where the problem of test data generation is reduced to one of minimizing a function. In our work, the function is minimized by using one of two genetic algorithms in place of the local minimization techniques used in earlier research. We describe the implementation of our GA-based system and examine the effectiveness of this approach on a number of programs, one of which is significantly larger than those for which results have previously been reported in the literature. We also examine the effect of program complexity on the test data generation problem by executing our system on a number of synthetic programs that have varying complexities.

AN INTEGRATED CLASSIFICATION-TREE METHODOLOGY FOR TEST CASE GENERATION

This paper describes an integrated methodology for the construction of test cases from functional specifications using the classification-tree method. It is an integration of our extensions to the classification-hierarchy table, the classification tree construction algorithm, and the classification tree restructuring technique. Based on the methodology, a prototype system ADDICT, which stands for AutomateD test Data generation system using the Integrated Classification-Tree method, has been built.