Test Data Generation Research Articles

Genetic Algorithm Based Test Data Generation for Structured Software Testing

Conventional software testing is time consuming and expensive task. The techniques for automated test case generation of software is very important as it can reduce the time and cost of this process. Search based test data generation received lot of attention in last two decades for solving this problem. The well known metaheuristic search technique genetic algorithm is used to obtain a specific coverage in software testing. This technique automatically generates test data in order to obtain branch coverage in software testing. The authors first use a technique to identify the unique paths from a control flow graph and then apply genetic algorithm for test case generation. This paper defines the basic research issues of genetic algorithm like chromosome, fitness function, crossing over, mutation in this problem domain and uses a case study to illustrate it.

Multiple-path testing for cross site scripting using genetic algorithms

Use path coverage as a test coverage criteria.Formalize the problem of test data generation as a search problem.Focus on cross site scripting, in short XSS, type of vulnerability.Propose an approach that uses genetic algorithms (GA) along with a database of XSS vulnerability patterns to generate multiple-path coverage for a script under test (SUT) against XSS vulnerability.Design genetic algorithms to allow the generation of multiple test data, in one run, to cover multiple vulnerable paths. Web applications suffer from different security vulnerabilities that could be exploited by hackers to cause harm in a variety of ways. A number of approaches have been proposed to test for such vulnerabilities. However, some gaps are still to be addressed. In this paper, we address one of such gaps: the problem of automatically generating test data (i.e., possible attacks) to test for cross site scripting (XSS) type of vulnerability. The objective is to generate a set of test data to exercise candidate security-vulnerable paths in a given script. The desirable set of test data must be effective in the sense that it uncovers whether any path can indeed be exploited to launch an attack. We designed a genetic algorithm-based test data generator that uses a database of XSS attack patterns to generate possible attacks and assess whether the attack is successful. We considered different types of XSS vulnerability: stored, reflected and DOM based. We empirically validated our test data generator using case studies of Web applications developed using PHP and MySQL. Empirical results show that our test data generator is effective in generating, in one run, multiple test data to cover multiple target paths.

Automated Expected Output Generation

What is the importance of automated test data generation? This question is of concern for the entire software community. We want to know the meaning and usefulness of automated test data generation. Is it related to design of test cases or execution of already designed test cases or both? How can we generate expected output using automated test data generation technique? This is a weak area and limits the applicability of automated testing. The test case creation, execution and evaluation are three distinct activities. The execution and evaluation of test cases can be easily automated, but creation needs expected output that is difficult to automate due to its dependency on functionality of the program. This paper addresses the significance of techniques and tools available for automated test data generation. In this work, we analyze and assess the existing techniques for automated expected output generation and discuss the practical significance of these techniques.

IMPROVING THE RELIABILITY AND VALIDITY OF TEST DATA ADEQUACY IN PROGRAMMING ASSESSMENTS

Automatic Programming Assessment (or APA) has recently become a notable method in assisting educators of programming courses to automatically assess and grade students’ programming exercises as its counterpart; the typical manual tasks are prone to errors and lead to inconsistency. Practically, this method also provides an alternative means of reducing the educators’ workload effectively. By default, test data generation process plays an important role to perform a dynamic testing on students’ programs. Dynamic testing involves the execution of a program against different inputs or test data and the comparison of the results with the expected output, which must conform to the program specifications. In the software testing field, there have been diverse automated methods for test data generation. Unfortunately, APA rarely adopts these methods. Limited studies have attempted to integrate APA and test data generation to include more useful features and to provide a precise and thorough quality program testing. Thus, we propose a framework of test data generation known as FaSt-Gen covering both the functional and structural testing of a program for APA. Functional testing is a testing that relies on specified functional requirements and focuses the output generated in response to the selected test data and execution, Meanwhile, structural testing looks at the specific program logic to verify how it works. Overall, FaSt-Gen contributes as a means to educators of programming courses to furnish an adequate set of test data to assess students’ programming solutions regardless of having the optimal expertise in the particular knowledge of test cases design. FaSt-Gen integrates the positive and negative testing criteria or so-called reliable and valid test adequacy criteria to derive desired test data and test set schema. As for the functional testing, the integration of specification-derived test and simplified boundary value analysis techniques covering both the criteria. Path coverage criterion guides the test data selection for structural testing. The findings from the conducted controlled experiment and comparative study evaluation show that FaSt-Gen improves the reliability and validity of test data adequacy in programming assessments.

Automatic path-oriented test data generation by boundary hypercuboids

Designing test cases and generating data are very important phases in software engineering these days. In order to generate test data, some generators such as random test data generators, data specification generators and path-oriented (Path-Wise) test data generators are employed. One of the most important problems in the path-oriented test data generator is the lack of attention given to discovering faults by the test data. In this paper an approach is proposed to generate some test data automatically so that we can realize the goal of discovering more faults in less time. The number of faults near the boundaries of the input domain is more than the center, according to the Pareto 80–20 principle the test data of this approach will be generated at 20% of the allowable area boundary. To do this, we extracted the boundary hypercuboids and then the test data will be generated by exploiting these hypercuboids. The experimental results show that the fault detection probability and the fault detection speed are improved significantly, compared with the previous approaches. By generating data in this way, more faults are discovered in a short period of time which makes it more possible to deliver products on time.

Search-Based Test Data Generator for Data-Flow Dependencies Using Dominance Concepts, Branch Distance and Elitism

Software testing is a complex and expensive activity of the software development life cycle. Software testing includes test data generation according to a test adequacy criterion. The use of search-based techniques has been the focus of researchers to automate the process of software test data generation for structural control-flow criteria. Automating test data generation remains a challenging problem for more robust adequacy criterion such as satisfying data-flow dependencies of a program. This study proposes a search-based approach that generates test data for data-flow dependencies of a program using dominance concepts, branch distance, and elitism. Genetic algorithm is used for the proposed approach and Gray encoding is used to encode test data. A set of subject programs is taken from the research literature to evaluate efficiency and effectiveness of the proposed approach. For the proposed approach, the measures considered are the mean number of generations and mean percentage coverage achieved. The performance of the proposed approach is evaluated by comparing the results with those of random search and earlier studies on data-flow testing. Over several experiments, it is shown that the proposed approach performed significantly better than random search and earlier studies with respect to data-flow test data generation and optimization. There is an increasing performance gap for more complex subject programs.

User behavior based traffic emulator: A framework for generating test data for DPI tools

Deep Packet Inspection (DPI) engines rely highly on the operation environment i.e., the traffic mix they supposed to work with. A well performing DPI engine requires real-world traffic mixes to be tested on. Due to privacy issues real-world traffic is usually only available at the site of the network operator at a secure measurement point. Furthermore, in order to make signature update, performance tweaks, etc. of the DPI engine, real-like measurements are essential. In this paper we present a traffic generation framework that provides up-to-date traffic mixes continuously. The basic idea of the framework is to generate traffic based on automatic user behavior emulation. Real-world traffic measurements are processed to analyze and extract the most typical user behavior scenarios. Our proposed method uses these typical user behaviors for emulation of users on remote controlled hosts while the network traffic of the user equipment is recorded. As a final step, the framework can build high-speed multiplexed traces from the recorded data which mimic the behavior of real traffic. The characteristics of the constructed traffic compared to real world traffic measurements are also evaluated in the paper showing that the framework is able to generate realistic traffic traces that are both suitable for DPI testing and can be publicly distributed without any privacy concerns. The proof of concept implementation of the presented system is open to the public [1].

Survey on Mutation-based Test Data Generation

<span>The critical activity of testing is the systematic selection of suitable test cases, which be able to reveal highly the faults. Therefore, mutation coverage is an effective criterion for generating test data. Since the test data generation process is very labor intensive, time-consuming and error-prone when done manually, the automation of this process is highly aspired. The researches about automatic test data generation contributed a set of tools, approaches, development and empirical results. In this paper, we will analyse and conduct a comprehensive survey on generating test data based on mutation. The paper also analyses the trends in this field.</span>

Fast curve fitting algorithm for parameter evaluation in lightning impulse test technique

This paper proposes a fast curve fitting technique for the evaluation of the base curve of lightning impulse voltage and current. The proposed method is based on the waveform parameter estimation employing a numerical integration and linear least square method. This method is derived from an ordinary differential equation. The proposed algorithm is able to fit the base curve of lightning impulse voltage and current. The formula of the base curve is in the complex form of two exponential functions. The proposed form is superior to the conventional real exponential form, since it can be rewritten in a real conventional form used for fitting the impulse voltage or in a damped/undamped sinusoidal form with phase shift for fitting the impulse current. The decomposition base curve procedure was tested with some impulse voltage and current waveforms collected from the test data generator attached with IEC 61083- 2 (2013). The waveform parameters evaluated by the proposed method are compared with those recommended by the standards. The proposed method shows the better performance in computation time than the conventional method recommended by the standards. Due to no requirement of iteration in the proposed curve fitting method, the computation time is much shorter than the conventional iterative method. Moreover, the utilization of the established method does not allow the recorded impulse waveform distortion. Besides, the developed algorithm technique can be done easily with markedly accuracy and noise immunity. For the aforementioned reasons, there is no doubt that the proposed technique is a superior one for impulse parameter evaluation.

Software Testing Case Generation of Ant Colony Optimization based on Quantum Dynamic Self-Adaptation

Automatic generation of testing data is the most crucial technology in the testing phase with some certain real value to improve software’s testing automation degree. Although some certain results have been achieved by introducing the ant colony optimization to the testing process, the algorithm itself has the defects like too fast convergence speed and being easy to fall into local optimum. Based on this, self-adaptive factor is introduced in this paper to balance the algorithm’s local optimum. Meanwhile, quantum algorithm is introduced to realize the individual renewal of the ant colony and reduce the ant colony’s size so as to further optimize the target problem. Simulation experiments show that in the multipath test, algorithm in this paper has achieved remarkable progress compared with the ant colony optimization and accuracy of automatic data’s optimal solution has been improved.

Does Automated Unit Test Generation Really Help Software Testers? A Controlled Empirical Study

Work on automated test generation has produced several tools capable of generating test data which achieves high structural coverage over a program. In the absence of a specification, developers are expected to manually construct or verify the test oracle for each test input. Nevertheless, it is assumed that these generated tests ease the task of testing for the developer, as testing is reduced to checking the results of tests. While this assumption has persisted for decades, there has been no conclusive evidence to date confirming it. However, the limited adoption in industry indicates this assumption may not be correct, and calls into question the practical value of test generation tools. To investigate this issue, we performed two controlled experiments comparing a total of 97 subjects split between writing tests manually and writing tests with the aid of an automated unit test generation tool, E vo S uite . We found that, on one hand, tool support leads to clear improvements in commonly applied quality metrics such as code coverage (up to 300% increase). However, on the other hand, there was no measurable improvement in the number of bugs actually found by developers. Our results not only cast some doubt on how the research community evaluates test generation tools, but also point to improvements and future work necessary before automated test generation tools will be widely adopted by practitioners.

Test Generation for Programs with Binary Tree Structure as Input

Test data generation is a process of creating program inputs that satisfy specific testing criteria. Many works have been focused on test generation with respect to numeric and string data. Dynamic data structures, such as trees and linked lists, have been widely used in modern programming, but on which there are few studies presented. In general, generating a dynamic data structure is associated with a proper shape and valid values generation. It would be difficult to generate such dynamic data structures, as both shapes and values are necessary to be valid simultaneously. This paper focuses on binary tree structures and proposes a novel test generation approach that combines search based testing with constraint solving techniques. The approach creates the shapes of binary tree structures by using GA, and generates the values in their data fields by using constraint solving techniques. The experimental results show that the presented approach is promising and effective. Moreover, the studies investigate factors affecting the performance of the approach, and arrive at a conclusion that the test generation cost is cubic growing as the number of pointer constraints increases.

On the applicability of diploid genetic algorithms

The heuristic search processes like simple genetic algorithms help in achieving optimization but do not guarantee robustness so there is an immediate need of a machine learning technique that also promises robustness. Diploid genetic algorithms ensure consistent results and can therefore replace Simple genetic algorithms in applications such as test data generation and regression testing, where robustness is more important. However, there is a need to review the work that has been done so far in the field. It is also important to analyse the applicability of the premise of the dominance techniques applied so far in order to implement the technique. The work presents a systematic review of diploid genetic algorithms, examines the premise of the dominance relation used in different works and discusses the future scope. The work also discusses the biological basis of evaluating dominance. The work is important as the future of machine learning relies on techniques that are robust as well as efficient.

The application of iterative interval arithmetic in path-wise test data generation

Research of path-wise test data generation is crucial and challenging in the area of software testing, which can be formulated as a constraint satisfaction problem. In our previous research, a look-ahead search method has been proposed as the constraint solver for path-wise test data generation. This paper analytically studies interval arithmetic of the search method in detail, which enforces arc consistency, and introduces the iterative operator to improve it, aiming at detecting infeasible paths as well as shortening generation time. Experiments were conducted to compare the iterative operator with the classical look-ahead operator AC-3, and to compare the test data generation method using the iterative operator with some currently existing methods. Experimental results validate the effectiveness and practical deployment of the proposed iterative method, and demonstrate that it is applicable in engineering.

Constrained multi‐objective test data generation based on set evolution

A crucial task of software testing is the generation of high-quality test data, so as to find defects and errors during various periods of software development. However, existing coverage-based testing methods seldom consider the fault finding ability of the test data. This paper establishes a constrained multi-objective model of test data generation, so that the generated test suite has better spatial distribution on the basis of satisfying statement coverage criterion, and thereby enhance its error detection ability. In addition, the authors propose a genetic algorithm (GA) based on set evolution to solve the model. The experimental results show that the test data generated by the proposed model have higher fault finding ability than statement coverage testing and adaptive random testing; in addition, compared with conventional GAs, the proposed algorithm needs less execution time with the number of test data not increasing significantly.

Introduction to the special issue on Mutation Testing

Introduction to the special issue on Mutation Testing

Global tests for novelty

Outlier detection covers the wide range of methods aiming at identifying observations that are considered unusual. Novelty detection, on the other hand, seeks observations among newly generated test data that are exceptional compared with previously observed training data. In many applications, the general existence of novelty is of more interest than identifying the individual novel observations. For instance, in high-throughput cancer treatment screening experiments, it is meaningful to test whether any new treatment effects are seen compared with existing compounds. Here, we present hypothesis tests for such global level novelty. The problem is approached through a set of very general assumptions, making it innovative in relation to the current literature. We introduce test statistics capable of detecting novelty. They operate on local neighborhoods and their null distribution is obtained by the permutation principle. We show that they are valid and able to find different types of novelty, e.g. location and scale alternatives. The performance of the methods is assessed with simulations and with applications to real data sets.

Automatic Test Data Generation using Metaheuristic Cuckoo Search Algorithm

The proposed work emphasizes on the automated process of test data generation for unit testing of structured programs, targeting complete path coverage of the software under test. In recent years, Cuckoo Search (CS) has been successfully applied in many engineering applications because of its high convergence rate to the global solution. The authors motivated with the performance of Cuckoo search, utilized it to generate test suits for the standard benchmark problems, covering entire search space of the input data in less iterations. The experimental results reveal that the proposed approach covers entire search space generating test data for all feasible paths of the problem in few number of generations. It is observed that proposed approach gives promising results and outperforms other reported algorithms and it can be an alternative approach in the field of test data generation.

ユニットテストの入力データ生成のためのBit-vector logicに基づくCOBOL記号実行器

ユニットテストの入力データ生成のためのBit-vector logicに基づくCOBOL記号実行器

Improving the performance of OCL constraint solving with novel heuristics for logical operations: a search-based approach

A common practice to specify constraints on the Unified Modeling Language (UML) models is using the Object Constraint Language (OCL). Such constraints serve various purposes, ranging from simply providing precise meaning to the models to supporting complex verification and validation activities. In many applications, these constraints have to be solved to obtain values satisfying the constraints, for example, in the case of model-based testing (MBT) to generate test data for the purpose of generating executable test cases. In our previous work, we proposed novel heuristics for various OCL constructs to efficiently solve them using search algorithms. These heuristics are enhanced in this paper to further improve the performance of OCL constraint solving. We performed an empirical evaluation comprising of three case studies using three search algorithms: Alternating Variable Method (AVM), (1ź+ź1) Evolutionary Algorithm (EA), and a Genetic Algorithm (GA) and in addition Random Search (RS) was used as a comparison baseline. In the first case study, we evaluated each heuristics using carefully designed artificial problems. In the second case study, we evaluated the heuristics on various constraints of Cisco's Video Conferencing Systems defined to support MBT. Finally, the third case study is about EU-Rent Car Rental specification and is obtained from the literature. The results of the empirical evaluation showed that (1ź+ź1) EA and AVM with the improved heuristics significantly outperform the rest of the algorithms.