Test Data Generation Research Articles

Specification testing of agent-based simulation using property-based testing

The importance of Agent-Based Simulation (ABS) as scientific method to generate data for scientific models in general and for informed policy decisions in particular has been widely recognised. However, the important technique of code testing of implementations like unit testing has not generated much research interested so far. As a possible solution, in previous work we have explored the conceptual use of property-based testing. In this code testing method, model specifications and invariants are expressed directly in code and tested through automated and randomised test data generation. This paper expands on our previous work and explores how to use property-based testing on a technical level to encode and test specifications of ABS. As use case the simple agent-based SIR model is used, where it is shown how to test agent behaviour, transition probabilities and model invariants. The outcome are specifications expressed directly in code, which relate whole classes of random input to expected classes of output. During test execution, random test data is generated automatically, potentially covering the equivalent of thousands of unit tests, run within seconds on modern hardware. This makes property-based testing in the context of ABS strictly more powerful than unit testing, as it is a much more natural fit due to its stochastic nature.

Format-aware learn&fuzz: deep test data generation for efficient fuzzing

Appropriate test data are a crucial factor to succeed in fuzz testing. Most of the real-world applications, however, accept complex structure inputs containing data surrounded by meta-data which is processed in several stages comprising of the parsing and rendering (execution). The complex structure of some input files makes it difficult to generate efficient test data automatically. The success of deep learning to cope with complex tasks, specifically generative tasks, has motivated us to exploit it in the context of test data generation for complicated structures such as PDF files. In this respect, a neural language model (NLM) based on deep recurrent neural networks (RNNs) is used to learn the structure of complex inputs. To target both the parsing and rendering steps of the software under test (SUT), our approach generates new test data while distinguishing between data and meta-data that significantly improve the input fuzzing. To assess the proposed approach, we have developed a modular file format fuzzer, IUST-DeepFuzz. Our experimental results demonstrate the relatively high coverage of MuPDF code by our proposed fuzzer, IUST-DeepFuzz, in comparison with the state-of-the-art tools such as learn&fuzz, AFL, Augmented-AFL, and random fuzzing. In summary, our experiments with many deep learning models revealed the fact that the simpler the deep learning models applied to generate test data, the higher the code coverage of the software under test will be.

A novel method of grouping target paths for parallel programs

Genetic algorithms can be employed to automatically generate desired test data, with the advantage of freeing up manpower. For the path coverage criterion, the problem of test data generation needs to be transformed into an optimization problem before applying genetic algorithms. However, when the number of paths to be covered is large, the transformed optimization problem will be very complicated. Correspondingly, the difficulty of problem solving will be greatly increased. In view of this, the complex optimization problem is divided into a number of sub-optimization problems by grouping paths. However, the existing method of grouping paths has not fully taken the characteristic of multiple processes existing in a parallel program into consideration. As a result, inappropriate paths will be put into the same group, which heavily restricts the efficiency of test data generation. To overcome the above drawback, this study proposes a novel method of grouping paths. This method refines the measurement of the path similarity when grouping target paths, dynamically increases the number of benchmark paths in a group, and groups the remaining ones based on the similarity between a path and each of these benchmark paths, with the purpose of a large similarity between each pair of paths in the same group. The proposed method is applied to test nine typical programs, and compared with the method of randomly grouping paths and the existing method of grouping paths. The experimental results show that paths in the same group obtained by the proposed method have a larger similarity, which is beneficial to efficiently generating test data that satisfy the path coverage criterion.

Large-Scale Data Pollution with Apache Spark

Because of the increasing volume of autonomously collected data objects, duplicate detection is an important challenge in today's data management. To evaluate the efficiency of duplicate detection algorithms with respect to big data, large test data sets are required. Existing test data generation tools, however, are either not able to produce large test data sets or are domain-dependent which limits their usefulness to a few cases. In this paper, we describe a new framework that can be used to pollute a clean, homogeneous and large data set from an arbitrary domain with duplicates, errors and inhomogeneities. To prove its concept, we implemented a prototype which is built upon the cluster computing framework Apache Spark and evaluate its performance in several experiments.

Automatic Test Data Generation Using the Activity Diagram and Search-Based Technique

In software testing, generating test data is quite expensive and time-consuming. The manual generation of an appropriately large set of test data to satisfy a specified coverage criterion carries a high cost and requires significant human effort. Currently, test automation has come at the cost of low quality. In this paper, we are motivated to propose a model-based approach utilizing the activity diagram of the system under test as a test base, focusing on its data flow aspect. The technique is incorporated with a search-based optimization heuristic to fully automate the test data generation process and deliver test cases with more improved quality. Our experimental investigation used three open-source software systems to assess and compare the proposed technique with two alternative approaches. The experimental results indicate the improved fault-detection performance of the proposed technique, which was 11.1% better than DFAAD and 38.4% better than EvoSuite, although the techniques did not differ significantly in terms of statement and branch coverage. The proposed technique was able to detect more computation-related faults and tends to have better fault detection capability as the system complexity increases.

Generation and evaluation of synthetic cone penetration test (CPT) data using various spatial interpolation techniques

The cone penetration test (CPT) has been widely used in many geotechnical engineering applications, including soil identification and classification, and evaluation of different soil properties. However, the uncertainties associated with site variation are typical characteristics of subsurface soil conditions that cannot be ignored. Therefore, the effect of site variability on the correlated soil properties from collected field data, such as CPT data, obtained from discrete locations across the site needs to be evaluated. In this study, six well-established spatial interpolation techniques — ordinary kriging (OK), simple kriging (SK), universal kriging (UK), inverse distance weight (IDW), spline, and natural neighbor (NaN) — were investigated to evaluate the best interpolation method for incorporating site variability. Six CPT test sites were used to evaluate the performance of these spatial interpretation methods. For each site, CPT profiles at specified locations were generated using the different interpolation techniques, and the generated CPT profiles were compared with the measured CPT profiles. The best-fit line of measured versus predicted cone tip resistance (qc) values, mean bias factor (λ), coefficient of variation (COV), and root mean square error (RMSE) were calculated for each CPT profile and used as a criteria for evaluating the different spatial interpolation methods. The results of this study demonstrated the ability of these spatial interpolation methods for generating CPT profiles with good accuracy. The slope of best-fit line of measured versus predicted qcranges from 0.93 to 0.95, the mean of λ ranges from 0.90 to 0.98, and the COV ranges from 0.34 to 0.53. The IDW, OK, and SK showed the best spatial interpolation methods (in order) for four out of the six CPT sites. For site 4, OK, IDW, and SK showed the best spatial interpolation methods (in order); while for site 5, the three best spatial interpolation techniques are SK, IDW, and OK (in order).

Practical Constraint Solving for Generating System Test Data

The ability to generate test data is often a necessary prerequisite for automated software testing. For the generated data to be fit for their intended purpose, the data usually have to satisfy various logical constraints. When testing is performed at a system level, these constraints tend to be complex and are typically captured in expressive formalisms based on first-order logic. Motivated by improving the feasibility and scalability of data generation for system testing, we present a novel approach, whereby we employ a combination of metaheuristic search and Satisfiability Modulo Theories (SMT) for constraint solving. Our approach delegates constraint solving tasks to metaheuristic search and SMT in such a way as to take advantage of the complementary strengths of the two techniques. We ground our work on test data models specified in UML, with OCL used as the constraint language. We present tool support and an evaluation of our approach over three industrial case studies. The results indicate that, for complex system test data generation problems, our approach presents substantial benefits over the state-of-the-art in terms of applicability and scalability.

A feedback-directed method of evolutionary test data generation for parallel programs

Abstract Context: Genetic algorithms can be utilized for automatic test data generation. Test data are encoded as individuals which are evolved for a number of generations using genetic operators. Test data of a parallel program include not only the program input, but also the communication information between each pair of processes. Traditional genetic algorithms, however, do not make full use of information provided by a population’s evolution, resulting in a low efficiency in generating test data. Objective: This paper emphasizes the problem of test data generation for parallel programs, and presents a feedback-directed genetic algorithm for generating test data of path coverage. Method: Information related to a schedule sequence is exploited to improve genetic operators. Specifically, a scheduling sequence is evaluated according to how well an individual covers the target path. The probability of the crossover and mutation points being located in the region is determined based on the evaluation result, which prevents a good schedule sequence from being destroyed. If crossover and mutation are performed in the scheduling sequence, the location of crossover and mutation points is further determined according to the relationship between nodes to be covered and the scheduling sequence. In this way, the population can be evolved in a narrowed search space. Results: The proposed algorithm is applied to test 11 parallel programs. The experimental results show that, compared with the genetic algorithm without utilizing information during the population evolution, the proposed algorithm significantly reduces the number of generations and the time consumption. Conclusion: The proposed algorithm can greatly improve the efficiency in evolutionary test data generation.

Software Test Data Generation based on Path Testing using Genetic Algorithms

Test data generation is the task of constructing test cases for predicting the acceptability of novel or updated software. Test data could be the original test suite taken from previous run or imitation data generated afresh specifically for this purpose. The simplest way of generating test data is done randomly but such test cases may not be competent enough in detecting all defects and bugs. In contrast, test cases can also be generated automatically and this has a number of advantages over the conventional manual method. Genetic Algorithms, one of the automation techniques, are iterative algorithms and apply basic operations repeatedly in greed for optimal solutions or in this case, test data. By finding out the most error-prone path using such test cases one can reduce the software development cost and improve the testing efficiency. During the evolution process such algorithms pass on the better traits to the next generations and when applied to generations of software test data they produce test cases that are closer to optimal solutions. Most of the automated test data generators developed so far work well only for continuous functions. In this study, we have used Genetic Algorithms to develop a tool and named it TG-GA (Test Data Generation using Genetic Algorithms) that searches for test data in a discontinuous space. The goal of the work is to analyze the effectiveness of Genetic Algorithms in automated test data generation and to compare its performance over random sampling particularly for discontinuous spaces.

Diversity-Oriented Test Suite Generation for EFSM Model

In this article, test diversity has been suggested to be a valid way to improve test suite effectiveness. Extended finite state machine (EFSM) is a widely used formal model, but little attention is paid on the test suite generation with more diversity. EFSM test suite generation involves test paths generation and test data generation. Considering the discrepancy between test paths has a more crucial impact on the diversity of test suite, compared with the difference between test data, this article, therefore, mainly concerns the test paths generation with more diversity for EFSM models. Hence, the factors that influence the discrepancy between test paths are investigated. Then based on these factors, an integrated distance metric is designed to evaluate the dissimilarity between test paths, and a diversity measurement for EFSM test suite is presented. Furthermore, a diversity-oriented test suite generation (DOTSG) method is proposed where a dissimilarity-based fitness function and diversity-oriented update strategy are adopted in traditional coverage-oriented EFSM test suite generation (COTSG) by genetic algorithm. The experimental results show that, compared to COTSG, our DOTSG can not only generate more diverse test suite to satisfy a certain coverage criteria, improving the fault detection capability of the test suite, but also decrease the evolution time cost and the size of test suite generated.

Using likely invariants for test data generation

Various approaches have been developed to tackle the problem of automatic test data generation. Among them search-based methods use metaheuristic algorithms to guide search in the input space of the program under test. This paper presents a new approach for improving search-based test data generation methods. This approach is based on learning the relationships between program input values and program parts covered by those values. The learned relationships are used to accelerate achieving test coverage goals. We introduce the concepts of branch likely invariant and path likely invariant as the basis for the learning method. In addition, we utilize simple predicates (based on some predefined templates) over program input variables to generate better initial candidate solutions, and use the mutation of the mentioned predicates to cover unexplored program parts. The current version of the proposed approach only considers numeric and string input parameters. To evaluate the performance of the proposed approach, a series of experiments have been carried out on a number of different benchmark programs. Through experiments and analysis, we show that the proposed approach enhances the effectiveness of common search-based test data generation methods, in terms of the coverage percentage.

Automated Test Cases and Test Data Generation for Dynamic Structural Testing in Automatic Programming Assessment Using MC/DC

Automatic Programming Assessment (or APA) is known as a method to assist educators in executing automated assessment and grading on students’ programming exercises and assignments. Having to execute dynamic testing in APA, providing an adequate set of test data via a systematic process of test data generation is necessarily essential. Though researches respecting to software testing have proposed various significant methods to realize automated test data generation, it occurs that recent studies of APA rarely utilized these methods. Merely some of the limited studies appeared to resolve this circumstance, yet the focus on realizing test set and test data covering more thorough dynamic-structural testing are still deficient. Thus, we propose a method that utilizes MC/DC coverage criteria to support more thorough automated test data generation for dynamic-structural testing in APA (or is called DyStruc-TDG). In this paper, we reveal the means of deriving and generating test cases and test data for the DyStruc-TDG method and its verification concerning the reliability criteria (or called positive testing) of test data adequacy in programming assessments. This method offers a significant impact on assisting educators dealing with introductory programming courses to derive and generate test cases and test data via APA regardless of having knowledge of designing test cases mainly to execute structural testing. As regards to this, it can effectively reduce the educators’ workload as the process of manual assessments is typically prone to errors and promoting inconsistency in marking and grading.

Implementation of Sine Cosine Algorithm (SCA) for Combinatorial Testing

Before being released to the market, software should be screened to ensure that the quality assurance measurement goals have been attained. To achieve this, one of the types of testing sorts is combinatorial testing (CT) aimed at discovering the faults that occur by interacting with the software. A minimization strategy for test cases is indeed important for optimizing test cases and reducing time. As NP hard (where NP is a non-deterministic polynomial) is the problem of generating the minimum test suite of combinatorial interaction testing (CIT). this paper discusses the implementation, and validation of an efficient strategy for t-way testing. The main contribution of the sine cosine algorithm SCA is to show that the strategy was sufficiently competitive as compared to other strategies in terms of the generated test suite size. Unlike most paper. The main contribution of SCA is to show the generation of test data for a high coverage strength (t < 12).

An Algorithm for Finding the Optimal Path In Basis Path Testing using GABVIE Model

Software testing is one of the most vital factors in software development life cycle. It is mainly used for testing the program code, known as white box testing and to test the functionality of the program, known as black box testing. Manual generation of test data is very costly, error vulnerable and time consuming task. Subsequently, there is a need to make the process automated as could be expected under the circumstances. This paper presented the automated generation of optimal path with intention of attaining the maximum coverage. The work being done considers the optimal path coverage in minimum cost. The task of generating test cases can be done through the concept Genetic Algorithm with importance of variable (GABVIE). The proposed algorithm additionally considers programs having numerous modules. This is vital as a large portion of the current test data generators not succeeded to establish the communication between the modules. The approach has been implemented on various program code and the outcomes got have been confirmed. The proposed work considers the white box testing.

A Dynamically-Reconfigurable Burst-Mode Link Using a Nanosecond Photonic Switch

Fast optical circuit switching has significant potential to solve scaling issues with traditional packet-switch-based networks and improve bandwidth and power efficiency in datacenter and high-performance computing applications. One difficultly in utilizing a fast optical switch is that the optical signal must be rapidly reacquired after each switching event so as to not significantly impact the overall throughput of the system. Here we present a system-level integration of a nanosecond-scale $2\times 2$ silicon photonic switch with a 25-Gbps burst-mode receiver that can lock in 31 ns. A novel FPGA-based control plane was used to generate test data, control the silicon photonic switch and burst-mode receiver, and measure bit errors at 12.5 and 20 Gbps. Error-free links (BER $ ) with system-level reconfiguration times below 60 ns (at 20 Gbps) and 90 ns (at 12.5 Gbps) including bit- and frame-synchronization are demonstrated.

A Search-Based Test Data Generation Method for Concurrent Programs

Concurrent programs are being widely adopted in development of multi-core and many-core processors. However, these types of programs present some features such as concurrency, communication and synchronization which make their testing more challenging than sequential programs. Search-based techniques, which use meta-heuristic search algorithms, have frequently been used for testing sequential programs, especially in the test data generation activity. However, application of search-based techniques in test data generation for concurrent programs has seldom been covered in the literature. The first contribution of this paper is to presenta search-based test data generation framework for concurrent programs. Additionally, a hybrid meta-heuristic algorithm, called SFLA-VND, is proposed, which could be used in the mentioned framework as well as other meta-heuristic algorithms. SFLA-VND is a combination of the shuffled frog leaping algorithm (SFLA) and the variable neighborhood descent (VND). The proposed framework has been experimented on five concurrent benchmark programs by applying genetic algorithm (GA), ant colony optimization (ACO), particle swarm optimization (PSO), SFLA and SFLA-VND. Experimental results demonstrate the effectiveness and efficiency of this framework. Also, the results confirm the superiority of SFLA-VND in comparison with some popular meta-heuristic algorithms, when they are used for test data generation.

Statistical testing data generation for UAS

The reliability of UAS(Unmanned Aircraft Systems) is of vital importance in practical applications. The most effective method to guarantee the reliability of UAS is to test UAS with multiple types of testing data. In this paper, we propose a method for the testing data generation of UAS in consideration of the combination and weight of the input data, which improves the availability of the method in UAS. Then we make four experiments to verify the effectiveness of the proposed method: (1) the random testing data generation experiment; (2) testing data generation considering the combination of input data experiment; (3) testing data generation considering the combination and weight of input data experiment; (4) the history data from the actual usage situation. Experiments results show that the method proposed in this paper is closer to the reality. At last, we analyse the code coverage by using Code Test System(CTS). And the analysis reflects that the proposed method has a better coverage than other methods.

Dynamic input domain reduction for test data generation with iterative partitioning

A major difficulty concerning test data generation for white box testing is to detect the domain of input variables covering a certain path. With this aim, a new concept, domain coverage, is introduced in this article. In search of appropriate input variable subdomains, covering a desired path, the domains are randomly partitioned as far as subdomains whose boundaries satisfy the path constraints are found. When partitioning, the priority is given to those subdomains whose boundary variables do not satisfy the path constraints. Representing the relation between the subdomains and their parents as a directed acyclic graph, an Euler/Venn reasoning system could be applied to select the most appropriate subdomains. To evaluate our proposed path oriented test data generation method, the results of applying the method to six known benchmark programs, triangle, GCD, calday, shellsort, quicksort and heapsort, is presented.

Testing Method for Software With Randomness Using Genetic Algorithm

The main task of software testing is the generation of test data with high quality in restricted time. But for software with uncertainties, such as randomness, existing methods of generating test data often lose their effectiveness. For software with randomness, a test datum might cover the test target in one run, but obtains different results in the next run. Therefore, special testing methods must be exploited for this kind of software. In order to test software with randomness, we first propose a novel test adequacy criterion, and then build a mathematical model for generating test data according to the new criterion. Finally, we present a method of solving the optimization model using a set-based genetic algorithm. We apply the proposed method to test 12 programs, and compare with traditional genetic algorithm and the random method. From the experimental results we can see that, the proposed adequacy criterion is available for the software with randomness and the proposed algorithm can effectively generate test data.

Multi-Task Optimization-Based Test Data Generation for Mutation Testing via Relevance of Mutant Branch and Input Variable

Mutation testing is a powerful software testing technique. However, it is difficult to obtain test data for killing a large number of mutants, especially for hard-to-kill mutants. Mutant branch is formed by an original statement of a program under test and its mutated statement. The true branch of a mutant branch is covered by a test datum, suggesting the corresponding mutant is killed under the criterion of weak mutation testing. This article focuses on efficiently generating test data for a large number of mutant branches. When generating test data using a search-based method, the size of the search domain is a determining factor affecting the search performance. The key observation is that only partial input variables affect whether a mutant will be killed, so the search domain can be reduced by deleting irrelevant variables. Along this line, we first group the mutant branches based on their relevant input variables, followed by formulating a multi-task optimization model of test data generation for the grouped mutant branches, in which the relevant input variables are taken as the decision variables. Finally, a multi-population genetic algorithm with individual sharing is employed to generate test data by multi-tasking. The experiments based on eight programs of various sizes show that removing related variable helps reduce the search domain, and the efficiency of test data generation by grouping and multitasking is improved.