Test Data Generation Research Articles

Improvements of Directed Automated Random Testing in Test Data Generation for C++ Projects

This paper improves the breadth-first search strategy in directed automated random testing (DART) to generate a fewer number of test data while gaining higher branch coverage, namely Static DART or SDART for short. In addition, the paper extends the test data compilation mechanism in DART, which currently only supports the projects written in C, to generate test data for C++ projects. The main idea of SDART is when it is less likely to increase code coverage with the current path selection strategies, the static test data generation will be applied with the expectation that more branches are covered earlier. Furthermore, in order to extend the test data compilation of DART for C++ context, the paper suggests a general test driver technique for C++ which supports various types of parameters including basic types, arrays, pointers, and derived types. Currently, an experimental tool has been implemented based on the proposal in order to demonstrate its efficacy in practice. The results have shown that SDART achieves higher branch coverage with a fewer number of test data in comparison with that of DART in practice.

SFLA‐based heuristic method to generate software structural test data

AbstractSoftware testing is one of the significant stages in software development life cycle which is a costly and time‐consuming task. Automatic tests data generation is one of the traditional techniques to reduce the cost and time spent in software testing. Different evolutionary algorithms have been proposed to generate test data which cover target paths in a software program. In this paper, shuffled frog leaping algorithm (SFLA) is proposed to generate structural test data. The proposed SFLA algorithm is characterized by high convergence speed and simple implementation. In the proposed SFLA, branch coverage is used as the fitness function to generate effective test data. For comparing the performance of the proposed SFLA with genetic algorithm (GA), particle swarm optimization (PSO), ant colony optimization (ACO), and artificial bee colony (ABC), seven benchmark programs were used. The results indicated that the proposed SFLA has an average of 99.99% for branch coverage, average 99.97% for success rate, and 2.03 for the average number of generation for covering all branches.

An approach to solving non-linear real constraints for symbolic execution

Constraint solvers are well-known tools for solving many real-world problems such as theorem proving and real-time scheduling. One of the domains that strongly relies on constraint solvers is the technique of symbolic execution for automatic test data generation. Many researchers have tried to alleviate the shortcomings of the available constraint solvers to improve their applications in symbolic execution for test data generation. Despite many recent improvements, constraint solvers are still unable to efficiently deal with certain types of constraints. In particular, constraints that include non-linear real arithmetic are among the most challenging ones. In this paper, we propose a new approach to solving non-linear real constraints for symbolic execution. This approach emphasizes transforming constraints into functions with specific properties, which are named Satisfaction Functions. A satisfaction function is generated in a way that by maximizing it, values that satisfy the corresponding constraint are obtained. We compared the performance of our technique with three constraint solvers that were known to be able to solve non-linear real constraints. The comparison was made regarding the speed and correctness criteria. The results showed that our technique was comparable with other methods regarding the speed criterion and outperformed these methods regarding the correctness criterion.

Scheduling sequence selection for generating test data to cover paths of MPI programs

Abstract Context: As one of key tasks in software testing, test data generation has been receiving widespread attention in recent years. Message-passing Interface (MPI) programs, which are one representative type of parallel programs, have the characteristic of non-determinism, which is reflected by the non-deterministic execution under different scheduling sequences against the same program input. Previous studies have shown that different difficulties are raised in generating test data under different scheduling sequences, suggesting that selecting appropriate scheduling sequences is beneficial to a high efficiency. Objective: We propose a method of selecting a superior and feasible scheduling sequence for generating test data in the criterion of path coverage against each target path of an MPI program. Method: In the proposed method, a number of program inputs are first sampled by Latin hypercube sampling. Then, the program is executed against each sample under each scheduling sequence, and all the scheduling sequences are sorted according to the similarities between the paths traversed by these samples and the target one. Finally, the feasibility of a scheduling sequence with the best quality is investigated based on the symbolic execution. Results: We apply the proposed method to seven typical MPI programs and compare it with the random one. The experimental results show that test data covering the target path can be generated under the selected scheduling sequence with high success rate and low time consumption. Conclusion: The proposed method takes the influence of scheduling sequences on generating test data into consideration, thus providing a competent way to test parallel programs.

Evolutionary algorithms for path coverage test data generation and optimization: a review

Software testing is very time consuming, labor-intensive and complex process. It is found that 50% of the resources of the software development are consumed for testing. Testing can be done in two different ways such as manual testing and automatic testing. Automatic testing can overcomes the limitations of manual testing by decreasing the cost and time of testing process. Path testing is the strongest coverage criteria among all white box testing techniques as it can detect about 65% of defects present in a SUT. With the help of path testing, the test cases are created and executed for all possible paths which results in 100% statement coverage and 100% branch coverage .This paper presents a systematic review of test data generation and optimization for path testing using Evolutionary Algorithms (EAs). Different EAs like GA, PSO, ACO, and ABCO based methods has been already proposed for automatic test case generation and optimization to achieve maximum path coverage.

An Analysis on the Applicability of Meta-Heuristic Searching Techniques for Automated Test Data Generation in Automatic Programming Assessment

Automatic Programming Assessment (APA) has been gaining lots of attention among researchers mainly to support automated grading and marking of students’ programming assignments or exercises systematically. APA is commonly identified as a method that can enhance accuracy, efficiency and consistency as well as providing instant feedback on students’ programming solutions. In achieving APA, test data generation process is very important so as to perform a dynamic testing on students’ assignment. In software testing field, many researches that focus on test data generation have demonstrated the successful of adoption of Meta-Heuristic Search Techniques (MHST) so as to enhance the procedure of deriving adequate test data for efficient testing. Nonetheless, thus far the researches on APA have not yet usefully exploited the techniques accordingly to include a better quality program testing coverage. Therefore, this study has conducted a comparative evaluation to identify any applicable MHST to support efficient Automated Test Data Generation (ATDG) in executing a dynamic-functional testing in APA. Several recent MHST are included in the comparative evaluation combining both the local and global search algorithms ranging from the year of 2000 until 2018. Result of this study suggests that the hybridization of Cuckoo Search with Tabu Search and lévy flight as one of promising MHST to be applied, as it’s outperforms other MHST with regards to number of iterations and range of inputs.

Framework for cloud‐based software test data generation service

SummaryThis paper presents the framework of cloud‐based software test data generation service (CSTS) that caters to cost‐effective test data generation service in a cloud environment. In contrast to existing conventional or cloud‐based testing frameworks, CSTS has a number of unique benefits. First, CSTS is designed to facilitate test data generation in minimum time and cost. Second, unlike existing frameworks which mandates clients to opt for resources to test their jobs, CSTS guides customer for selecting best cluster configuration in order to minimize the cost. While the existing models do not provide any solution for trust establishment in cloud computing services, CSTS delivers it by implementing security mechanism with the provision of role based access control. The security mechanism proposed in this paper ensures the protection of data and code of different users. Third, CSTS provides a mathematical pricing model to fulfill the expectations of customers and also to maximize the net profit of service providers. Cloud service request model has also been designed that postulates service level agreements between customers and service providers. We have evaluated, compared, and analyzed our framework and have found that it outperforms other existing cloud‐based frameworks.

Evolutionary generation of test data for EFSM based on irrelevant variable separation

Evolutionary generation of test data for EFSM based on irrelevant variable separation

Particle Swarm and Genetic Algorithm applied to mutation testing for test data generation: A comparative evaluation

Search based test data generation has gained popularity in recent times. Mutation testing can assist in improving the effectiveness of the generated test data. We recently proposed PSO-MT (Particle Swarm Optimization along with Mutation Testing) for generation of test data. In this paper, we fortify our proposal by applying the proposed approach on larger programs from Software-artifact Infrastructure Repository (SIR). PSO exhibits similar working characteristics with those of Genetic Algorithm (GA) which has extensively been applied for evolution of test data with mutation testing. The results are evaluated against comparison with GA used with mutation testing (GA-MT) for generation of test data which is already implemented in the literature of Search based Mutation Testing. The results depict that PSO-MT exhibits better computational efficiency than GA-MT for most of the benchmark programs. Statistical test (MannWhitney U-test) has been conducted to statistically analyze the presented results.

Unit Test Data Generation for C Using Rule-Directed Symbolic Execution

Unit testing is widely used in software development. One important activity in unit testing is automatic test data generation. Constraint-based test data generation is a technique for automatic generation of test data, which uses symbolic execution to generate constraints. Unit testing only tests functions instead of the whole program, where individual functions typically have preconditions imposed on their inputs. Conventional symbolic execution cannot detect these preconditions, let alone converting these preconditions into constraints. To overcome these limitations, we propose a novel unit test data generation approach using rule-directed symbolic execution for dealing with functions with missing input preconditions. Rule-directed symbolic execution uses predefined rules to detect preconditions in the individual function, and generates constraints for inputs based on preconditions. We introduce implicit constraints to represent preconditions, and unify implicit constraints and program constraints into integrated constraints. Test data generated based on integrated constraints can explore previously unreachable code and help developers find more functional faults and logical faults. We have implemented our approach in a tool called CTS-IC, and applied it to real-world projects. The experimental results show that rule-directed symbolic execution can find preconditions (implicit constraints) automatically from an individual function. Moreover, the unit test data generated by our approach achieves higher coverage than similar tools and efficiently mitigates missing input preconditions problems in unit testing for individual functions.

Genetic algorithm based test data generation for MPI parallel programs with blocking communication

Parallel computing is one of mainstream techniques for high-performance computation in which MPI parallel programs have gained more and more attention. Genetic algorithms (GAs) have been widely employed in automated test data generation, leading to a major family of search-based software testing techniques. However, previous GA-based methods have limitations when testing MPI parallel programs with blocking communication. In this paper, we focus on the path coverage problem for MPI parallel programs with blocking communication, and formulate the problem as an optimization problem with its decision variable being the program input and the execution order of sending nodes. In addition, we develop target amending strategies for candidates when solving the problem using genetic algorithms. The proposed method is evaluated and compared with several state-of-the-art methods through a series of controlled experiments on five typical programs. The experimental results show that the proposed method can effectively and efficiently generate test data for path coverage.

I love journal papers and you should too

I love journal papers and you should too

Recursive feature elimination in random forest classification supports nanomaterial grouping

Nanomaterials (NMs) can be produced in numerous different variants of the same chemical substance. An in-depth safety assessment for each variant by generating test data will simply not be feasible. Thus, NM grouping approaches that would significantly reduce the time and amount of testing for novel NMs are urgently needed. However, identifying structurally similar NM variants remains challenging as many physico-chemical properties could be relevant.Here, we aimed at emphasizing on the value of machine learning models in the process of NM grouping by considering a case study on eleven selected, well-characterized NMs. To that end, we linked physico-chemical properties of these NMs to characterized hallmarks for inhalation toxicity. We applied unsupervised and supervised machine learning techniques to determine which combination of properties is most predictive. First, we assessed NM similarity in an unsupervised manner using principal component analysis (PCA) followed by subsequent superposition of activity labels combined with a k-nearest neighbors approach. Then, we used random forests (RFs) as a supervised machine learning technique which directly uses the knowledge on the activity class in the process of defining NM similarity. Thus, similarity was defined only on those properties showing the highest correlation with the activity and therefore had the highest discriminative power. In order to improve the performance, we then used recursive feature elimination (RFE) to delete uninformative features biasing the results. The best performance was achieved by the reduced RF model based on RFE where a balanced accuracy of 0.82 was obtained. Out of eleven different properties we determined zeta potential, redox potential and dissolution rate to have the strongest predicting impact on biological NM activity in the present dataset. Though the dataset is too small with respect to the number of NMs studied and the applicability domain is expected to be very limited due to the fact that only few material classes were covered, our study demonstrates how machine learning and feature selection methods can be implemented for identifying the most relevant physico-chemical NM properties with respect to toxicity. We suggest that once the most relevant properties have been detected in a model built on a sufficient number of different NMs and across multiple NM classes, they should obtain special emphasis in future grouping approaches.

Process variable effect on the strength of autoclave-bonded film adhesive joints

This study investigates the effect of five independently-controlled process variables and variable combinations on the shear strength of autoclave-bonded film adhesive joints. Studied variables include the cure temperature, cure pressure and their respective ramp rates, as well as the duration of cure time. A full factorial design of experiment (DoE) at two levels for each variable is conducted with 3 replicas of each test. Test coupons are made of two layers of polycarbonate lexan that are autoclave-bonded using aliphatic polyether film adhesive (Huntsman PE399). Two set of test joints are used for generating test data on shear strength and failure mode. Bonded joints in the first set are tested prior to any environmental cycling, in order to generate baseline data on joint shear strength and failure mode. However, samples from the second set of autoclave-bonded joints were heat-cycled, in an environmental chamber at high relative humidity, prior to testing for shear strength. Test data on shear strength and failure mode is statistically analyzed using ANOVA.

Test Suite Optimization Using Firefly and Genetic Algorithm

Software testing is essential for providing error-free software. It is a well-known fact that software testing is responsible for at least 50% of the total development cost. Therefore, it is necessary to automate and optimize the testing processes. Search-based software engineering is a discipline mainly focussed on automation and optimization of various software engineering processes including software testing. In this article, a novel approach of hybrid firefly and a genetic algorithm is applied for test data generation and selection in regression testing environment. A case study is used along with an empirical evaluation for the proposed approach. Results show that the hybrid approach performs well on various parameters that have been selected in the experiments.

Automatic Classification of Program Paths Feasibility Using Active Learning

One of the challenging problems that faces the automated test data generation for path testing is the existence of infeasible paths, where no input data can be found to exercise them. Substantial time and effort may be wasted in trying to generate input data to exercise such paths. This paper proposes an active-learning approach to the automatic feasibility classification of program paths. This approach is based on the hypothesis that certain features of program behavior are stochastic processes that exhibit the Markov property, and that the resultant Markov models of individual program paths can be automatically clustered into effective predictors of path feasibility . To this end, the paper presents a technique that represents program paths as Markov models, and a clustering algorithm for Markov models that aggregates them into an effective path feasibility classifier. In this approach, the classifier is a map from program path statistics, namely, edge, branch, or definition-use profiles, to a label for the path, “feasible” or “infeasible”. The presented technique employs the bootstrapping active learning strategy, where the classifier is trained incrementally on a series of labeled instances, to extend its scope of training to be able to succeed in classifying new paths. The paper also presented the results of the experiments that were conducted to evaluate the effectiveness of the three paths feasibility classifiers built by using the proposed technique, and the bootstrapping technique.

A greedy particle swarm optimization (GPSO) algorithm for testing real-world smart card applications

Software testing continues to be regarded as a necessary and critical step in the software development life cycle. Among the multitudes of existing techniques, particle swarm optimization (PSO) algorithm, in particular, has shown superior merits for automatically generating software test cases for its easy implementation and for relying on fewer parameters that require tuning. Hence, several state-of-the-art PSO-based algorithms have been successfully used as a test data generator. On the other hand, greedy-based algorithms, which are commonly used to solve complex and multi-step combinatorial problems, are starting to gain momentum as a solution for the complexity problem of software testing. Greedy algorithms favored over other techniques when the solution of the problem is guaranteed to be near-optimal. As a result, the utilization of both greedy and PSO algorithms in a single solution for automatically generating test data represents a strong candidate if designed carefully. In this paper, we propose a novel hybrid greedy and PSO algorithm (GPSO) that jointly guarantees the effectiveness and close to optimality results for generating a minimum number of test data. Compared with the widely employed genetic algorithm (GA), our proposed GPSO outperforms the GA in terms of the average number of iterations, execution time, and coverage percentage. Experimental trials with six different typical Java card applications show that the use of the proposed GPSO as a test data generator results in an outstanding performance.

FSCT: A new fuzzy search strategy in concolic testing

Abstract Context Concolic testing is a promising approach to automate structural test data generation. However, combinatorial explosion of the path space, known as path explosion, and also constrained testing budget, makes achieving high code coverage in concolic testing a challenging task. Objective All branches of the previously explored paths make up the search space of concolic testing and search strategy define the mechanism of choosing branches to be flipped to drive the execution toward testing goals. With regard to the large number of candidate branches, choosing the right branch to continue the search is so crucial and has a direct impact on coverage rate and effort. This paper aims to improve the effectiveness of branch testing by considering the characteristics of paths reaching uncovered branches and presenting a novel search strategy for effectively and efficiently exploring the search space. Method We model the branch selection process in concolic testing as a decision making system and introduce a new Fuzzy Search Strategy in Concolic Testing (FSCT). FSCT chooses a branch to be filliped in which the most suitable path with respect to the proposed coverage factors reaches an uncovered branch with the highest priority and this priority is assigned by the designed fuzzy expert system. The proposed coverage factors effectively help to determine the characteristics of paths. Results We implemented FSCT on top of CREST and evaluated it using several popular benchmarks. The experimental results show that FSCT outperforms the state-of-the-art techniques in terms of coverage rate and coverage effort. Conclusion FSCT helps concolic testing to better cope with path explosion problem and shows its capabilities to achieve higher code coverage while at the same time decreases testing efforts in terms of both runtime and number of iterations.

Speeding up symbolic reasoning for relational queries

The ability to reason about relational queries plays an important role across many types of database applications, such as test data generation, query equivalence checking, and computer-assisted query authoring. Unfortunately, symbolic reasoning about relational queries can be challenging because relational tables are multisets (bags) of tuples, and the underlying languages, such as SQL, can introduce complex computation among tuples. We propose a space refinement algorithm that soundly reduces the space of tables such applications need to consider. The refinement procedure, independent of the specific dataset application, uses the abstract semantics of the query language to exploit the provenance of tuples in the query output to prune the search space. We implemented the refinement algorithm and evaluated it on SQL using three reasoning tasks: bounded query equivalence checking, test generation for applications that manipulate relational data, and concolic testing of database applications. Using real world benchmarks, we show that our refinement algorithm significantly speeds up (up to 100×) the SQL solver when reasoning about a large class of challenging SQL queries, such as those with aggregations.

An automated functional testing approach for virtual reality applications

SummarySoftware testing is regarded as an important method for fault revealing. Despite this advantage, it has been poorly used within the scope of virtual reality (VR) applications because they are highly complex and have peculiar features. Most testing performed of this VR applications are usability, which is conducted manually and only at final of the development process. Although some works try to propose criteria for this domain, there are no approaches that automatize the generation of test data from requirements specification in the VR domain. This paper proposes an approach called virtual reality—requirements specification and testing (VR‐ReST) to assist the requirements specification through a semiformal language and uses structural test criteria to generate test requirements and test data automatically for VR applications using scene graph concepts. The paper also examines the empirical results concerning the cost‐effectiveness of the approach for three different VR applications through two experiments. Mutation testing was used to evaluate effectiveness. We found that the approach achieved a high mutation score outperforming random testing, by 20%, on average. Our results also demonstrate that the approach is promising since it assists in writing and validating the requirements, as well as in reducing the risks of requirement specification by adopting a semiformal language.