Automatic Test Data Generation Research Articles

Exploiting deep reinforcement learning and metamorphic testing to automatically test virtual reality applications

SummaryDespite the rapid growth and popularization of virtual reality (VR) applications, which have enabled new concepts for handling and solving existing problems through VR in various domains, practices related to software engineering have not kept up with this growth. Recent studies indicate that one of the topics that is still little explored in this area is software testing, as VR applications can be built for practically any type of purpose, making it difficult to generalize knowledge to be applied. In this paper, we present an approach that combines metamorphic testing, agent‐based testing and machine learning to test VR applications, focusing on finding collision and camera‐related faults. Our approach proposes the use of metamorphic relations to detect faults in collision and camera components in VR applications, as well as the use of intelligent agents for the automatic generation of test data. To evaluate the proposed approach, we conducted an experimental study on four VR applications, and the results showed an of the solution ranging from 93% to 69%, depending on the complexity of the application tested. We also discussed the feasibility of extending the approach to identify other types of faults in VR applications. In conclusion, we discussed important trends and opportunities that can benefit both academics and practitioners.

Guided Intelligent Hyper-Heuristic Algorithm for Critical Software Application Testing Satisfying Multiple Coverage Criteria

This paper proposes a novel algorithm that combines symbolic execution and data flow testing to generate test cases satisfying multiple coverage criteria of critical software applications. The coverage criteria considered are data flow coverage as the primary criterion, software safety requirements, and equivalence partitioning as sub-criteria. The characteristics of the subjects used for the study include high-precision floating-point computation and iterative programs. The work proposes an algorithm that aids the tester in automated test data generation, satisfying multiple coverage criteria for critical software. The algorithm adapts itself and selects different heuristics based on program characteristics. The algorithm has an intelligent agent as its decision support system to accomplish this adaptability. Intelligent agent uses the knowledge base to select different low-level heuristics based on the current state of the problem instance during each generation of genetic algorithm execution. The knowledge base mimics the expert’s decision in choosing the appropriate heuristics. The algorithm outperforms by accomplishing 100% data flow coverage for all subjects. In contrast, the simple genetic algorithm, random testing and a hyper-heuristic algorithm could accomplish a maximum of 83%, 67% and 76.7%, respectively, for the subject program with high complexity. The proposed algorithm covers other criteria, namely equivalence partition coverage and software safety requirements, with fewer iterations. The results reveal that test cases generated by the proposed algorithm are also effective in fault detection, with 87.2% of mutants killed when compared to a maximum of 76.4% of mutants killed for the complex subject with test cases of other methods.

A Hybrid Method for Test Data Generation for Unit Testing of C/C++ Projects

In recent years, automated test data generation from source code has gained a significantpopularity in software testing. This paper proposes a method, named Hybrid, to generate test datafor unit testing C/C++ projects. The method is a combination of two test data generation methodsnamed IBVGT and WCFT. In IBVGT method, the source code is analyzed to find simple conditions.Then, bases on these conditions, IBVGT generates test data for boundary values without having tosolve test paths constraints. This makes the method faster than BVTG method when generating testdata. In Hybrid method, while generating test data using WCFT, simple conditions are collected forboundary values test data generation. Test data generated by Hybrid are able to ensure both highsource code coverage and error detection ability. In addition, Hybrid is capable of finding infeasibleexecution paths and dead code. Experimental results with some popular unit functions show thatHybrid outperforms STCFG method in terms of test data generation time and boundary values relatederror detection. IBVGT is superior to BVTG in term of test data generation time whilst its boundaryvalues related error detection ability depends on the number of simple conditions inside each unitfunction.

Hybrid Approach for Automated Test Data Generation

Software testing has long been thought to be a good technique to improve the software quality and reliability. Path testing is the most reliable software testing technique and the key method for improving software quality among all testing approaches. On the other hand, test data quality has a big impact on the software testing activity’s ability to detect errors or defects. To solving testing problem, one must locate the entire search space for the relevant input data to encompass the different paths in the testable program. To satisfy path coverage, it is vital test to look at the accumulated test data across the thorough search area. A new approach based on ant colony optimization and negative selection algorithm (HACO-NSA) is presented in this research which overcome the flaws associated with search-based test data by generated automated test data. The optimum path testing objective is to generate appropriate test data to maximise coverage and to enhance the test data’s efficacy, as a result, the test data’s adequacy is validated using a path-based fitness function. In the NSA generation stage, the suggested method alters the new detectors creation using ACO. The proposed approach is evaluated for metrics such as average coverage, average generation, average time, and success rate and comparison has been done with random testing, ant colony optimization and negative selection algorithm Different benchmark programs have been used for object-oriented system. The findings show that the hybrid methodology escalates the coverage percentage and curtail test data size, reduces the redundancy in data and enhances the efficiency. The proposed approach is follows IEEE 829-2008 test documentation in entire testing process.

An automatic generation of software test data based on improved Markov model

In order to overcome the problems of low data reliability and long generation time of traditional automatic generation methods of software test data, an automatic generation method of software test data based on improved Markov model is designed. Firstly, collect software test data in different stages; Then, by calculating the similarity of the collected software test data, remove the test data with high similarity, calculate the importance of the software test data with the help of entropy weight method, and complete the data preprocessing; Finally, the Markov model is improved with the help of genetic algorithm, generation path and variation factor of software test data are set, and the improved Markov model is used to automatically generate high quality software test data. Experimental results show that when the number of experiments is 50, the generation time of this method is about 2.8 s, the reliability coefficient is always higher than 0.8.

Bio‐inspired optimization to support the test data generation of concurrent software

SummaryConcurrent programming is increasingly present in modern applications. Although it provides higher performance and better use of available resources, the mechanisms of interaction between processes/threads result in a greater challenge for software testing activity. The nondeterminism present in those applications is one of the main issues during the test activity since the same test input can produce different possible execution paths, which may or not contain defects. The test data automatic generation can alleviate this problem, ensuring higher speed and reliability in software testing activity. This paper explores the automatic test data generation for concurrent programs through Genetic Algorithm, a bioinspired optimization technique, and proposes a test data generation approach for concurrent programs, called BioConcST, and a new operator for the selection of test subjects, called FuzzyST, which uses fuzzy logic. The approaches were evaluated in an experimental study towards their validation. The results showed that BioConcST is more promising than the other approaches at all analyzed levels. FuzzyST, together with Elitism and Tournament operators, provided the best results; however, it proved more suitable for concurrent programs of higher complexity.

Automatic Test Data Generation for Basis Path Testing

Objectives: This paper presents a new hybrid ACO-NSA algorithm for the automatic test data generation problem with path coverage as an objective function. Method: In it, at the first instance, test data (detectors) are generated with the ant colony optimization algorithm (ACO), and then the generated data set (detector set) has been refined by a negative selection algorithm (NSA) with Hamming distance. Findings: The algorithm’s performance is tested on several benchmark problems with different data types and variables for metrics average coverage, average generations, average time and success rate, Iteration value 1000 is set for average coverage, average generations, average time and 200 for success rate. The obtained results from the proposed approach are compared with some existing approaches. The results are very efficient with high efficacy, higher path coverage, minimal data redundancy, and less execution time. Applications: This approach can be applied in any type of software development process in software engineering to reduce the testing efforts. Novelty: The approach is based on two distinct methodologies: metaheuristic search and artificial immune search, and its fitness is measured using path coverage as the fitness function. The approach provides 99.5% average path coverage, 2.72% average number of generations in 0.07 ns, and 99.9% success rate, which is significantly better than comparable approaches. Keywords: Test data generation; Metaheuristic search; Artificial immune search; Ant colony optimization; Negative selection algorithm; Path coverage

An automated test data generation method for void pointers and function pointers in C/C++ libraries and embedded projects

Automated test data generation for unit testing C/C++ functions using concolic methods is well-known for improving software quality while reducing human testing effort. However, there have been only a few researches related to generating test data for void pointers and function pointers which are commonly used in C/C++ libraries and embedded projects. This paper proposes a concolic-based method named VFP (Void and Function Pointers test data generation) to generate test data for void pointers and function pointers. The key idea of VFP method is to preprocess the source code of the project under test to find all possible types of void pointers and references of function pointers. These types and references are used in the initial test data generating phase of the concolic testing method. VFP method is implemented in VFP verification tool to test on various C/C++ libraries and embedded projects. The experimental results show that VFP significantly improves the coverage of the generated test data in comparison with existing methods.

A Research for Executable Path Automatic Generation Method Based on EFSM

Extended finite state machine (EFSM) is currently one of the most widely used model in the field of software testing. EFSM model is an enhanced model based on finite state machine (FSM). Automated test data generation is still a challenging problem due to the complexity of EFSM which extends the input and output parameters, context variables as well as the predicate condition. These reasons lead to conflict of the context variable with the enable conditions in the transition path. In order to avoid infeasible path generation, this paper proposes a method based on modified breadth first search to generate feasible transition path (MBFS-FTP). To solve the problem of state explosion in path generation, this paper converts state diagram to transition diagram on EFSM model. In order to make the EFSM static model can be driven execution, this paper uses UML model and generate executable model, so that implements EFSM execute dynamically. When using breadth-first search algorithm (MBFS) on every target transition to generate an executable transition path, the conflict detection algorithm is utilized to the transition path for conflict decision, avoiding the occurrence of an infeasible transition path. Considering target transition has multiple feasible transition paths, this paper combines suggested penalized value of definition-predicate-use (def-p-use) pair and length of feasible transition path, and then develops the measurement method of feasible transition path, and obtained a set of feasible transition path containing all of transitions. Through the experiment on two actual EFSM model, verifying the effectiveness of MBFS-FTP for feasible transition automatic generation, the experimental results show that MBFS-FTP can reduce the feasible transition path length, and make transition paths are more easily to be triggered at the times of generating test cases lately, furthermore, it can improve the efficiency of generate feasible transition path and save a lot of time.

Automated Software Test Data Generation With Generative Adversarial Networks

With the rapid increase of software scale and complexity, the cost of traditional software testing methods will increase faster than the scale of software. In order to improve test efficiency, it is particularly important to automatically generate high-quality test cases. This paper introduces a framework for automatic test data generation based on the generative adversarial network (GAN). GAN is employed to train a generative model over execution path information to learn the behavior of the software. Then we can use the trained generative model to produce new test data, and select the test data that can improve the branch coverage according to our proposed selection strategy. Compared to prior work, our proposed method is able to handle programs under test with large-scale branches without analyzing branch expressions. In the experiment, we exhibit the performance of our method by using two modules in GNU Scientific Library. In particular, we consider the application of our method in two testing scenarios; unit testing and integration testing, and conduct a series of experiments to compare the performance of three types of GAN models. Results indicate that the WGAN-GP shows the best performance in our framework. Compared with the random testing method, the WGAN-GP based framework improves the test coverage of five functions out of the seven in the unit testing.

Automation of software test data generation using genetic algorithm and reinforcement learning

Software testing is one of the most important methods of analyzing software quality assurance. This process is very time consuming and expensive and accounts for almost 50% of the software production cost. In addition to the cost problem, the nature of the test, which seeks errors in the program, is such that software engineers are not interested in doing the process, so we are looking to use automated methods to reduce the cost and time of the test. In the last decade, various methods have been introduced for the automatic test data generation, the purpose of which is to maximize the detection of errors by generating minimum amount of test data. The main issue in the test data generation process is to determine the input data of the program in such a way that it meets the specified test criterion. In this research, a structural method has been used in order to automate the process of test data generation considering the criterion of covering all finite paths. In structural methods, the problem is converted into a search problem and meta-heuristic algorithms are used to solve it. The proposed method in this paper is a memetic algorithm in which reinforcement learning is used as a local search method within a genetic algorithm. Experimental results have shown that this method is faster for test data generation than many existing evolutionary or meta-heuristic algorithms and can provide better coverage with fewer evaluations. Compared algorithms include: conventional genetic algorithm, a variety of improvements to the genetic algorithm, random search, particle swarm optimization, bees algorithm, ant colony optimization, simulated annealing, hill climbing, and tabu search.

Gender-Based Deep Learning Firefly Optimization Method for Test Data Generation.

Software testing is a widespread validation means of software quality assurance in industry. Intelligent optimization algorithms have been proved to be an effective way of automatic test data generation. Firefly algorithm has received extensive attention and been widely used to solve optimization problems because of less parameters and simple implement. To overcome slow convergence rate and low accuracy of the firefly algorithm, a novel firefly algorithm with deep learning is proposed to generate structural test data. Initially, the population is divided into male subgroup and female subgroup. Following the randomly attracted model, each male firefly will be attracted by another randomly selected female firefly to focus on global search in whole space. Each female firefly implements local search under the leadership of the general center firefly, constructed based on historical experience with deep learning. At the final period of searching, chaos search is conducted near the best firefly to improve search accuracy. Simulation results show that the proposed algorithm can achieve better performance in terms of success coverage rate, coverage time, and diversity of solutions.

Automated Test Data Generation Based on a Genetic Algorithm with Maximum Code Coverage and Population Diversity

In the present paper, we investigate an approach to intelligent support of the software white-box testing process based on an evolutionary paradigm. As a part of this approach, we solve the urgent problem of automated generation of the optimal set of test data that provides maximum statement coverage of the code when it is used in the testing process. We propose the formulation of a fitness function containing two terms, and, accordingly, two versions for implementing genetic algorithms (GA). The first term of the fitness function is responsible for the complexity of the code statements executed on the path generated by the current individual test case (current set of statements). The second term formulates the maximum possible difference between the current set of statements and the set of statements covered by the remaining test cases in the population. Using only the first term does not make it possible to obtain 100 percent statement coverage by generated test cases in one population, and therefore implies repeated launch of the GA with changed weights of the code statements which requires recompiling the code under the test. By using both terms of the proposed fitness function, we obtain maximum statement coverage and population diversity in one launch of the GA. Optimal relation between the two terms of fitness function was obtained for two very different programs under testing.

Mining constant information for readable test data generation

Automated test data generation tools produce test data that can achieve high coverage faster than test data generated manually by a tester. However, the test data generated by automated tools has been shown to not help developers find more bugs. The main reason is that it is difficult for human testers to understand and evaluate the test data. In this paper, an approach is introduced to automatically generate readable test data, which has been implemented in a tool called CTS. CTS can mine constant information from projects under testing and obtain heuristic information by aggregating and rating related constants. CTS adds heuristic information to the automatic test data generation process to generate test data that is quick and easy for a human to comprehend and check. Empirical experiments show that the proposed approach can improve the efficiency of test data generation and generate test data that is more convenient for a human oracle.

Efficient Framework for Fully Automatic Test Case Generation and Prioritization Using Genetic Algorithm in Software Testing

Automation in software testing is significantly growing in recent situation. Most part of the system is automated with help of the software. Today every modern software developers are trying to automate the software development process as much as possible. Therefore to develop any software more skills and expertise are needed. For the software development process, testing of software is the exceedingly significant and considerable phase. Automatic test data generation had an essential function in specific area regarding software testing. Test case creation is technique of gathering the data which completes the testing standards, all criteria’s and conditions. During testing process, the software goes through frequent modifications. As a result, due to all of these modifications and repetitive retesting the cost of testing process increases. This is called regression testing. Regression testing requires more expertise, more effort, more time and more cost. Here to reduce the time and expenditure, many type of techniques are proposed. The changes in one test case will affect all the others test cases. To triumph over this problem, when the changes occurred in the software every the test case have to be retested repeatedly. And this problem leads to make the testing process time consuming with unnecessary increased cost. Here In this research paper, the work’s focal point on automatic test cases generation and prioritization with improved evolutionary genetic algorithm.

A feedback-directed method of evolutionary test data generation for parallel programs

Abstract Context: Genetic algorithms can be utilized for automatic test data generation. Test data are encoded as individuals which are evolved for a number of generations using genetic operators. Test data of a parallel program include not only the program input, but also the communication information between each pair of processes. Traditional genetic algorithms, however, do not make full use of information provided by a population’s evolution, resulting in a low efficiency in generating test data. Objective: This paper emphasizes the problem of test data generation for parallel programs, and presents a feedback-directed genetic algorithm for generating test data of path coverage. Method: Information related to a schedule sequence is exploited to improve genetic operators. Specifically, a scheduling sequence is evaluated according to how well an individual covers the target path. The probability of the crossover and mutation points being located in the region is determined based on the evaluation result, which prevents a good schedule sequence from being destroyed. If crossover and mutation are performed in the scheduling sequence, the location of crossover and mutation points is further determined according to the relationship between nodes to be covered and the scheduling sequence. In this way, the population can be evolved in a narrowed search space. Results: The proposed algorithm is applied to test 11 parallel programs. The experimental results show that, compared with the genetic algorithm without utilizing information during the population evolution, the proposed algorithm significantly reduces the number of generations and the time consumption. Conclusion: The proposed algorithm can greatly improve the efficiency in evolutionary test data generation.

Software Test Data Generation based on Path Testing using Genetic Algorithms

Test data generation is the task of constructing test cases for predicting the acceptability of novel or updated software. Test data could be the original test suite taken from previous run or imitation data generated afresh specifically for this purpose. The simplest way of generating test data is done randomly but such test cases may not be competent enough in detecting all defects and bugs. In contrast, test cases can also be generated automatically and this has a number of advantages over the conventional manual method. Genetic Algorithms, one of the automation techniques, are iterative algorithms and apply basic operations repeatedly in greed for optimal solutions or in this case, test data. By finding out the most error-prone path using such test cases one can reduce the software development cost and improve the testing efficiency. During the evolution process such algorithms pass on the better traits to the next generations and when applied to generations of software test data they produce test cases that are closer to optimal solutions. Most of the automated test data generators developed so far work well only for continuous functions. In this study, we have used Genetic Algorithms to develop a tool and named it TG-GA (Test Data Generation using Genetic Algorithms) that searches for test data in a discontinuous space. The goal of the work is to analyze the effectiveness of Genetic Algorithms in automated test data generation and to compare its performance over random sampling particularly for discontinuous spaces.

Using likely invariants for test data generation

Various approaches have been developed to tackle the problem of automatic test data generation. Among them search-based methods use metaheuristic algorithms to guide search in the input space of the program under test. This paper presents a new approach for improving search-based test data generation methods. This approach is based on learning the relationships between program input values and program parts covered by those values. The learned relationships are used to accelerate achieving test coverage goals. We introduce the concepts of branch likely invariant and path likely invariant as the basis for the learning method. In addition, we utilize simple predicates (based on some predefined templates) over program input variables to generate better initial candidate solutions, and use the mutation of the mentioned predicates to cover unexplored program parts. The current version of the proposed approach only considers numeric and string input parameters. To evaluate the performance of the proposed approach, a series of experiments have been carried out on a number of different benchmark programs. Through experiments and analysis, we show that the proposed approach enhances the effectiveness of common search-based test data generation methods, in terms of the coverage percentage.

Automated Test Cases and Test Data Generation for Dynamic Structural Testing in Automatic Programming Assessment Using MC/DC

Automatic Programming Assessment (or APA) is known as a method to assist educators in executing automated assessment and grading on students’ programming exercises and assignments. Having to execute dynamic testing in APA, providing an adequate set of test data via a systematic process of test data generation is necessarily essential. Though researches respecting to software testing have proposed various significant methods to realize automated test data generation, it occurs that recent studies of APA rarely utilized these methods. Merely some of the limited studies appeared to resolve this circumstance, yet the focus on realizing test set and test data covering more thorough dynamic-structural testing are still deficient. Thus, we propose a method that utilizes MC/DC coverage criteria to support more thorough automated test data generation for dynamic-structural testing in APA (or is called DyStruc-TDG). In this paper, we reveal the means of deriving and generating test cases and test data for the DyStruc-TDG method and its verification concerning the reliability criteria (or called positive testing) of test data adequacy in programming assessments. This method offers a significant impact on assisting educators dealing with introductory programming courses to derive and generate test cases and test data via APA regardless of having knowledge of designing test cases mainly to execute structural testing. As regards to this, it can effectively reduce the educators’ workload as the process of manual assessments is typically prone to errors and promoting inconsistency in marking and grading.

Automated optimal test data generation for OCL specification using harmony search algorithm

Exploring software testing possibilities at an early software life cycle is increasingly necessary to avoid the propagation of defects to the subsequent phases. This requirement demands technique that can generate automated test cases at the initial phases of software development. Thus, we propose a novel framework for automated test data generation using formal specifications written in object constraint language (OCL). We also defined a novel fitness function named exit-predicate-wise branch coverage (EPWBC) to evaluate the generated test data. Another focus of the proposed approach is to optimise the test case generation process by applying, harmony search (HS) algorithm. The experimental results indicate that the proposed framework outperforms the other OCL-based test case generation techniques. Furthermore, it has been inferred that OCL based testing adopting HS algorithm forms an excellent combination to produce more test coverage and an optimal test suite thereby improving the quality of a system.