Automatic Generation Of Test Suites Research Articles

Automated Test Suite Generation for Software Product Lines Based on Quality-Diversity Optimization

A Software Product Line (SPL) is a set of software products that are built from a variability model. Real-world SPLs typically involve a vast number of valid products, making it impossible to individually test each of them. This arises the need for automated test suite generation, which was previously modeled as either a single-objective or a multi-objective optimization problem considering only objective functions. This article provides a completely different mathematical model by exploiting the benefits of Quality-Diversity (QD) optimization that is composed of not only an objective function (e.g., t -wise coverage or test suite diversity) but also a user-defined behavior space (e.g., the space with test suite size as its dimension). We argue that the new model is more suitable and generic than the two alternatives because it provides at a time a large set of diverse (measured in the behavior space) and high-performing solutions that can ease the decision-making process. We apply MAP-Elites, one of the most popular QD algorithms, to solve the model. The results of the evaluation, on both realistic and artificial SPLs, are promising, with MAP-Elites significantly and substantially outperforming both single- and multi-objective approaches, and also several state-of-the-art SPL testing tools. In summary, this article provides a new and promising perspective on the test suite generation for SPLs.

Automated Test Suite Generation Tool based on GWO Algorithm

In succession, the size and complexity of the program increase and the scope of testing expand. So, to ensure deadline delivery and reduce development testing costs, program testing efficiency must be improved. Therefore, to ensure that the product is delivered on the deadline and the cost of testing development is reduced, the efficiency of the program's testing must be enhanced. In this study, highlighting is placed to generate test suite automatically to reach increase the coverage of paths based on two algorithms Grey Wolf Optimizer algorithm (GWO) and Particle swarm optimization (PSO). The results of the implementation will be compared with each other to demonstrate the performance and efficiency of the proposed methodology and the results show that the PSO algorithm has better than the GWO algorithm.

Cooperative verifier-based testing with CoVeriTest

Testing is a widely applied technique to evaluate software quality, and coverage criteria are often used to assess the adequacy of a generated test suite. However, manually constructing an adequate test suite is typically too expensive, and numerous techniques for automatic test-suite generation were proposed. All of them come with different strengths. To build stronger test-generation tools, different techniques should be combined. In this paper, we study cooperative combinations of verification approaches for test generation, which exchange high-level information. We present CoVeriTest, a hybrid technique for test-suite generation. CoVeriTest iteratively applies different conditional model checkers and allows users to adjust the level of cooperation and to configure individual time limits for each conditional model checker. In our experiments, we systematically study different CoVeriTest cooperation setups, which either use combinations of explicit-state model checking and predicate abstraction, or bounded model checking and symbolic execution. A comparison with state-of-the-art test-generation tools reveals that CoVeriTest achieves higher coverage for many programs (about 15%).

Plain random test generation with PRTest

Automatic test-suite generation tools are often complex and their behavior is not predictable. To provide a minimum baseline that test-suite generators should be able to surpass, we present PRTest, a random black-box test-suite generator for C programs: To create a test, PRTest natively executes the program under test and creates a new, random test value whenever an input value is required. After execution, PRTest checks whether any new program branches were covered and, if this is the case, the created test is added to the test suite. This way, tests are rapidly created either until a crash is found, or until the user aborts the creation. While this naive mechanism is not competitive with more sophisticated, state-of-the-art test-suite generation tools, it is able to provide a good baseline for Test-Comp and a fast alternative for automatic test-suite generation for programs with simple control flow. PRTest is publicly available and open source.

Performance analysis of six meta-heuristic algorithms over automated test suite generation for path coverage-based optimization

There exists a direct need to automate the process of test suite generation to get the most optimal results as testing accounts for more than 40% of total cost. One method to solve this problem is the use of meta-heuristic algorithms which iteratively improve the test data to reach the most optimized test suites. This study focuses on the performance evaluation of six meta-heuristic algorithms namely: hill-climbing algorithm (HCA), particle swarm optimization (PSO), firefly algorithm (FA), cuckoo search algorithm (CS), bat algorithm (BA) and artificial bee colony algorithm (ABC) by using their standard implementation to optimize the path coverage and branch coverage produced by the test data. The goal of the study was to find the best-suited algorithm to narrow down the future research in the field of test automation for path coverage-based optimization approaches. Each algorithm was first implemented to automatically generate test suites based on the program under test. This was followed by the performance evaluation of each algorithm for five programs written in Java. The algorithms were compared using process metrics: average time, best time, worst time and product metrics: path coverage & objective function values of the generated test suites. Results indicated ABC as the best-suited algorithm as it gave the most optimal test suites in reasonable time. BA was found to be the fastest but produced less optimal results. FA was found to be the slowest algorithm, while CS, PSO and HCA performed in between. These results show the relative performance of the six algorithms for this scenario and may be used by the future researchers to narrow down and improve the best performing algorithms for path coverage-based optimization approaches.

Empirical Evaluation of Automated Test Suite Generation and Optimization

Software testing is one of the most labor-intensive and crucial phases of the software development life cycle. It accounts for about 50–60% of the total cost of a project and requires a substantial amount of effort to ensure that the quality of the software is not compromised. Hence, optimizing the process of software testing has been of paramount importance in the IT industry. In recent times, there has been a significant shift toward software test automation to ensure the quality of the software while minimizing the cost and effort. This study proposes a statistical model analyzing the various factors that affect the testing cost, quality and effort by studying the behavioral pattern of professionals employed in the field of testing. The paper studies the factors, project duration, Software Complexity (SC), testing tools and automated test suite generation, while contrasting it with the overall software test automation, with the aim of finding a measure of how much automation affects the cost, quality and effort in practice. The results indicated that automation accounted for a variation of about 25% in each of the factors: cost, quality and effort much more than factors such as project duration, SC and testing tool . Further, the analysis indicated that the effect of SC on cost and effort was extremely low. This suggests that automation improved the quality while reducing the cost and effort irrespective of the complexity of the software being developed.

Mapping the Effectiveness of Automated Test Suite Generation Techniques

Automated test suite generation (ATSG) is an important topic in software engineering, with a wide range of techniques and tools being used in academia and industry. While their usefulness is widely recognized, due to the labor-intensive nature of the task, the effectiveness of the different techniques in automatically generating test cases for different software systems is not thoroughly understood. Despite many studies introducing various ATSG techniques, much remains to be learned, however, about what makes a particular technique work well (or not) for a specific software system. In this paper, we seek an answer to the question: “What features of a software system impact the effectiveness of ATSG techniques?” Once these features are identified, can they be used to select the most effective ATSG technique for a particular software system? To this end, we have implemented the mapping the effectiveness of test automation (META) tool, a new framework that identifies important software features that can be used to select suitable ATSG techniques to apply to new software systems. We evaluate the framework on a large set of open-source software projects and three ATSG techniques. The evaluation indicates that the number of methods in a class, the coupling between object classes, and the response for a class are the most indicative of what makes a software system hard to test by different techniques. The decision tree for ATSG technique selection generated by the META framework has an 88% accuracy, as shown by n-fold cross validation.

Using Subject Area Ontology for Automating Processes in Sphere of Scientific Investigation and Education

This paper presents methods of building ontologies, which are used in “ISTINA” current research information system (the name is the abbreviation for Russian equivalent of “Intellectual System for Thematic Investigation of Scientometrical Data”), used in Moscow State University and a number of other institutions. These methods include thematic analysis of scientific results present in the system; information search; context analysis of publications; a number of applications in the learning process, in particular, automatic generation of test suites and forming students’ individual learning trajectories. In the framework of the suggested approach, ontology elements (concepts and relations) are added by users of the system, which work in different branches of science, using the web editor to be described in the paper. These elements are stored in a relational database. The editor works on the client side and communicates with the server using the SPARQL query language and specially developed protocol for modifying the ontology. The concepts and relations between them may be verified by experts in the corresponding areas.

A novel strategy for automatic test data generation using soft computing technique

Software testing is one of the most crucial and analytical aspect to assure that developed software meets prescribed quality standards. Software development process invests at least 50% of the total cost in software testing process. Optimum and efficacious test data design of software is an important and challenging activity due to the nonlinear structure of software. Moreover, test case type and scope determines the quality of test data. To address this issue, software testing tools should employ intelligence based soft computing techniques like particle swarm optimization (PSO) and genetic algorithm (GA) to generate smart and efficient test data automatically. This paper presents a hybrid PSO and GA based heuristic for automatic generation of test suites. In this paper, we described the design and implementation of the proposed strategy and evaluated our model by performing experiments with ten container classes from the Java standard library. We analyzed our algorithm statistically with test adequacy criterion as branch coverage. The performance adequacy criterion is taken as percentage coverage per unit time and percentage of faults detected by the generated test data. We have compared our work with the heuristic based upon GA, PSO, existing hybrid strategies based on GA and PSO and memetic algorithm. The results showed that the test case generation is efficient in our work.

Incremental Approach to the Technology of Test Design for Industrial Projects

The paper presents an approach to effort reduction in developing test suites for industrial software products based on the incremental technology. The main problems to be solved by the incremental technology are full automation design of test scenarios and significant reducing of test explosion. The proposed approach provides solutions to the mentioned problems through joint co-working of a designer and a customer, through the integration of symbolic verification with the automatic generation of test suites; through the usage of an efficient technology with the toolset VRS/TAT.

Incremental testing of finite state machines

SUMMARYThe automatic generation of test suites for systems modelled as finite state machines (FSMs) is an important problem that impacts several critical applications. Known methods that automatically generate tests for FSMs, specially the W‐method and some derivations, strongly assume that the number of system states is small. If the overall number of states in the FSM specification is relatively large, such methods become difficult to use. However, often in practice, a system is defined as a combination of several subsystems, with the latter already independently designed, developed and tested. In this paper, we define the concept of combined FSMs and introduce a new method to test modular compositions of FSMs. This method allows for a new incremental testing strategy that turns the testing of new systems into a much more scalable process. As an example, we present an infinite family of naturally occurring FSM models for which our method produces exponentially more compact test suites than the W‐method. Copyright © 2012 John Wiley & Sons, Ltd.

A method for the automatic generation of test suites from object models

This paper shows how object-oriented specifications, written in the Unified Modeling Language (UML) can be translated into formal, behavioural descriptions and used as a basis for automatic test generation. The behavioural descriptions are written in a language of communicating state machines: the Intermediate Format (IF). The translation from UML to IF is based upon an earlier formal semantics, written in the Abstract State Machine (ASM) notation. Descriptions written in IF can be automatically explored; the results of these explorations are test trees, ready for input to a variety of testing packages.

Projected state machine coverage for software testing

Our research deals with test generation for software based on finite state machine (FSM) models of the program specification. We describe a set of coverage criteria and testing constraints for use in the automatic generation of test suites. We also describe the algorithms used to generate test suites based on these coverage criteria, and the implementation of these algorithms as an extension of the Murφ model checker[4]. The coverage criteria are simple but powerful in that they generate test suites of high quality and moderate volume, without requiring the user to have a sophisticated grasp of the test generation technology. The testing constraints are used to combat the endemic problem of state explosion, typically encountered in FSM techniques. We illustrate our techniques on several well-known problems from the literature and describe two industrial trials, to demonstrate the validity of our claims.

An experiment in automatic generation of test suites for protocols with verification technology

In this paper we describe an experiment in automatic generation of test suites for protocol testing. We report the results gained with generation of test suites based on advanced verification techniques applied to a real industrial protocol. In this experiment, several tools have been used: the commercial tool GEODE (VERILOG) was used for the generation of finite state graph models from SDL specifications, the tool Aldebaran of the CADP toolbox for the minimization of transition systems, and a prototype named TGV (for Test Generation using Verification techniques) for the generation of test suites which has been developed in the CADP toolbox. TGV is based on verification techniques such as synchronous product and on-the-fly verification. These tools have been applied to an industrial protocol, the DREX protocol. The comparison of produced test suites with hand written test suites proves the relevance of the used techniques.

The validation and integration of the TRIBUNE test‐bed

AbstractThe RACE project TRIBUNE is focused on a number of issues pertaining to the introduction of the Integrated Broadband Communication. The project has implemented a broadband test‐bed with a fully standardized Broadband User Network Interface (B‐UNI) according to the latest ITU‐T recommendations. A lot of effort was therefore spent on the specification, implementation and verification of the B‐UNI and the integration of the test‐bed. This paper is intended to focus on the evaluation of the innovative methods used and experience gained during the three year period of specification, implementation and verification. In this paper we will start with focusing on the specification of the B‐UNI. Particularly the decision to use formal specification languages (‐SDL 88) will be evaluated. The actual implementation of the B‐UNI is discussed in the second part, specially the experience and result of the use of automatic code generation modules. We will elaborate on how the verification of the implementation of the B‐UNI was managed in the third part of this document. Specially the use of the Ferry Clip method and the automatic generation of test suites will be focused upon. Finally, we will show that the taken steps in the previous phases have led to relatively short and easy integration phase of a complex test‐bed around a B‐UNI.