Automated Test Generation Tools Research Articles

An empirical study of automated unit test generation for Python

Various mature automated test generation tools exist for statically typed programming languages such as Java. Automatically generating unit tests for dynamically typed programming languages such as Python, however, is substantially more difficult due to the dynamic nature of these languages as well as the lack of type information. Our Pynguin framework provides automated unit test generation for Python. In this paper, we extend our previous work on Pynguin to support more aspects of the Python language, and by studying a larger variety of well-established state of the art test-generation algorithms, namely DynaMOSA, MIO, and MOSA. Furthermore, we improved our Pynguin tool to generate regression assertions, whose quality we also evaluate. Our experiments confirm that evolutionary algorithms can outperform random test generation also in the context of Python, and similar to the Java world, DynaMOSA yields the highest coverage results. However, our results also demonstrate that there are still fundamental remaining issues, such as inferring type information for code without this information, currently limiting the effectiveness of test generation for Python.

Kex: A Platform For Analysis Of JVM Programs

Introduction: Over the last years program analysis methods were widely used for software quality assurance. Different types of program analysis require various levels of program representation, analysis methods, etc. Platforms that provide utilities to implement different types of analysis on their basis become very important because they allow one to simplify the process of development. Purpose: Development of a platform for analysis of JVM programs. Results: In this paper we present Kex, a platform for building program analysis tools for JVM bytecode. Kex provides three abstraction levels. First is Kfg, which is an SSA-based control flow graph representation for bytecode-level analysis and transformation. Second is a symbolic program representation called Predicate State, which consists of first order logic predicates that represent instructions of the original program, constraints, etc. The final level is SMT integration layer for constraint solving. It currently provides an interface for interacting with three SMT solvers. Practical relevance: We have evaluated our platform by considering two prototypes. First prototype is an automatic test generation tool that has participated in SBST 2021 tool competition. Second prototype is a tool for detection of automatic library integration errors. Both prototypes have proved that Kex can be used to implement competitive and powerful program analysis tools.

TORACLE 2021 Workshop Summary

Testing is an important activity in software engineering, especially with the growing adoption of software systems in safety-critical domains. Testing research, however, is mostly focused on deter- mining which test inputs to use (e.g., proposing and evaluating test coverage criteria or automatic test generation tools). Re- gardless of the used coverage criterion, we need to know whether a given program executes correctly on a given input. Indeed, a test execution for which we cannot discriminate between success or failure serves little purpose. This corresponds to the so-called øracle problem", the problem of knowing whether a program be- haves correctly for a speci c input. Although the importance of the oracle problem is well understood, only a few alternatives exist to manually deriving test oracles. This makes automated test or- acle generation one of the main bottlenecks for achieving full test automation. Therefore, novel approaches and tools are needed to address this important problem.

Implementation of Randoop Automated Test Generation Tool for Novel Perspective on Software Development

Implementation of Randoop Automated Test Generation Tool for Novel Perspective on Software Development

Coverage Prediction for Accelerating Compiler Testing

Compilers are one of the most fundamental software systems. Compiler testing is important for assuring the quality of compilers. Due to the crucial role of compilers, they have to be well tested. Therefore, automated compiler testing techniques (those based on randomly generated programs) tend to run a large number of test programs (which are test inputs of compilers). The cost for compilation and execution for these test programs is significant. These techniques can take a long period of testing time to detect a relatively small number of compiler bugs. That may cause many practical problems, e.g., bringing a lot of costs including time costs and financial costs, and delaying the development/release cycle. Recently, some approaches have been proposed to accelerate compiler testing by executing test programs that are more likely to trigger compiler bugs earlier according to some criteria. However, these approaches ignore an important aspect in compiler testing: different test programs may have similar test capabilities (i.e., testing similar functionalities of a compiler, even detecting the same compiler bug), which may largely discount their acceleration effectiveness if the test programs with similar test capabilities are executed all the time. Test coverage is a proper approximation to help distinguish them, but collecting coverage dynamically is infeasible in compiler testing since most test programs are generated on the fly by automatic test-generation tools like Csmith. In this paper, we propose the first method to predict test coverage statically for compilers, and then propose to prioritize test programs by clustering them according to the predicted coverage information. The novel approach to accelerating compiler testing through coverage prediction is called COP (short for COverage Prediction). Our evaluation on GCC and LLVM demonstrates that COP significantly accelerates compiler testing, achieving an average of 51.01 percent speedup in test execution time on an existing dataset including three old release versions of the compilers and achieving an average of 68.74 percent speedup on a new dataset including 12 latest release versions. Moreover, COP outperforms the state-of-the-art acceleration approach significantly by improving <inline-formula><tex-math notation="LaTeX">$17.16\%\sim 82.51\%$</tex-math></inline-formula> speedups in different settings on average.

Adapting automated test generation to GUI testing of industry applications

ContextAutomated test generation promises to improve the effectiveness of software testing and to reduce the involved manual effort. While automated test generation has been successfully applied for code-level API testing, it has not found widespread adoption in practice for testing of graphical user interfaces. Tools for test generation do not support GUI testing out-of-the-box but require dedicated extensions. ObjectiveThis paper explores the applicability of automated test generation for testing GUIs of industry applications. We propose a test adapter approach to bridge the gap between automated test generation tools and industry applications. MethodA multiple case study was conducted in which automated test generation with test adapters has been applied at the unit, integration, and system test level in three industry projects from two different companies. ResultsAutomated test generation via test adapters could be applied at all test levels. It has led to an increase of coverage as well as the detection of new defects that were not found by preceding testing activities in the projects. While test adapters can easily be implemented at the unit test level, their complexity and the corresponding effort for providing adapter implementations rises at higher test levels. ConclusionTest adapters can be used for applying automated test generation for testing GUIs of industry applications. They bridge the gap between automated test generation tools and industry applications. The development of test adapters requires no tool-specific knowledge and can be performed by members of the development team.

Does Automated Unit Test Generation Really Help Software Testers? A Controlled Empirical Study

Work on automated test generation has produced several tools capable of generating test data which achieves high structural coverage over a program. In the absence of a specification, developers are expected to manually construct or verify the test oracle for each test input. Nevertheless, it is assumed that these generated tests ease the task of testing for the developer, as testing is reduced to checking the results of tests. While this assumption has persisted for decades, there has been no conclusive evidence to date confirming it. However, the limited adoption in industry indicates this assumption may not be correct, and calls into question the practical value of test generation tools. To investigate this issue, we performed two controlled experiments comparing a total of 97 subjects split between writing tests manually and writing tests with the aid of an automated unit test generation tool, E vo S uite . We found that, on one hand, tool support leads to clear improvements in commonly applied quality metrics such as code coverage (up to 300% increase). However, on the other hand, there was no measurable improvement in the number of bugs actually found by developers. Our results not only cast some doubt on how the research community evaluates test generation tools, but also point to improvements and future work necessary before automated test generation tools will be widely adopted by practitioners.

The Risks of Coverage-Directed Test Case Generation

A number of structural coverage criteria have been proposed to measure the adequacy of testing efforts. In the avionics and other critical systems domains, test suites satisfying structural coverage criteria are mandated by standards. With the advent of powerful automated test generation tools, it is tempting to simply generate test inputs to satisfy these structural coverage criteria. However, while techniques to produce coverage-providing tests are well established, the effectiveness of such approaches in terms of fault detection ability has not been adequately studied. In this work, we evaluate the effectiveness of test suites generated to satisfy four coverage criteria through counterexample-based test generation and a random generation approach—where tests are randomly generated until coverage is achieved—contrasted against purely random test suites of equal size. Our results yield three key conclusions. First, coverage criteria satisfaction alone can be a poor indication of fault finding effectiveness, with inconsistent results between the seven case examples (and random test suites of equal size often providing similar—or even higher—levels of fault finding). Second, the use of structural coverage as a supplement—rather than a target—for test generation can have a positive impact, with random test suites reduced to a coverage-providing subset detecting up to 13.5 percent more faults than test suites generated specifically to achieve coverage. Finally, Observable MC/DC, a criterion designed to account for program structure and the selection of the test oracle, can—in part—address the failings of traditional structural coverage criteria, allowing for the generation of test suites achieving higher levels of fault detection than random test suites of equal size. These observations point to risks inherent in the increase in test automation in critical systems, and the need for more research in how coverage criteria, test generation approaches, the test oracle used, and system structure jointly influence test effectiveness.

Formal Specification Based Automatic Test Generation for Embedded Network Systems

Embedded systems have become increasingly connected and communicate with each other, forming large-scaled and complicated network systems. To make their design and testing more reliable and robust, this paper proposes a formal specification language calledSENSand aSENS-based automatic test generation tool calledTGSENS. Our approach is summarized as follows: (1) A user describes requirements of target embedded network systems by logical property-based constraints usingSENS. (2) GivenSENSspecifications, test cases are automatically generated using a SAT-based solver. Filtering mechanisms to select efficient test cases are also available in our tool. (3) In addition, given a testing goal by the user, test sequences are automatically extracted from exhaustive test cases. We’ve implemented our approach and conducted several experiments on practical case studies. Through the experiments, we confirmed the efficiency of our approach in design and test generation of real embedded air-conditioning network systems.

Applying formal methods to PCEP: an industrial case study from modeling to test generation

SUMMARYThis paper presents the experimental results in applying formal methods to an industrial protocol for constraint‐based path computation, called Path Computation Element Communication Protocol (PCEP). The experiments include a number of major activities in model‐based testing from modeling to test generation. From the PCEP specification defined by IETF (Internet Engineering Task Force), the functionalities of PCEP are divided into two parts: application and protocol. The protocol part of PCEP is then described in the IF (Intermediate Format) language which is based on communicating timed automata. A number of basic requirements are identified from the PCEP specification and then described as properties in IF. Based on these properties, the validation and verification of the formal specification are carried out using the IF toolset. Test cases are generated using an automatic test generation tool, called TestGen‐IF, which uses partial state space exploration guided by test purposes. As a result, some errors and ambiguities have been found in the PCEP standard specification. Copyright © 2011 John Wiley &amp; Sons, Ltd.