Automatic Test Generation Research Articles

TSTL: the template scripting testing language

A test harness, in automated test generation, defines the set of valid tests for a system, as well as their correctness properties. The difficulty of writing test harnesses is a major obstacle to the adoption of automated test generation and model checking. Languages for writing test harnesses are usually tied to a particular tool and unfamiliar to programmers, and often limit expressiveness. Writing test harnesses directly in the language of the software under test (SUT) is a tedious, repetitive, and error-prone task, offers little or no support for test case manipulation and debugging, and produces hard-to-read, hard-to-maintain code. Using existing harness languages or writing directly in the language of the SUT also tends to limit users to one algorithm for test generation, with little ability to explore alternative methods. In this paper, we present TSTL, the template scripting testing language, a domain-specific language (DSL) for writing test harnesses. TSTL compiles harness definitions into an interface for testing, making generic test generation and manipulation tools for all SUTs possible. TSTL includes tools for generating, manipulating, and analyzing test cases, including simple model checkers. This paper motivates TSTL via a large-scale testing effort, directed by an end-user, to find faults in the most widely used geographic information systems tool. This paper emphasizes a new approach to automated testing, where, rather than focus on developing a monolithic tool to extend, the aim is to convert a test harness into a language extension. This approach makes testing not a separate activity to be performed using a tool, but as natural to users of the language of the system under test as is the use of domain-specific libraries such as ArcPy, NumPy, or QIIME, in their domains. TSTL is a language and tool infrastructure, but is also a way to bring testing activities under the control of an existing programming language in a simple, natural way.

Incremental approach to the technology of test design for industrial projects

The paper presents a technology to effort reduction in developing test suites for industrial software products based on an incremental technology. The main problems to be solved by incremental technology are full automation design of test scenarios and significant reducing of test explosion. The proposed approach provides solutions to the mentioned problems through joint co-working of the designer and the customer with usage of formal models; and through methods of symbolic verification with automatic generation of test suites with the toolset VRS/TAT.

Automated test generation and error localisation for Simulink/Stateflow modelled systems using extended automata

Model-based development frameworks for cyber-physical systems (CPSs) such as Simulink and Stateflow are popular for many applications. For safety and security concerns, verification and testing/validation must be performed on the model-based CPS designs. In this study, the authors present an automatic test generation approach for model-based CPS designs in Simulink/Stateflow based on its translation to input/output extended finite automata (I/O-EFA) developed in the authors’ prior works. The test generation problem requires identifying the executable paths of the I/O-EFA model and also generating a test input for those paths. To execute a path, a certain sequence of other paths must be executed first, which they automatically identify. The approach is implemented by applying two different techniques, model checking and constraint solving. Both test generation implementations are validated by a case study. The results show that both implementations can generate test cases, while the implementation based on constraint solving is in general faster. The approach is further extended to requirements-based test generation. These tests are then used for validation purposes, and the failed versus passed tests are used to localise the fault to plausible Simulink/Stateflow blocks using the notion of fault-seed used in their earlier work. The approaches are applied on a bounded counter and a thermal control of a house as two different case studies.

Optimization of Test Case Generation using Genetic Algorithm (GA)

Testing provides means pertaining to assuring software performance. The total aim of software industry is actually to make a certain start associated with high quality software for the end user. However, associated with software testing has quite a few underlying concerns, which are very important and need to pay attention on these issues. These issues are effectively generating, prioritization of test cases, etc. These issues can be overcome by paying attention and focus. Solitary of the greatest Problems in the software testing area is usually how to acquire a great proper set associated with cases to confirm software. Some other strategies and also methodologies are proposed pertaining to shipping care of most of these issues. Genetic Algorithm (GA) belongs to evolutionary algorithms. Evolutionary algorithms have a significant role in the automatic test generation and many researchers are focusing on it. In this study explored software testing related issues by using the GA approach. In addition to right after applying some analysis, better solution produced, that is feasible and reliable. The particular research presents the implementation of GAs because of its generation of optimized test cases. Along these lines, this paper gives proficient system for the optimization of test case generation using genetic algorithm.

RAMBUTANS: automatic AOP-specific test generation tool

Aspect-oriented programming (AOP) is a programmatic methodology to handle better modularized code by separating crosscutting concerns from the traditional abstraction boundaries. Automated testing, as one of the most demanding needs of the software development to reduce both human effort and costs, is a delicate issue in testing aspect-oriented programs. Prior studies in the automated test generation for aspect-oriented programs have been very limited with respect to the need for both adequate tool support and capability concerning effectiveness and efficiency. This paper describes a new AOP-specific tool for testing aspect-oriented programs, called RAMBUTANS. The RAMBUTANS tool uses a directed random testing technique that is especially well suited for generating tests for aspectual features in AspectJ. The directed random aspect of the tool is parameterized by associating weights to aspects, advice, methods, and classes by controlling object and joint point creations during the test generation process. We present a comprehensive empirical evaluation of our tool against the current AOP test generation approaches on three industrial aspect-oriented projects. The results of the experimental and statistical tests showed that RAMBUTANS tool produces test suites that have higher fault-detection capability and efficiency for AspectJ-like programs.

La revolución digital y la evaluación: un nuevo paradigma

La revolución digital ha dado lugar a una nueva era en la evaluación educativa a gran escala; ofrece posibilidades que hace una década eran impensables. Junto con los avances de las ciencias cognitivas y la psicometría fue posible lograr avances sustanciales, como la generación automática de reactivos. El propósito de este trabajo es presentar un nuevo paradigma de evaluación, basado en campos semánticos, que se vale de la tecnología digital. Este paradigma permite evaluar competencias cognitivas de mayor complejidad, a través de respuestas construidas o semi-construidas y de crédito parcial, superando así muchas de las limitaciones del formato tradicional de lápiz y papel y de opción múltiple. Se describen las características teóricas, conceptuales, técnicas y computacionales del Examen de Competencias Básicas (EXCOBA) y el Generador Automático de Exámenes (GenerEx), lo que permite generar ítems isomorfos y exámenes equivalentes.

Test Data Generation for Cyclic Executives with CBMC and Frama-C: A Case Study

Testers of cyclic executive systems are required to make considerable and repetitive efforts to determine input sequences (tests) for leading a system from a start state to a target state. This task is time-consuming and it might lead testers to produce fewer tests than required; which affects negatively the quality of systems and can reduce time-to-market. We propose an automated test generation approach which integrates two renowned tools: the code slicing plugin of Frama-C and the bounded model-checker CBMC. We also suggest several code metrics to better assess the complexity of code and the effect of code slicing on test derivation with CBMC. The proposed approach has been tested on industrial case studies and significantly reduces test generation effort and computation time on two automotive controllers. The proposed approach alleviates the workload of testers so they are able to focus on producing more tests; thus, increasing the quality of systems.

Генерация функциональных тестов для HDL-описаний на основе проверки моделей

Automated test generation is a promising direction in hardware verification research area. Functional test generation methods based on models are widespread at the moment. In this paper, a functional test generation method based on model checking is proposed and compared to existing solutions. Automated model extraction from the hardware design’s source code is used. Supported HDLs include VHDL and Verilog. Several kinds of models are used at different steps of the test generation method: guarded action decision diagram (GADD), high-level decision diagram (HLDD) and extended finite-state machine (EFSM). The high-level decision diagram model (which is extracted from the GADD model) is used as a functional model. The extended finite-state machine model is used as a coverage model. The aim of test generation is to cover all the transitions of the extended finite state machine model. Such criterion leads to the high HDL source code coverage. Specifications based on transition and state constraints of the EFSM are extracted for this purpose. Later, the functional model and the specifications are automatically translated into the input format of the nuXmv model checking tool. nuXmv performs model checking and generates counterexamples. These counterexamples are translated to functional tests that can be simulated by the HDL simulator. The proposed method has been implemented as a part of the HDL Retrascope framework. Experiments show that the method can generate shorter tests than the FATE and RETGA methods providing the same or better source code coverage.

Using a bounded model checker for test generation: How to kill two birds with one SMT solver

Automated test generation has received a lot of attention in recent decades, because it is one possible solution to the problems inherent to software testing: the need to write tests in the first place and providing test coverage for the human factor. De facto the most promising technique to automatically generate a test is dynamic symbolic execution assisted by an automated constraint solver, e.g., an SMT solver. This process is very similar to bounded model checking, which also deals with generating models from source code, asserting logic properties in it, and processing the returned model. This paper describes a prototype unit test generator for C based on a working bounded model checker called Borealis and shows that these two techniques are very similar and can be easily implemented using the same basic components. The prototype test generator has been evaluated on a number of examples and has shown good results in terms of test coverage and test excessiveness.

Finding and Emulating Keyboard, Mouse, and Touch Interactions and Gestures while Crawling RIAs

Existing techniques for crawling Javascript-heavy Rich Internet Applications tend to ignore user interactions beyond mouse clicking, and therefore often fail to consider potential mouse, keyboard and touch interactions. We propose a new technique for automatically finding and exercising such interactions by analyzing and exercising event handlers registered in the DOM. A basic form of gesture emulation is employed to find states accessible via swiping and tapping. Testing the tool against 6 well-known gesture libraries and 5 actual RIAs, we find that the technique discovers many states and transitions resulting from such interactions, and could be useful for cases such as automatic test generation and error discovery, especially for mobile web applications.

Advances in test generation for testing software and systems

Among the software quality assurance techniques, testing stands out as one of the most frequently used techniques in practice. Consequently, testing is also widely studied in research. An important aspect in testing that receives much attention is the problem of automated test generation. This introductory paper summarizes the papers selected from the 25th IFIP International Conference on Testing Software and Systems (ICTSS 2013), which all deal with this topic. The field is briefly introduced by pointing out to some of the existing survey articles and relating the selected papers in this issue to each other and to the known body of knowledge.

Does Automated Unit Test Generation Really Help Software Testers? A Controlled Empirical Study

Work on automated test generation has produced several tools capable of generating test data which achieves high structural coverage over a program. In the absence of a specification, developers are expected to manually construct or verify the test oracle for each test input. Nevertheless, it is assumed that these generated tests ease the task of testing for the developer, as testing is reduced to checking the results of tests. While this assumption has persisted for decades, there has been no conclusive evidence to date confirming it. However, the limited adoption in industry indicates this assumption may not be correct, and calls into question the practical value of test generation tools. To investigate this issue, we performed two controlled experiments comparing a total of 97 subjects split between writing tests manually and writing tests with the aid of an automated unit test generation tool, E vo S uite . We found that, on one hand, tool support leads to clear improvements in commonly applied quality metrics such as code coverage (up to 300% increase). However, on the other hand, there was no measurable improvement in the number of bugs actually found by developers. Our results not only cast some doubt on how the research community evaluates test generation tools, but also point to improvements and future work necessary before automated test generation tools will be widely adopted by practitioners.

The Risks of Coverage-Directed Test Case Generation

A number of structural coverage criteria have been proposed to measure the adequacy of testing efforts. In the avionics and other critical systems domains, test suites satisfying structural coverage criteria are mandated by standards. With the advent of powerful automated test generation tools, it is tempting to simply generate test inputs to satisfy these structural coverage criteria. However, while techniques to produce coverage-providing tests are well established, the effectiveness of such approaches in terms of fault detection ability has not been adequately studied. In this work, we evaluate the effectiveness of test suites generated to satisfy four coverage criteria through counterexample-based test generation and a random generation approach—where tests are randomly generated until coverage is achieved—contrasted against purely random test suites of equal size. Our results yield three key conclusions. First, coverage criteria satisfaction alone can be a poor indication of fault finding effectiveness, with inconsistent results between the seven case examples (and random test suites of equal size often providing similar—or even higher—levels of fault finding). Second, the use of structural coverage as a supplement—rather than a target—for test generation can have a positive impact, with random test suites reduced to a coverage-providing subset detecting up to 13.5 percent more faults than test suites generated specifically to achieve coverage. Finally, Observable MC/DC, a criterion designed to account for program structure and the selection of the test oracle, can—in part—address the failings of traditional structural coverage criteria, allowing for the generation of test suites achieving higher levels of fault detection than random test suites of equal size. These observations point to risks inherent in the increase in test automation in critical systems, and the need for more research in how coverage criteria, test generation approaches, the test oracle used, and system structure jointly influence test effectiveness.

An Efficient Functional Test Generation Method For Processors Using Genetic Algorithms

Abstract The paper presents a new functional test generation method for processors testing based on genetic algorithms and evolutionary strategies. The tests are generated over an instruction set architecture and a processor description. Such functional tests belong to the software-oriented testing. Quality of the tests is evaluated by code coverage of the processor description using simulation. The presented test generation method uses VHDL models of processors and the professional simulator ModelSim. The rules, parameters and fitness functions were defined for various genetic algorithms used in automatic test generation. Functionality and effectiveness were evaluated using the RISC type processor DP32.

Automatic Generation of Unit Tests for Correlated Variables in Parallel Programs

A notorious class of concurrency bugs are race condition related to correlated variables, which make up about 30 % of all non-deadlock concurrency bugs. A solution to prevent this problem is the automatic generation of parallel unit tests. This paper presents an approach to generate parallel unit tests for variable correlations in multithreaded code. We introduce a hybrid approach for identifying correlated variables. Furthermore, we estimate the number of potentially violated correlations for methods executed in parallel. In this way, we are capable of creating unit tests that are suited for race detectors considering correlated variables. We were able to identify more than 85 % of all race conditions on correlated variables in eight applications after applying our parallel unit tests. At the same time, we reduced the number of unnecessary generated unit tests. In comparison to a test generator unaware of variable correlations, redundant unit tests are reduced by up to 50 %, while maintaining the same precision and accuracy in terms of the number of detected races.

An Automated Test Generation Technique for Software Quality Assurance

The world's increased dependence on software-enabled systems has raised major concerns about software reliability and security. New cost-effective tools for software quality assurance are needed. This paper presents an automated test generation technique, called Model-based Integration and System Test Automation (MISTA), for integrated functional and security testing of software systems. Given a Model-Implementation Description (MID) specification, MISTA generates test code that can be executed immediately with the implementation under test. The MID specification uses a high-level Petri net to capture both control- and data-related requirements for functional testing, access control testing, or penetration testing with threat models. After generating test cases from the test model according to a given criterion, MISTA converts the test cases into executable test code by mapping model-level elements into implementation-level constructs. MISTA has implemented test generators for various test coverage criteria of test models, code generators for various programming and scripting languages, and test execution environments such as Java, C, ${\rm C}++$ , C#, HTML-Selenium IDE, and Robot Framework. MISTA has been applied to the functional and security testing of various real-world software systems. Our experiments have demonstrated that MISTA can be highly effective in fault detection.

How to Optimize the Use of SAT and SMT Solvers for Test Generation of Boolean Expressions

In the context of automatic test generation, the use of propositional satisfiability (SAT) and Satisfiability Modulo Theories (SMT) solvers is becoming an attractive alternative to traditional algorithmic test generation methods, especially when testing Boolean expressions. The main advantages are the capability to deal with constraints over the inputs, the generation of compact test suites and the support for fault-detecting test generation methods. However, these solvers normally require more time and a greater amount of memory than classical test generation algorithms, making their applicability not always feasible in practice. In this paper, we propose several ways to optimize the SAT/SMT-based process of test generation for Boolean expressions and we compare several solving tools and propositional transformation rules. These optimizations promise to make SAT/SMT-based techniques as efficient as standard methods for testing purposes, especially when dealing with Boolean expressions, as proved by our experiments.

Integrating Evolutionary Testing with Reinforcement Learning for Automated Test Generation of Object‐Oriented Software

Recent advances in evolutionary test generation greatly facilitate the testing of Object-oriented (OO) software. Existing test generation approaches are still limited when the Software under test (SUT) includes Inherited class hierarchies (ICH) and Non-public methods (NPM). This paper presents an approach to generate test cases for OO software via integrating evolutionary testing with reinforcement learning. For OO software with ICH and NPM, two kinds of particular isomorphous substitution actions are presented and a Q-value matrix is maintained to assist the evolutionary test generation. A prototype called EvoQ is developed based on this approach and is applied to generate test cases for actual Java programs. Empirical results show that EvoQ can efficiently generate test cases for SUT with ICH and NPMand achieves higher branch coverage than two state-of-the-art test generation approaches within the same time budget.

Diagnostic Test Generation for Transition Delay Faults Using Stuck-At Fault Detection Tools

By adding a few logic gates and one or two modeling flip-flops to the circuit under test (CUT), we create a detection or diagnostic automatic test pattern generation (ATPG) model of transition delay faults usable by a conventional single stuck-at fault test pattern generator. Given a transition delay fault pair, the diagnostic ATPG model can either find an exclusive test or prove the equivalence of the fault pair. Our work offers advantages over existing work. First, the detection of a transition delay fault or the diagnosis of a fault pair can be modeled in only one instead of two or four time-frames of the CUT. Second, an exclusive test can be generated under either launch off capture (LOC) or launch off shift (LOS) mode for a full-scan sequential circuit. Third, the proposed ATPG models can be expanded into two time frames to facilitate the use of combinational ATPG tools, though with lower modeling complexity than was possible before. As a result, the percentage of distinguished transition delay fault pairs is larger and the proposed automatic exclusive test generation system is more time-efficient.

Correctness via compilation to logic

Advances in automated theorem provers over the last decade have led to a renaissance in software tools that compile problems of correctness to problems over logic formula. In this talk, I will review progress in automated theorem provers, such as Z3 from Microsoft Research, and consider a variety of program correctness tools that build upon Z3, such as automated test generators, automated safety/termination checkers, as well as interactive functional verifiers. I'll then describe a number of new projects that make use of the "correctness via compilation to logic" approach, including the design of new programming languages, ensuring the security of data centers, and safely programming gesture recognizers such as Kinect.