Test Generation Method Research Articles

KeYGenU: combining verification-based and capture and replay techniques for regression unit testing

Unit testing plays a major role in the software development process. Two essential criteria to achieve effective unit testing are: (1) testing each unit in isolation from other parts of the program and (2) achieving high code coverage. The former requires a lot of extra work such as writing drivers and stubs, whereas the latter is difficult to achieve when manually writing the tests. When changing existing code it is advocated to run the unit tests to avoid regression bugs. However, in many cases legacy software has no unit tests. Writing those tests from scratch is a hard and tedious process, which might not be cost-effective. This paper presents a tool chain approach that combines verification-based testing (VBT) and capture and replay (CaR) test generation methods. We have built a concrete tool chain, KeYGenU, which consists of two existing tools—KeY and GenUTest. The KeY system is a deductive verification and test-generation tool. GenUTest automatically generates JUnit tests for a correctly working software. This combination provides isolated unit test suites with high code-coverage. The generated tests can also be used for regression testing.

A SAT Based Test Generation Method for Delay Fault Testing of Macro Based Circuits

This letter addresses the problem of delay fault test generation in circuits using macros whose implementation is not known. The proposed approach uses a new signal representation that allows us to evaluate any kind of sensitization conditions (robust, non-robust, and functional) by means of Boolean differential calculus. Such an approach makes use of binary decision diagrams to support the computation of sensitization conditions for each macro along a path and of Boolean satisfiability to justify such conditions at primary inputs. Results are shown for a set of benchmarks.

Markov modelling and parameterisation of genetic evolutionary test generations

Genetic evolutionary algorithms are effective and optimal test generation methods. However, the methods to select the algorithm parameters are often ad hoc, relying on empirical data. We used a Markov-based method to model the genetic evolutionary test generation process, parameterise the process characteristics, and derive analytical solutions for selecting the optimisation parameters. The method eliminates preliminary test generation calibration and experimentation effort needed to select these parameters, which are used in current practice.

Further Results on Fault Classes in Boolean Specifications

Fault conditions can be used to analyze existing testing methods. The hierarchy of fault classes may not only explain experimental results of various testing methods, but also help to design effective test cases. We extend the known hierarchy to a wider range of fault classes based on detection conditions and prove the relationships between them. As a result, we show that for fault classes discussed in this paper test generation methods should give priority to the detection of variable insertion faults and missing expression faults.

GENERATING FUNCTIONAL DELAY FAULT TESTS FOR NON-SCAN SEQUENTIAL CIRCUITS

The paper presents two functional fault models that are devoted for functional delay test generation for non-scan synchronous sequential circuits. These fault models form one joint functional fault model. The non-scan sequential circuit is represented as the iterative logic array model consisting of k copies of the combinational logic of the circuit. The value k defines the length of clock sequence. The length of clock sequence is determined using the presented functional fault models. The experimental results demonstrate the superiority of the delay test patterns generated at the functional level using the introduced functional fault models against the transition test patterns obtained at the gate level by deterministic test pattern generator. The functional delay test generation method especially is useful for the circuits, when the long test sequences are needed in order to detect transition faults.

Test generation algorithm for analog systems based on support vector machine

In some methods for test generation, an analog device under test (DUT) is treated as a discrete-time digital system by placing it between a digital-to-analog converter and an analog-to-digital converter. Then the test patterns and responses can be performed and analyzed in the digital domain. We propose a novel test generation algorithm based on a support vector machine (SVM). This method uses test patterns derived from the test generation algorithm as input stimuli, and sampled output responses of the analog DUT for classification and fault detection. The SVM is used for classification of the response space. When the responses of normal circuits are similar to those of faulty circuits (i.e., the latter have only small parametric faults), the response space is mixed and traditional algorithms have difficulty in distinguishing the two groups. However, the SVM provides an effective result. This paper also proposes an algorithm to calculate the test sequence for input stimuli using the SVM results. Numerical experiments prove that this algorithm can enhance the precision of test generation.

Fault Detection of Bridging Faults in Digital Circuits by Shared Binary Decision Diagram

A new test generation method for the bridging faults in digital circuits is proposed in this paper, the method is based on shared binary decision diagram. The shared binary decision diagram can represent many logic functions simultaneously by sharing isomorphic subgraphs, it is used to represent the digital circuits with multiple primary outputs. The binary decision diagram is constructed respectively for the normal circuit and faulty circuit having a bridging fault. The test vectors of the bridging fault can be produced by a XOR operation of the two binary decision diagrams. The experimental results on a lot of benchmark circuits demonstrate that the test method proposed in this paper can get the test vectors of the bridging faults if the faults are testable.

Checking experiments for stream X-machines

Stream X-machines are a state based formalism that has associated with it a particular development process in which a system is built from trusted components. Testing thus essentially checks that these components have been combined in a correct manner and that the orders in which they can occur are consistent with the specification. Importantly, there are test generation methods that return a checking experiment: a test that is guaranteed to determine correctness as long as the implementation under test (IUT) is functionally equivalent to an unknown element of a given fault domain Ψ . Previous work has show how three methods for generating checking experiments from a finite state machine (FSM) can be adapted to testing from a stream X-machine. However, there are many other methods for generating checking experiments from an FSM and these have a variety of benefits that correspond to different testing scenarios. This paper shows how any method for generating a checking experiment from an FSM can be adapted to generate a checking experiment for testing an implementation against a stream X-machine. This is the case whether we are testing to check that the IUT is functionally equivalent to a specification or we are testing to check that every trace (input/output sequence) of the IUT is also a trace of a nondeterministic specification. Interestingly, this holds even if the fault domain Ψ used is not that traditionally associated with testing from a stream X-machine. The results also apply for both deterministic and nondeterministic implementations.

Coverage Driven High-Level Test Generation Using a Polynomial Model of Sequential Circuits

This paper proposes a high-level test generation method which considers the control part as well as data path of a register transfer level circuit as a set of polynomial functions to generate behavioral test patterns from faulty behavior instead of comparing the faulty and fault-free circuits based on a hybrid Boolean-word canonical representation called Horner expansion diagram. Since this set of polynomial functions express primary outputs and next states with respect to primary inputs and present states, it is not necessary to perform justification/propagation phase which leads to a minimum number of backtracks. It improves fault coverage and reduces test generation time over logic-level techniques. We assess then the effectiveness of high-level test generation with a simple gate-level automatic test pattern generation algorithm. Experimental results show robustness and reliability of our method compared to other contemporary approaches in terms of fault coverage and CPU time.

A Fault Dependent Test Generation Method for State-Observable FSMs to Increase Defect Coverage under the Test Length Constraint

Since scan testing is not based on the function of the circuit, but rather the structure, it is considered to be both a form of over testing and under testing. Moreover, it is important to test VLSIs using the given function. Since the functional specifications are described explicitly in the FSMs, high test quality is expected by performing logical fault testing and timing fault testing. This paper proposes a fault-dependent test generation method to detect specified fault models completely and to increase defect coverage as much as possible under the test length constraint. We present experimental results for MCNC'91 benchmark circuits to evaluate bridging fault coverage, transition fault coverage, and statistical delay quality level and to show the effectiveness of the proposed test generation method compared with a stuck-at fault-dependent test generation method.

Increasing the Efficiency of Simulation-Based Functional Verification Through Unsupervised Support Vector Analysis

<para xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"> Success of simulation-based functional verification depends on the quality and diversity of the verification tests that are simulated. The objective of test generation methods is to generate tests that exercise as much different functionality of the hardware designs as possible. In this paper, we propose a novel methodology that generates a model of the verification tests in a given test set using unsupervised support vector analysis. One potential application is to use this model to select tests that are likely to exercise functionality that has not been tested so far. Since this selection can be done before simulation, it can be used to filter redundant tests and reduce required simulation cycles. Our methodology can be combined with a test generation method like constrained-random test generation to increase its effectiveness without making fundamental changes to the verification flow. Experimental results based on application of the proposed methodology to the OpenSparc T1 processor are reported to demonstrate the practicality of our approach. </para>

A C-Testable 4-2 Adder Tree for an Easily Testable High-Speed Multiplier

A C-testable 4-2 adder tree for an easily testable high-speed multiplier is proposed, and a recursive method for test generation is shown. By using the specific patterns that we call ‘alternately inverted patterns, ’ the adder tree, as well as partial product generators, can be tested with 14 patterns regardless of its operand size under the cell fault model. The test patterns are easily fed through the partial product generators. The hardware overhead of the 4-2 adder tree with partial product generators for a 64-bit multiplier is about 15%. By using a previously proposed easily testable adder as the final adder, we can obtain an easily testable high-speed multiplier.

Reaching and Distinguishing States of Distributed Systems

Some systems interact with their environment at physically distributed interfaces, called ports, and in testing such a system it is normal to place a tester at each port. Each tester observes only the events at its port and it is known that this limited observational power introduces additional controllability and observability problems into testing. Given a multiport finite state machine (FSM) $M$, we consider the problems of defining strategies for the testers either to reach a given state of $M$ or to distinguish two states of $M$. These are important problems since most techniques for testing from a single-port FSM use sequences that reach and distinguish states. Both problems can be solved in low-order polynomial time for single-port FSMs but we prove that the corresponding decision problems are undecidable for multiport FSMs. However, we also show that they can be solved in low-order polynomial times for deterministic FSMs if we restrict our attention to controllable tests. These results have important ramifications for testing from a multiport FSM since they suggest that methods for testing from a single-port FSM cannot be easily adapted. In addition, two FSMs can be distinguished if and only if their initial states can be distinguished and so the results suggest that, in contrast to single-port FSMs, we cannot expect to produce general complete test generation methods for multiport FSMs.

ATPG-XP: Test Generation for Maximal Crosstalk-Induced Faults

In this paper, we propose a new test-generation method for delay faults considering crosstalk-induced delay effects, based on a conventional delay automatic-test-pattern-generation (ATPG) technique in order to reduce the complexity of previous ATPG algorithms and to consider multiple-aggressor crosstalk faults to maximize the noise of the victim line. Since the proposed ATPG for crosstalk-induced delay faults uses physical and timing information, it can reduce the search space of the backward implication of the aggressor's constraints, and it is helpful for reducing the ATPG time cost compared to previous works. In addition, since the proposed technique targets the critical path for the original delay test as the victim lines, it can improve test effectiveness of delay testing. Experimental results demonstrate the effectiveness of the proposed method.

Process Variation-Aware Test for Resistive Bridges

This paper analyzes the behavior of resistive bridging faults under process variation and shows that process variation has a detrimental impact on test quality in the form of test escapes. To quantify this impact, a novel metric called test robustness is proposed and to mitigate test escapes, a new process variation-aware test generation method is presented. The method exploits the observation that logic faults that have high probability of occurrence and correspond to significant amounts of undetected bridge resistance have a high impact on test robustness and therefore should be targeted by test generation. Using synthesized International Symposium on Circuits and Systems benchmarks with realistic bridge locations, results show that for all the benchmarks, the method achieves better results (less test escapes) than tests generated without consideration of process variation.

Validation of SDL specifications using EFSM-based test generation

Existing methods for testing an SDL specification mainly allow for either black box simulation or conformance testing to verify that the behavior of an implementation matches its corresponding model. However, this relies on the potentially hazardous assumption that the model is completely correct. We propose a test generation method that can accomplish conformance verification as well as coverage criteria-driven white box testing of the specification itself. We first reformat a set of EFSMs equivalent to the processes in an SDL specification and identify “hot spots” – nodes or edges in the EFSM which should be prioritized during testing to effectively increase coverage. Then, we generate test sequences intended to cover selected hot spots; we address the possible infeasibility of such a test sequence by allowing for its rejection decided by a constraint solver and re-generation of an alternate test sequence to the hot spot. In this paper, we present our test generation method and tool, and provide case studies on five SDL processes demonstrating the effectiveness of our coverage-based test sequence selection.

Searching for Optimal Homing Sequences for Testing Timed Communication Protocols

With the eruption of communication technology and its supportive protocols, the field of testing of timed-IUT (Implementation Under Test with time constraints) was quick to follow. New and extended test generation methods using timed automata were in the need. Many papers were written on a broad range of timed automata testing. Few papers, however, dealt with generation of state identification sequences problems. Those who did, keyed their algorithms towards transformation of a timed-IUT into a regular, untimed IUT; thus, avoiding the need of redefining well known generation algorithms. This method, however, only functions if we search for optimal sequences w.r.t. their lengths. Optimality w.r.t. execution time was not considered in this context. In this paper we present a search algorithm for the direct generation of timed homing sequences optimal w.r.t. to execution time. This is accomplished by expanding and modifying the search tree used in previous algorithms such that it contains the capability of satisfying our goal. The solution found is proven to be optimal with respect to execution time. A comparison with other related algorithms is presented.

Coverage-guided test generation for continuous and hybrid systems

In this paper, we describe a formal framework for conformance testing of continuous and hybrid systems, using the international standard `Formal Methods in Conformance Testing' FMCT. We propose a novel test coverage measure for these systems, which is defined using the star discrepancy notion. This coverage measure is used to quantify the validation `completeness'. It is also used to guide input stimulus generation by identifying the portions of the system behaviors that are not adequately examined. We then propose a test generation method, which is based on a robotic motion planning algorithm and is guided by the coverage measure. This method was implemented in a prototype tool that can handle high dimensional systems (up to 100 dimensions).

Three Generalizations to a Generic Integrated Test Generation Method for Finite State Machines

In a previous paper, we proposed a generic test generation method for deterministic implementations of deterministic finite state machines. The method supports a wide class of testing strategies, multi-criteria optimization and integrated handling of all the usual optimization concerns. The present paper generalizes the method to an even wider class of strategies, to non-deterministic machines and to machines handling data.

A formal method to real-time protocol interoperability testing

Interoperability testing is an important technique to ensure the quality of implementations of network communication protocol. In the next generation Internet protocol, real-time applications should be supported effectively. However, time constraints were not considered in the related studies of protocol interoperability testing, so existing interoperability testing methods are difficult to be applied in real-time protocol interoperability testing. In this paper, a formal method to real-time protocol interoperability testing is proposed. Firstly, a formal model CMpTIOA (communicating multi-port timed input output automata) is defined to specify the system under test (SUT) in real-time protocol interoperability testing; based on this model, timed interoperability relation is then defined. In order to check this relation, a test generation method is presented to generate a parameterized test behavior tree from SUT model; a mechanism of executability pre-determination is also integrated in the test generation method to alleviate state space explosion problem to some extent. The proposed theory and method are then applied in interoperability testing of IPv6 neighbor discovery protocol to show the feasibility of this method.