Test Generation Method Research Articles

NLPLego: Assembling Test Generation for Natural Language Processing Applications

With the development of Deep Learning, Natural Language Processing (NLP) applications have reached or even exceeded human-level capabilities in certain tasks. Although NLP applications have shown good performance, they can still have bugs like traditional software and even lead to serious consequences. Inspired by Lego blocks and syntax structure analysis, we propose an assembling test generation method for NLP applications or models and implement it in NLPLego . The key idea of NLPLego is to assemble the sentence skeleton and adjuncts in order by simulating the building of Lego blocks to generate multiple grammatically and semantically correct sentences based on one seed sentence. The sentences generated by NLPLego have derivation relations and different degrees of variation. These characteristics make it well-suited for integration with metamorphic testing theory, addressing the challenge of test oracle absence in NLP application testing. To validate NLPLego , we conduct experiments on three commonly used NLP tasks (i.e., machine reading comprehension, sentiment analysis, and semantic similarity measures), focusing on the efficiency of test generation and the quality and effectiveness of generated tests. We select five advanced NLP models and one popular industrial NLP software as the tested subjects. Given seed tests from SQuAD 2.0, SST, and QQP, NLPLego successfully detects 1,732, 3,140, and 261,879 incorrect behaviors with around 93.1% precision in three tasks, respectively. The experiment results show that NLPLego can efficiently generate high-quality tests for multiple NLP tasks to detect erroneous behaviors effectively. In the case study, we analyze the testing results provided by NLPLego to obtain intuitive representations of the different NLP capabilities of the tested subjects. The case study confirms that NLPLego can provide developers with clarity on the direction to improve NLP models or applications, laying the foundation for enhancing performance.

A Bioinspired Test Generation Method Using Discretized and Modified Bat Optimization Algorithm

The process of software development is incomplete without software testing. Software testing expenses account for almost half of all development expenses. The automation of the testing process is seen to be a technique for reducing the cost of software testing. An NP-complete optimization challenge is to generate the test data with the highest branch coverage in the shortest time. The primary goal of this research is to provide test data that covers all branches of a software unit. Increasing the convergence speed, the success rate, and the stability of the outcomes are other goals of this study. An efficient bioinspired technique is suggested in this study to automatically generate test data utilizing the discretized Bat Optimization Algorithm (BOA). Modifying and discretizing the BOA and adapting it to the test generation problem are the main contributions of this study. In the first stage of the proposed method, the source code of the input program is statistically analyzed to identify the branches and their predicates. Then, the developed discretized BOA iteratively generates effective test data. The fitness function was developed based on the program’s branch coverage. The proposed method was implemented along with the previous one. The experiments’ results indicated that the suggested method could generate test data with about 99.95% branch coverage with a limited amount of time (16 times lower than the time of similar algorithms); its success rate was 99.85% and the average number of required iterations to cover all branches is 4.70. Higher coverage, higher speed, and higher stability make the proposed method suitable as an efficient test generation method for real-world large software.

Enhanced Built-In Self-Diagnosis and Self-Repair Techniques for Daisy-Chain Design in MEDA Digital Microfluidic Biochips

Digital microfluidic biochips have emerged as a promising alternative for various laboratory procedures in biochemistry such as drug discovery and DNA sequencing. A recent generation of digital biochips uses a micro-electrode-dotarray (MEDA) architecture, which provides finer controllability of droplets and seamlessly integrates microelectronics and microfluidics. To simplify the wiring design of such biochips, all microelectrodes and their control registers are daisy-chained together. Therefore, the ability to both identify faults in the chain and tolerate them is required in MEDA biochips. In this study, a new daisy-chain design approach is proposed, integrating a built-in self-repair scheme that can automatically detect faults and correct them. Moreover, an efficient test generation method that requires only a small number of test vectors is proposed to achieve 100% fault coverage without degrading the electrodes. The proposed self-repair scheme can be used in both offline and online modes. Experimental results show that detection and repair can be carried out for various types of faults that can occur in daisy chains.

Increasing the Readiness of Railway Traffic Control Devices Using a Functional Test Generation Method

Railway traffic control (RTC) equipment are technical devices developed for maintaining the safety of rail vehicle traffic operation, and must satisfy certain requirements related to a specific level of traffic safety and effectiveness. They are designed to ensure operational reliability under the intended application conditions throughout their entire service life. Regardless of their type and kind, however, they must behave in accordance with the assumed functional rules and requirements. This is why periodic tests of their correct operation are so crucial. Functional tests are applied to this end. They are employed in the course of device design and approval for use as well as in equipment operational diagnostics. Previous methods for generating test sets and their processing have been intuitive and often rely on the experience of testers that have been acquired in the course of implementing other tests. Such tests may be incomplete or fail to take into account all studied functions. Therefore, this article presents a proprietary method for generating functional tests covering railway traffic control devices that improve their readiness. The authors describe a test determination procedure algorithm and specify the conditions to be met by a test set and by optimization criteria. The article ends with a computer-based exemplification of the developed functional test generation method.

Input Distribution Coverage: Measuring Feature Interaction Adequacy in Neural Network Testing

Testing deep neural networks (DNNs) has garnered great interest in the recent years due to their use in many applications. Black-box test adequacy measures are useful for guiding the testing process in covering the input domain. However, the absence of input specifications makes it challenging to apply black-box test adequacy measures in DNN testing. The Input Distribution Coverage (IDC) framework addresses this challenge by using a variational autoencoder to learn a low dimensional latent representation of the input distribution, and then using that latent space as a coverage domain for testing. IDC applies combinatorial interaction testing on a partitioning of the latent space to measure test adequacy. Empirical evaluation demonstrates that IDC is cost-effective, capable of detecting feature diversity in test inputs, and more sensitive than prior work to test inputs generated using different DNN test generation methods. The findings demonstrate that IDC overcomes several limitations of white-box DNN coverage approaches by discounting coverage from unrealistic inputs and enabling the calculation of test adequacy metrics that capture the feature diversity present in the input space of DNNs.

Towards an Automatic Test Generation Method for Systems of Systems based on Fault Injection and Model-Based Systems Engineering

The emergence and development of systems of systems (SoSs) have expanded the complexity and adaptability of systems engineering. Due to the heterogeneity of its constituent systems, designing and analyzing an SoS faces enormous challenges. Therefore, the verification of an SoS is important in its design phase. However, related methods and techniques are still in the preliminary research state, mainly for requirements verification and system verification, and a lack of efforts in design verification for SoSs. Aiming to provide the ability to detect hazardous states at the design phase, we worked on a novel method for automatically generating test cases for model verification in SoSs. Considering the characteristics of SoSs, the method adopts fault injection and model-based system engineering as the foundations to extend the automated generation capability of test cases. This paper proposes an automatic test generation (ATG) method for an SoS based on fault injection and model-based systems engineering (MBSE). It is meant to generate test cases that can be used to check the effectiveness of fault detection or identification in SoSs and for the effective testing of their constituent systems (CSs). This paper discusses scenario generation and ATG in linear temporal logic and designs an algorithm to generate traces as test suites, taking into account the constraints of the models. By establishing a concept alignment example (CAE) as the experimental use case, three test scenarios are generated automatically, showing that the use of the proposed ATG method provides a reasonable hazardous detection capability for verification in SoS design. The main contributions of the paper are (1) the description of the proposed two-phase ATG approach with fault-centric modeling activities and generation-related activities; (2) the explanation of the designed ATG algorithm at both high and low levels; and (3) the discussion of the experiment in an emergency response CAE with three generated test scenarios.

Adaptive testing for specification coverage and refinement in CPS models

Design of correct cyber–physical systems (CPS) is of uttermost importance for safety-critical applications. This crucial yet extremely challenging property is often addressed in practice by simulation-based methods. The simulation activity can be made more systematic and rigorous by using formal specifications to express requirements and guide the testing of the system.In this paper, we develop a procedure for generating tests from formal specifications given in Signal Temporal Logic (STL), a declarative language used to express CPS requirements. The proposed test generation method is adaptive with the aim at achieving specification coverage. We devise to this goal cooperative reachability games, which we enhance with numerical optimization to facilitate exercising various parts of specifications. The resulting approach is effective in finding specification violations, but also in increasing confidence (via coverage) that the specification is satisfied. In the latter case, we also propose a method for automatically refining the specification into its part that is actually implemented, thus gaining additional insight into the system-under-test.

Automatic Generation of Test Cases Based on Genetic Algorithm and RBF Neural Network

Software testing plays an important role in improving the quality of software, but the design of test cases requires a lot of manpower, material resources, and time, and designers tend to be subjective when designing test cases. To solve this problem and make the test cases have objectivity and greater coverage, a branch coverage test case automatic generation method based on genetic algorithm and RBF neural network algorithm (GAR) is proposed. In terms of test case generation, based on the genetic algorithm optimized in this paper, a certain number of test case samples are randomly selected to train the RBF neural network to simulate the fitness function and to calculate the individual fitness value. The experiment uses 7 C language codes to automatically generate test cases and compares the experimental data generated by the branch coverage test case generation method based on adaptive genetic algorithm (PDGA), traditional genetic algorithm (SGA), and random test generation method (random) to evaluate the proposed algorithm. The experimental results show that the method is feasible and effective, the branch coverage is increased in the generation of test cases, and the number of iterations of the population is less.

Test Generation for Hardware Trojan Detection Using Correlation Analysis and Genetic Algorithm

Hardware Trojan (HT) is a major threat to the security of integrated circuits (ICs). Among various HT detection approaches, side channel analysis (SCA)-based methods have been extensively studied. SCA-based methods try to detect HTs by comparing side channel signatures from circuits under test with those from trusted golden references. The pre-condition for SCA-based HT detection to work is that the testers can collect extra signatures/anomalies introduced by activated HTs. Thus, activation of HTs and amplification of the differences between circuits under test and golden references are the keys to SCA-based HT detection methods. Test vectors are of great importance to the activation of HTs, but existing test generation methods have two major limitations. First, the number of test vectors required to trigger HTs is quite large. Second, the HT circuit’s activities are marginal compared with the whole circuit’s activities. In this article, we propose an optimized test generation methodology to assist SCA-based HT detection. Considering the HTs’ inherent surreptitious nature, inactive nodes with low transition probability are more likely to be selected as HT trigger nodes. Therefore, the correlations between circuit inputs and inactive nodes are first exploited to activate HTs. Then a test reordering process based on the genetic algorithm (GA) is implemented to increase the proportion of the HT circuit’s activities to the whole circuit’s activities. Experiments on 10 selected ISCAS benchmarks, wb_conmax benchmark, and b17 benchmark demonstrate that the number of test vectors required to trigger HTs reduces 28.8% on average compared with the result of MERO and MERS methods. After the test vector reordering process, the proportion of the HT circuit’s activities to the whole circuit’s activities is improved by 95% on average, compared with the result of MERS method.

Variation-Aware Delay Fault Testing for Carbon-Nanotube FET Circuits

Sensitivity to process variations and manufacturing defects are major showstoppers for the high-volume manufacturing of carbon nanotube field-effect transistors (CNFETs). These imperfections affect gate delay and may remain undetected when test patterns obtained using conventional test-generation techniques are used. We propose a new test generation method that takes CNFET-specific process variations into account and identifies multiple testable long paths through each node in a netlist. In contrast to state-of-the-art techniques, our method can also handle variations that have a nonlinear impact on the propagation delay. The generated test patterns ensure the detection of delay faults through the longest path, even under random CNFET process variations. The proposed method shows significant improvement in the statistical delay quality level (SDQL) compared with a state-of-the-art technique and a commercial ATPG tool for multiple benchmarks. We observed a minimum of 17.1% improvement in the SDQL offered by our patterns over a test set of the same size generated by the commercial tool. We also show that our method, when integrated with the conventional transition fault test flow, offers a significant improvement in the quality of test patterns under random variations. Moreover, the proposed method is flexible and can be easily extended to other emerging device technologies.

VP_TT: A value propagation based equivalence checker for testability transformations

Testability transformation (TT) is a source-to-source programme transformation that aims to improve the ability of a given test generation method to generate test data for the original programme. Herein, the correctness of testability transformations is shown. Translation validation is the process of proving that the transformed programme is a correct translation of the source programme being compiled. It is widely used to verify the correctness of various compiler optimizations and transformations during scheduling. The value propagation based equivalence checking (VP) method is an efficient translation validation approach proposed to verify the correctness of various compiler optimization applied during scheduling in high-level synthesis. VP-based translation validation of testability transformations is proposed. In particular, it is identified that the existing VP method fails to show the equivalence for some of the TTs. A dynamic cutpoint selection scheme and an enhancement to the VP method to overcome these limitations are shown. The enhanced VP method, called VP_TT, successfully shows the equivalence for the TTs where the VP method fails. Experimental results confirm the usefulness of VP_TT in the verification of testability transformations.

Test generation algorithm for the All-Transition-State criteria of Finite State Machines

In the current article a novel test generation algorithm is presented for deterministic finite state machine specifications based on the recently introduced All-Transition-State criteria. The size of the resulting test suite and the time required for test suite generation are investigated through analytical and practical analyses and are also compared to the Transition Tour, Harmonized State Identifiers and random walk test generation methods. The fault detection capabilities of the different approaches are also investigated with simulations applying randomly injected transfer faults.

Construction of complete minimal test set for single intra-level bridging and stuck-at faults in reversible circuits

Reversible logic computing is gaining enough interest in the field of low-power circuit design technology. Moreover, the concept of reversible computing is widely used in quantum circuit computation. Therefore, it has motivated researchers to explore reversible logic as a circuit design alternative. Several problems of synthesis and optimisation of the reversible circuits have been reported in the present literature. In this context, the testing plays an important role in the effective performance of these circuits. For detecting all possible faults in the reversible circuits, numerous fault models have been proposed, where many of the proposed fault models common to conventional logic circuits. In this paper, we consider the problem of testing in the NCT or GT library-based reversible circuits with respect to an efficient test set generation for single intra-level bridging faults and single stuck-at faults. The proposed test generation method has been developed based on the concept of one-to-one mapping mechanism (reversible circuit property). Finally, experimental results show that the generated test is capable of 100% fault coverage for both the fault models and also the results are compared with the existing methods for analysing the test set size.

Scalable Activation of Rare Triggers in Hardware Trojans by Repeated Maximal Clique Sampling

Hardware Trojans are serious threat to security and reliability of computing systems. It is hard to detect these malicious implants using traditional validation methods since an adversary is likely to hide them under rare trigger conditions. While existing statistical test generation methods are promising for Trojan detection, they are not suitable for activating extremely rare trigger conditions in stealthy Trojans. To address the fundamental challenge of activating rare triggers, we propose a new test generation paradigm for trigger activation by repeated maximal clique sampling (TARMAC). The basic idea is to utilize a satisfiability modulo theories (SMTs) solver to construct a test corresponding to each maximal clique. This article makes three fundamental contributions: 1) it proves that the trigger activation problem can be mapped to clique cover problem, and the test vectors generated by covering maximal cliques are complete and compact; 2) it proposes efficient test generation algorithms to activate trigger conditions by repeated maximal clique sampling; and 3) it outlines an efficient mechanism to run the clique sampling in parallel to significantly improve the scalability of our test generation framework. The experimental results demonstrate that our proposed approach is scalable and it outperforms state-of-the-art approaches by several orders-of-magnitude in detecting stealthy Trojans.

Can this fault be detected: A study on fault detection via automated test generation

Automated test generation can reduce the manual effort in improving software quality. A test generation method employs code coverage, such as the widely-used branch coverage, to guide the inference of tests. These tests can be used to detect hidden faults. An automatic tool takes a specific type of code coverage as a configurable parameter. Given an automated tool of test generation, a fault may be detected by one type of code coverage, but omitted by another. In frequently released software projects, the time budget of testing is limited. Configuring code coverage for a testing tool can effectively improve the quality of projects. In this paper, we conduct a study on whether a fault can be detected by specific code coverage in automated test generation. We build predictive models with 60 metrics of faulty source code to identify detectable faults under eight types of code coverage. In the experiment, an off-the-shelf tool, EvoSuite is used to generate test data. Experimental results based on four research questions show that different types of code coverage result in the detection of different faults; a code coverage can be used as a supplement to increase the number of detected faults if another coverage is applied first; for each coverage, the number of detected faults increases with its cutoff time in test generation. Our result shows that the choice of code coverage can be learned via multi-objective optimization from sampled faults and directly applied to new faults. This study can be viewed as a preliminary result to support the configuration of code coverage in the application of automated test generation.

Automated Test Case Generation for Digital System Designs: A Mapping Study on VHDL, Verilog, and SystemVerilog Description Languages

Researchers have proposed different methods for testing digital systems and circuits in the last couple of decades. The need for testing digital logic circuits has become more important than ever due to the growing complexity of such systems. During the design phase, testing is focusing on design defects, as well as manufacturing and wear out type of defects. Failures in digital systems could be caused, for example, by design errors, the use of inherently probabilistic devices, and manufacturing variability. As a way to test digital systems in a more efficient way, automated test generation has been proposed to automatically create tests that can quickly and accurately identify faulty components. Examples of such techniques are the sequential test generation, the scan path testing, and the random test generation techniques. With the research domain becoming more mature and growing, it is essential to systematically identify, analyze, and classify these contributions. We performed a systematic mapping study of automated test generation for digital circuits aimed at providing an overview of the application of these techniques. We focused on three of the most widely-used and well-supported hardware description languages (HDLs) for digital systems: Verilog, SystemVerilog, and VHDL. Our results suggest that the majority of the test generation methods for digital circuits are focused on the behavioral and register-transfer design levels. Fault-independent and fault-oriented test generation are the most frequently reported types of test generation methods, while HDL model simulation is the most common test generation technology used to search for test cases in these academic studies. While the results are suggesting a growing interest in this area, the majority of articles are published as conferences papers. Our results show that only 31% of the methods are implemented as software tools and only 63% of all contributions are actually generating executable test cases. This study makes three important contributions, (i) a state-of-the-art of test generation for digital system designs research is provided, (ii) the reported characteristics are identified in both the primary papers and experimental reports, (iii) gaps and opportunities for future test generation for digital system designs research are identified.

Construction of Complete Minimal Test Set for Single intra-level Bridging and Stuck-at Faults in Reversible Circuits

Reversible logic computing is gaining enough interest in the field of low-power circuit design technology. Moreover, the concept of reversible computing is widely used in quantum circuit computation. Therefore, it has motivated researchers to explore reversible logic as a circuit design alternative. Several problems of synthesis and optimisation of the reversible circuits have been reported in the present literature. In this context, the testing plays an important role in the effective performance of these circuits. For detecting all possible faults in the reversible circuits, numerous fault models have been proposed, where many of the proposed fault models common to conventional logic circuits. In this paper, we consider the problem of testing in the NCT or GT library-based reversible circuits with respect to an efficient test set generation for single intra-level bridging faults and single stuck-at faults. The proposed test generation method has been developed based on the concept of one-to-one mapping mechanism (reversible circuit property). Finally, experimental results show that the generated test is capable of 100% fault coverage for both the fault models and also the results are compared with the existing methods for analysing the test set size.

KLEESpectre

Spectre-style attacks disclosed in early 2018 expose data leakage scenarios via cache side channels. Specifically, speculatively executed paths due to branch mis-prediction may bring secret data into the cache, which are then exposed via cache side channels even after the speculative execution is squashed. Symbolic execution is a well-known test generation method to cover program paths at the level of the application software. In this article, we extend symbolic execution with modeling of cache and speculative execution. Our tool KLEE SPECTRE , built on top of the KLEE symbolic execution engine, can thus provide a testing engine to check for data leakage through the cache side channel as shown via Spectre attacks. Our symbolic cache model can verify whether the sensitive data leakage due to speculative execution can be observed by an attacker at a given program point. Our experiments show that KLEE SPECTRE can effectively detect data leakage along speculatively executed paths and our cache model can make the leakage detection more precise.

Speed-Up in Test Methods Using Probabilistic Merit Indicators

Deterministic test generation methods are time consuming and, this has led to emergence of simulation-based approaches. The basis of simulation-based methods is to propose a number of test vectors, evaluate the efficiency of the proposed vectors, and accept or reject them. In these methods, the efficiency of each test vector is traditionally evaluated based on fault coverage of that vector. In this paper an alternative criterion based on probabilistic simulation approaches is proposed that can be calculated much faster than fault coverage. Statistical evaluations confirm a strong correlation between the proposed measure and fault coverage. The proposed measure can be used in simulation-based and meta-heuristic test generation methods to achieve speedup. In order to indicate efficiency of the proposed probabilistic measure, two test generation methods that utilize this measure are suggested and exploited to generate test for ISCAS85 benchmark circuits. These test generation methods are called PMI_PRUNE and PMI_GA. Our results demonstrate that PMI_PRUNE can achieve up to 21.2x, while the maximum speedup in PMI_GA is 5.0x. The average speedup in PMI_PRUNE and PMI_GA is about 9.6x, and 2.7x respectively.

Error metrics determination in functionally approximated circuits using SAT solvers.

Approximate computing is an emerging design paradigm that offers trade-offs between output accuracy and computation efforts by exploiting some applications’ intrinsic error resiliency. Computation of error metrics is of paramount importance in approximate circuits to measure the degree of approximation. Most of the existing techniques for evaluating error metrics apply simulations which may not be effective for evaluation of large complex designs because of an immense increase in simulation runtime and a decrease in accuracy. To address these deficiencies, we present a novel methodology that employs SAT (Boolean satisfiability) solvers for fast and accurate determination of error metrics specifically for the calculation of an average-case error and the maximum error rate in functionally approximated circuits. The proposed approach identifies the set of all errors producing assignments to gauge the quality of approximate circuits for real-life applications. Additionally, the proposed approach provides a test generation method to facilitate design choices, and acts as an important guide to debug the approximate circuits to discover and locate the errors. The effectiveness of the approach is demonstrated by evaluating the error metrics of several benchmark-approximated adders of different sizes. Experimental results on benchmark circuits show that the proposed SAT-based methodology accurately determines the maximum error rate and an average-case error within acceptable CPU execution time in one go, and further provides a log of error-generating input assignments.