Test Generation Techniques Research Articles

Directed Test Generation for Hardware Validation: A Survey

The complexity of hardware designs has increased over the years due to the rapid advancement of technology coupled with the need to support diverse and complex features. The increasing design complexity directly translates to difficulty in verifying functional behaviors as well as non-functional requirements. Simulation is the most widely used form of validation using both random and constrained-random test patterns. The random nature of test sequences can cover a vast majority of scenarios, however, it can introduce unacceptable overhead to cover all possible functional and non-functional scenarios. Directed tests are promising to cover the remaining corner cases and hard-to-detect scenarios. Manual development of directed tests can be time-consuming and error-prone. A promising avenue is to perform automated generation of directed tests. In this article, we provide a comprehensive survey of directed test generation techniques for hardware validation. Specifically, we first introduce the complexity of hardware verification to highlight the need for directed test generation. Next, we describe directed test generation using various automated techniques, including formal methods, concolic testing, and machine learning. Finally, we discuss how to effectively utilize the generated test patterns in different validation scenarios, including pre-silicon functional validation, post-silicon debug, as well as validation of non-functional requirements.

Generating and detecting true ambiguity: a forgotten danger in DNN supervision testing

Deep Neural Networks (DNNs) are becoming a crucial component of modern software systems, but they are prone to fail under conditions that are different from the ones observed during training (out-of-distribution inputs) or on inputs that are truly ambiguous, i.e., inputs that admit multiple classes with nonzero probability in their labels. Recent work proposed DNN supervisors to detect high-uncertainty inputs before their possible misclassification leads to any harm. To test and compare the capabilities of DNN supervisors, researchers proposed test generation techniques, to focus the testing effort on high-uncertainty inputs that should be recognized as anomalous by supervisors. However, existing test generators aim to produce out-of-distribution inputs. No existing model- and supervisor independent technique targets the generation of truly ambiguous test inputs, i.e., inputs that admit multiple classes according to expert human judgment. In this paper, we propose a novel way to generate ambiguous inputs to test DNN supervisors and used it to empirically compare several existing supervisor techniques. In particular, we propose AmbiGuess to generate ambiguous samples for image classification problems. AmbiGuess is based on gradient-guided sampling in the latent space of a regularized adversarial autoencoder. Moreover, we conducted what is {-} to the best of our knowledge {-} the most extensive comparative study of DNN supervisors, considering their capabilities to detect 4 distinct types of high-uncertainty inputs, including truly ambiguous ones. We find that the tested supervisors’ capabilities are complementary: Those best suited to detect true ambiguity perform worse on invalid, out-of-distribution and adversarial inputs and vice-versa.

Toward the Generation of Test Vectors for the Detection of Hardware Trojan Targeting Effective Switching Activity

Hardware Trojans (HTs) are small circuits intentionally designed by an adversary for harmful purposes. These types of circuits are extremely difficult to detect. An HT often requires some specific signals to activate, which are almost impossible to discover. For this reason, test generation for side-channel analysis has gained significant attention in recent times and does not require HT activation. Such test generation techniques aim to generate a large amount of switching activity inside the HT circuit, increasing transient current measurement. However, such methods suffer from either long runtime or reliable results. In this work, a test generation technique is proposed based on the relative switching activity of the circuit to overcome the limitations of the existing works. Initially, the proposed technique measures the impact of each input on rare nets individually using random vector simulation. Potent inputs are selected to obtain a new set of test vectors that provide high relative switching inside a circuit. The proposed method is applied on 11 different ISCAS and 3 ITC 99 benchmark circuits. Experimental results endorse the efficacy of the proposed method outperforming traditional Hamming distance-based re-ordering techniques (up to 20×) while requiring a small runtime.

Fragment-Based Test Generation for Web Apps

Automated model-based test generation presents a viable alternative to the costly manual test creation currently employed for regression testing of web apps. However, existing model inference techniques rely on threshold-based whole-page comparison to establish state equivalence, which cannot reliably identify near-duplicate web pages in modern web apps. Consequently, existing techniques produce inadequate models for dynamic web apps, and fragile test oracles, rendering the generated regression test suites ineffective. We propose a model-based test generation technique, FRAGGEN, that eliminates the need for thresholds, by employing a novel state abstraction based on page fragmentation to establish state equivalence. FRAGGEN also uses fine-grained page fragment analysis to diversify state exploration and generate reliable test oracles. Our evaluation shows that FRAGGEN outperforms existing whole-page techniques by detecting more near-duplicates, inferring better web app models and generating test suites that are better suited for regression testing. On a dataset of 86,165 state-pairs, FRAGGEN detected 123% more near-duplicates on average compared to whole-page techniques. The crawl models inferred by FRAGGEN have 62% more precision and 70% more recall on average. FRAGGEN also generates reliable regression test suites with test actions that have nearly 100% success rate on the same version of the web app even if the execution environment is varied. The test oracles generated by FRAGGEN can detect 98.7% of the visible changes in web pages while being highly robust, making them suitable for regression testing.

Survey of Model-Based Security Testing Approaches in the Automotive Domain

Modern connected or autonomous vehicles (AVs) are highly complex cyber-physical systems. As a result of the high number of different technologies and connectivity features involved, testing these systems to identify security vulnerabilities is a big challenge. Security testing techniques, such as penetration testing, are often manual methods that are applied comparatively late in the vehicle development process. Thus, vulnerabilities are only detected late or after development, leading to higher costs and more patching effort. To reduce the amount of testing resources in general and enable early and automated testing, model-based testing methods have been established in several domains, such as information technology and the automotive domain. The transfer of model-based testing approaches to automotive security testing could help to detect vulnerabilities earlier than other, manual methods by automatically generating, executing, or simulating security tests. In this study, we review the literature on model-based test approaches in the automotive domain. First, we consider security-independent approaches to obtain an overview of applied models, formalisms, test selection criteria, and test generation techniques. In addition, we investigate, whether and how model-based approaches are applied for automotive security testing. Overall, we identified 63 publications related to model-based testing and 29 publications with regard to model-based security testing. The aim of this study is to provide an overview and direct comparison between these approaches. In this manner, the state of model-based security testing in the automotive domain, current challenges, and potential research areas are determined.

A Complete Design-for-Test Scheme for Reconfigurable Scan Networks

Reconfigurable Scan Networks (RSNs) are widely used for accessing instruments offline during debug, test and validation, as well as for performing system-level-test and online system health monitoring. The correct operation of RSNs is essential, and RSNs have to be thoroughly tested. However, due to their inherently sequential structure and complex control dependencies, large parts of RSNs have limited observability and controllability. As a result, certain faults at the interfaces to the instruments, control primitives and scan segments remain undetected by existing test methods. In the paper at hand, Design-for-test (DfT) schemes are developed to overcome the testability problems e.g. by resynthesizing the initial design. A DfT scheme for RSNs is presented, which allows detecting all single stuck-at-faults in RSNs by using existing test generation techniques. The developed scheme analyzes and ensures the testability of all parts of RSNs, which include scan segments, control primitives, and interfaces to the instruments. Therefore, the developed scheme is referred to as a complete DfT scheme. It allows for a test integration to cover multiple fault locations can with a single efficient test sequence and to reduce overall test cost.

Update with care: Testing candidate bug fixes and integrating selective updates through binary rewriting

Enterprise software updates depend on the interaction between user and developer organizations. This interaction becomes especially complex when a single developer organization writes software that services hundreds of different user organizations. Miscommunication during patching and deployment efforts lead to insecure or malfunctioning software installations. While developers oversee the code, the update process starts and ends outside their control. Since developer test suites may fail to capture buggy behavior finding and fixing these bugs starts with user generated bug reports and 3rd party disclosures. The process ends when the fixed code is deployed in production. Any friction between user, and developer results in a delay patching critical bugs.Two common causes for friction are a failure to replicate user specific circumstances that cause buggy behavior and incompatible software releases that break critical functionality. Existing test generation techniques are insufficient. They fail to test candidate patches for post-deployment bugs and to test whether the new release adversely effects customer workloads. With existing test generation and deployment techniques, users cannot choose (nor validate) compatible portions of new versions and retain their previous version’s functionality.We present two new technologies to alleviate this friction. First, Test Generation for Ad Hoc Circumstances transforms buggy executions into test cases. Second, Binary Patch Decomposition allows users to select the compatible pieces of update releases. By sharing specific context around buggy behavior and developers can create specific test cases that demonstrate if their fixes are appropriate. When fixes are distributed by including extra context users can incorporate only updates that guarantee compatibility between buggy and fixed versions.We use change analysis in combination with binary rewriting to transform the old executable and buggy execution into a test case including the developer’s prospective changes that let us generate and run targeted tests for the candidate patch. We also provide analogous support to users, to selectively validate and patch their production environments with only the desired bug-fixes from new version releases.This paper presents a new patching workflow that allows developers to validate prospective patches and users to select which updates they would like to apply, along with two new technologies that make it possible. We demonstrate our technique constructs tests cases more effectively and more efficiently than traditional test case generation on a collection of real world bugs compared to traditional test generation techniques, and provides the ability for flexible updates in real world scenarios.

Random or heuristic? An empirical study on path search strategies for test generation in KLEE

Randomness may be the most straightforward strategy and has been widely used in software engineering. One criticism for random strategy is its aimless for tasks. Therefore, many heuristic strategies have been proposed to improve the random strategy. However, it is impossible to prove that heuristics are better than randomness in theory, especial for software test generation. This open question is always left to empirical study and researchers expect to conclude some guidelines in practice. This paper studies a key technique, path search strategies, of test generation. We conducted an empirical evaluation and comparison among ten concrete path search approaches provided by KLEE, where two approaches belong to random search strategy and eight belong to heuristic search strategy, with 53 GNU Coreutils applications. We also investigated both cases with and without constraint optimization techniques for KLEE-based test generation. The experimental results show that without optimization, one approach from random strategy – random-path – performs better than other techniques in terms of the number of completed paths, statement coverage and branch coverage. These results indicate that random strategy can be a better choice for test generation in most cases without optimization, and heuristic strategies need further investigation. However, when combined with optimization, the selection of search strategy depends on the specific optimization applied. We further analyze the reasons behind the statistical results and provide guidelines to select the appropriate path search strategy for test generation in practice.

A Survey on Assertion-based Hardware Verification

Hardware verification of modern electronic systems has been identified as a major bottleneck due to the increasing complexity and time-to-market constraints. One of the major objectives in hardware verification is to drastically reduce the validation and debug time without sacrificing the design quality. Assertion-based verification is a promising avenue for efficient hardware validation and debug. In this article, we provide a comprehensive survey of recent progress in assertion-based hardware verification. Specifically, we outline how to define assertions using temporal logic to specify expected behaviors in different abstraction levels. Next, we describe state-of-the art approaches for automated generation of assertions. We also discuss test generation techniques for activating assertions to ensure that the generated assertions are valid. Finally, we present both pre-silicon and post-silicon assertion-based validation approaches that utilize simulation, formal methods as well as hybrid techniques. We conclude with a discussion on utilizing assertions for verifying both functional and non-functional requirements.

Minimizing Characterizing sets

• Provides an algorithm that generates compact Characterizing sets from deterministic FSMs . • Shows that constructing minimum sized Characterising sets is PSPACE complete for deterministic FSMs . • Shows that constructing Characterising sets with minimum total number of inputs is NP Complete for deterministic FSMs . • Presents the results of a set of experiment and discussions in a rigorous way. A characterizing set (CS) for a deterministic finite state machine (FSM) M is a set of input sequences that, between them, separate (distinguish) all of the states of M . CSs are used within several test generation techniques that return test suites with guaranteed fault detection power. The number of input sequences in a CS directly affects the cost of applying the resultant test suite. In this paper, we study the complexity of decision problems associated with deriving a smallest CS from an FSM, showing that checking the existence of a CS with K sequences is PSPACE-complete . We also consider the length of a CS, which is the sum of the lengths of the input sequences in the CS. It transpires that the problem of deciding whether there is a CS with length at most K is NP-complete . Motivated by these results, we introduce a heuristic to construct a CS, from a deterministic FSM, with the aim of minimizing the number of input sequences. We evaluated the proposed algorithm by assessing its effect when used within a classical test generation algorithm (the W-method). In the evaluation, we used both randomly generated FSMs and benchmark FSMs. The results are promising, with the proposed algorithm reducing the number of test sequences by 37.3% and decreasing the total length of the test suites by 34.6% on average.

Variation-Aware Delay Fault Testing for Carbon-Nanotube FET Circuits

Sensitivity to process variations and manufacturing defects are major showstoppers for the high-volume manufacturing of carbon nanotube field-effect transistors (CNFETs). These imperfections affect gate delay and may remain undetected when test patterns obtained using conventional test-generation techniques are used. We propose a new test generation method that takes CNFET-specific process variations into account and identifies multiple testable long paths through each node in a netlist. In contrast to state-of-the-art techniques, our method can also handle variations that have a nonlinear impact on the propagation delay. The generated test patterns ensure the detection of delay faults through the longest path, even under random CNFET process variations. The proposed method shows significant improvement in the statistical delay quality level (SDQL) compared with a state-of-the-art technique and a commercial ATPG tool for multiple benchmarks. We observed a minimum of 17.1% improvement in the SDQL offered by our patterns over a test set of the same size generated by the commercial tool. We also show that our method, when integrated with the conventional transition fault test flow, offers a significant improvement in the quality of test patterns under random variations. Moreover, the proposed method is flexible and can be easily extended to other emerging device technologies.

Will Dependency Conflicts Affect My Program's Semantics?

Java projects are often built on top of various third-party libraries. If multiple versions of a library exist on the classpath, JVM will only load one version and shadow the others, which we refer to as <i>dependency conflicts</i> . This would give rise to <i>semantic conflict</i> (SC) issues, if the library APIs referenced by a project have identical method signatures but inconsistent semantics across the loaded and shadowed versions of libraries. SC issues are difficult for developers to diagnose in practice, since understanding them typically requires domain knowledge. Although adapting the existing test generation technique for dependency conflict issues, <small>Riddle</small> , to detect SC issues is feasible, its effectiveness is greatly compromised. This is mainly because <small>Riddle</small> randomly generates test inputs, while the SC issues typically require specific arguments in the tests to be exposed. To address that, we conducted an empirical study of 316 real SC issues to understand the characteristics of such specific arguments in the test cases that can capture the SC issues. Inspired by our empirical findings, we propose an automated testing technique <small>Sensor</small> , which synthesizes test cases using ingredients from the project under test to trigger inconsistent behaviors of the APIs with the same signatures in conflicting library versions. Our evaluation results show that <small>Sensor</small> is effective and useful: it achieved a <inline-formula><tex-math notation="LaTeX">$Precision$</tex-math></inline-formula> of 0.898 and a <inline-formula><tex-math notation="LaTeX">$Recall$</tex-math></inline-formula> of 0.725 on open-source projects and a <inline-formula><tex-math notation="LaTeX">$Precision$</tex-math></inline-formula> of 0.821 on industrial projects; it detected 306 semantic conflict issues in 50 projects, 70.4 percent of which had been confirmed as real bugs, and 84.2 percent of the confirmed issues have been fixed quickly.

MaxSense

Detection of hardware Trojans is vital to ensure the security and trustworthiness of System-on-Chip (SoC) designs. Side-channel analysis is effective for Trojan detection by analyzing various side-channel signatures such as power, current, and delay. In this article, we propose an efficient test generation technique to facilitate side-channel analysis utilizing dynamic current. While early work on current-aware test generation has proposed several promising ideas, there are two major challenges in applying it on large designs: (i) The test generation time grows exponentially with the design complexity, and (ii) it is infeasible to detect Trojans, since the side-channel sensitivity is marginal compared to the noise and process variations. Our proposed work addresses both challenges by effectively exploiting the affinity between the inputs and rare (suspicious) nodes. The basic idea is to quickly find the profitable ordered pairs of test vectors that can maximize side-channel sensitivity. This article makes two important contributions: (i) It proposed an efficient test generation algorithm that can produce the first patterns in the test vectors to maximize activation of suspicious nodes using an SMT solver, and (ii) it developed a genetic-algorithm based test generation technique to produce the second patterns in the test vectors to maximize the switching in the suspicious regions while minimizing the switching in the rest of the design. Our experimental results demonstrate that we can drastically improve both the side-channel sensitivity (62× on average) and time complexity (13× on average) compared to the state-of-the-art test generation techniques.

Enabledness-based Testing of Object Protocols

A significant proportion of classes in modern software introduce or use object protocols, prescriptions on the temporal orderings of method calls on objects. This article studies search-based test generation techniques that aim to exploit a particular abstraction of object protocols (enabledness preserving abstractions (EPAs)) to find failures. We define coverage criteria over an extension of EPAs that includes abnormal method termination and define a search-based test case generation technique aimed at achieving high coverage. Results suggest that the proposed case generation technique with a fitness function that aims at combined structural and extended EPA coverage can provide better failure-detection capabilities not only for protocol failures but also for general failures when compared to random testing and search-based test generation for standard structural coverage.

Automated Test Case Generation for Digital System Designs: A Mapping Study on VHDL, Verilog, and SystemVerilog Description Languages

Researchers have proposed different methods for testing digital systems and circuits in the last couple of decades. The need for testing digital logic circuits has become more important than ever due to the growing complexity of such systems. During the design phase, testing is focusing on design defects, as well as manufacturing and wear out type of defects. Failures in digital systems could be caused, for example, by design errors, the use of inherently probabilistic devices, and manufacturing variability. As a way to test digital systems in a more efficient way, automated test generation has been proposed to automatically create tests that can quickly and accurately identify faulty components. Examples of such techniques are the sequential test generation, the scan path testing, and the random test generation techniques. With the research domain becoming more mature and growing, it is essential to systematically identify, analyze, and classify these contributions. We performed a systematic mapping study of automated test generation for digital circuits aimed at providing an overview of the application of these techniques. We focused on three of the most widely-used and well-supported hardware description languages (HDLs) for digital systems: Verilog, SystemVerilog, and VHDL. Our results suggest that the majority of the test generation methods for digital circuits are focused on the behavioral and register-transfer design levels. Fault-independent and fault-oriented test generation are the most frequently reported types of test generation methods, while HDL model simulation is the most common test generation technology used to search for test cases in these academic studies. While the results are suggesting a growing interest in this area, the majority of articles are published as conferences papers. Our results show that only 31% of the methods are implemented as software tools and only 63% of all contributions are actually generating executable test cases. This study makes three important contributions, (i) a state-of-the-art of test generation for digital system designs research is provided, (ii) the reported characteristics are identified in both the primary papers and experimental reports, (iii) gaps and opportunities for future test generation for digital system designs research are identified.

White-box software test generation with Microsoft Pex on open source C# projects: A dataset.

The paper presents a dataset on software tests generated using the Microsoft Pex (IntelliTest) test generator tool for 10 open source projects. The projects were selected randomly from popular GitHub repositories written in C#. The selected projects contain 7187 methods from which Pex was able to generate tests for 2596 methods totaling 38,618 lines of code. Data collection was performed on a cloud virtual machine. The dataset presents metrics about the attributes of the selected projects (e.g., cyclomatic complexity or number of external method calls) and the test generation (e.g., statement and branch coverage, number of warnings). This data is compared to an automated isolation technique in the paper Automated Isolation for White-box Test Generation [1]. To the best of our knowledge, this is the largest public dataset about the test generation performance of Microsoft Pex on open source projects. The dataset highlights current practical challenges and can be used as a baseline for new test generation techniques.

Scalable Concolic Testing of RTL Models

Simulation is widely used for validation of Register-Transfer-Level (RTL) models. While simulating with millions of random or constrained-random tests can cover majority of the functional scenarios, the number of remaining scenarios can still be huge (hundreds or thousands) in case of today's industrial designs. Hard-to-activate branches are one of the major contributors for such remaining/untested scenarios. While directed test generation techniques using formal methods are promising in activating branches, it is infeasible to apply them on large designs due to state space explosion. In this article, we propose a fully automated and scalable approach to cover the hard-to-activate branches using concolic testing of RTL models. While application of concolic testing on hardware designs has shown some promising results in improving the overall coverage, they are not designed to activate specific targets such as uncovered corner cases and rare scenarios. In other words, existing concolic testing approaches address state space explosion problem but leads to path explosion problem while searching for the uncovered targets. Our proposed approach maps directed test generation problem to target search problem while avoiding overlapping searches involving multiple targets. This article makes two important contributions. (1) We propose a directed test generation technique to activate a target by effective utilization of concolic testing on RTL models. (2) We develop efficient learning and clustering techniques to minimize the overlapping searches across targets to drastically reduce the overall test generation effort. Experimental results demonstrate that our approach significantly outperforms the state-of-the-art methods in terms of test generation time (up to 205X, 69X on average) as well as memory requirements (up to 31X, 7X on average).

An Empirical Comparison of Combinatorial Testing, Random Testing and Adaptive Random Testing

We present an empirical comparison of three test generation techniques, namely, Combinatorial Testing (CT), Random Testing (RT) and Adaptive Random Testing (ART), under different test scenarios. This is the first study in the literature to account for the (more realistic) testing setting in which the tester may not have complete information about the parameters and constraints that pertain to the system, and to account for the challenge posed by faults (in terms of failure rate). Our study was conducted on nine real-world programs under a total of 1683 test scenarios (combinations of available parameter and constraint information and failure rate). The results show significant differences in the techniques’ fault detection ability when faults are hard to detect (failure rates are relatively low). CT performs best overall; no worse than any other in 98 percent of scenarios studied. ART enhances RT, and is comparable to CT in 96 percent of scenarios, but its computational cost can be up to 3.5 times higher than CT when the program is highly constrained. Additionally, when constraint information is unavailable for a highly-constrained program, a large random test suite is as effective as CT or ART, yet its computational cost of test generation is significantly lower than that of other techniques.

MAESTRO: Automated test generation framework for high test coverage and reduced human effort in automotive industry

Abstract Context The importance of automotive software has been rapidly increasing because software controls many components of motor vehicles such as smart-key system, tire pressure monitoring system, and advanced driver assistance system. Consequently, the automotive industry spends a large amount of human effort to test automotive software and is interested in automated testing techniques to ensure high-quality automotive software with reduced human effort. Objective Applying automated test generation techniques to automotive software is technically challenging because of false alarms caused by imprecise test drivers/stubs and lack of tool supports for symbolic analysis of bit-fields and function pointers in C. To address such challenges, we have developed an automated testing framework MAESTRO. Method MAESTRO automatically builds a test driver and stubs for a target task (i.e., a software unit consisting of target functions). Then, it generates test inputs to a target task with the test driver and stubs by applying concolic testing and fuzzing together in an adaptive way. In addition, MAESTRO transforms a target program that uses bit-fields into a semantically equivalent one that does not use bit-fields. Also, MAESTRO supports symbolic function pointers by identifying the candidate functions of a symbolic function pointer through static analysis. Results MAESTRO achieved 94.2% branch coverage and 82.3% MC/DC coverage on the four target modules (238 KLOC) developed by Hyundai Mobis. Furthermore, it significantly reduced the cost of coverage testing by reducing the manual effort for coverage testing by 58.8%. Conclusion By applying automated testing techniques, MAESTRO can achieve high test coverage for automotive software with significantly reduced manual testing effort.

DEMINER: test generation for high test coverage through mutant exploration

SummaryMost software testing techniques test a target program as it is and fail to utilize valuable information of diverse test executions on many variants/mutants of the original program in test generation.This paper proposes a new test generation technique DEMINER, which utilizes mutant executions to guide test generation on the original program for high test coverage.DEMINER first generates various mutants of an original target program and then extracts runtime information of mutant executions, which covered unreached branches by the mutation effects. Using the obtained runtime information, DEMINER inserts guideposts, artificial branches to replay the observed mutation effects, to the original target programs. Finally, DEMINER runs automated test generation on the original program with guideposts and achieves higher test coverage.We implemented DEMINER for C programs through software mutation and guided test generation such as concolic testing and fuzzing. We have shown the effectiveness of DEMINER on six real‐world target programs: Busybox‐ls, Busybox‐printf, Coreutils‐sort, GNU‐find, GNU‐grep and GNU‐sed. The experiment results show that DEMINER improved branch coverage by 63.4% and 19.6% compared with those of the conventional concolic testing techniques and the conventional fuzzing techniques on average, respectively.