Test Generation Approach Research Articles

Unifying Behavioral and Feature Modeling for Testing of Software Product Lines

Existing software product line (SPL) engineering testing approaches generally provide positive testing that validates the SPL’s functionality. Negative testing is commonly neglected. This research aims to unify behavioral and feature models of an SPL, enable testing before and after variability binding for domain-centric and product-centric testing, and combine positive and negative testing for a holistic testing view. This study suggests behavioral modeling with event sequence graphs (ESGs). This heterogeneous modeling strategy supports bottom-up domain testing and top-down product testing with the feature model. This new feature-oriented ESG test creation method generates shorter test sequences than the original ESG optimum test sequences. Statechart and original ESG test-generating methods are compared. Positive testing findings are similar. The Statechart technique generated 12 test cases with 59 events, whereas the ESG technique created six test cases with 60 events. The ESG technique generated 205 negative test cases with 858 events with the Test Suite Designer tool. However, the Conformiq Designer tool for the Statechart technique does not have a negative test case generation capability. It is shown that the proposed ESG-based holistic approach confirms not only the desirable (positive) properties but also the undesirable (negative) ones. As an additional research, the traditional ESG test-generating approach is compared to the new feature-oriented method on six SPLs of different sizes and features. Our case study results show that the traditional ESG test generation approach demonstrated higher positive test generation scores compare to the proposed feature-oriented test generation approach. However, our proposed feature-oriented test generation approach is capable of generating shorter test sequences, which could be beneficial for reducing the execution time of test cases compared to traditional ESG approach. Finally, our case study has also shown that regardless of the test generation approach, there has been found no significant difference between the Bottom-up and Top-down test strategies with respect to their positive test generation scores.

Compact Functional Testing for Neuromorphic Computing Circuits

We address the problem of testing Artificial Intelligence (AI) hardware accelerators implementing Spiking Neural Networks (SNNs). We define a metric to quickly rank available samples for training and testing based on their fault detection capability. The metric measures the inter-class spike count difference of a sample for the fault-free design. In particular, each sample is assigned a score equal to the spike count difference between the first two top classes. The hypothesis is that samples with small scores achieve high fault coverage because they are prone to misclassification, i.e., a small perturbation in the network due to a fault will result in these samples being misclassififed with high probability. We show that the proposed metric correlates with the per-sample fault coverage and that retaining a set of high-ranked samples in the order of ten achieves near perfect fault coverage for critical faults that affect the SNN accuracy. The proposed test generation approach is demonstrated on two SNNs modelled in Python and on actual neuromorphic hardware. We discuss fault modeling and perform an analysis to reduce the fault space so as to speed up test generation time.

A model learning based testing approach for kernel P systems

Kernel P systems have been introduced as a unifying formalism allowing to specify, simulate and analyse various problems. Several applications of this model have been considered and a powerful tool built in order to support their development and analysis. Testing represents an important aspect of any system analysis and correctness. In this paper we introduce for the first time a bounded test generation approach for kernel P systems by considering bounded input sequences. A learning algorithm for kernel P systems is based on learning X-machine models that are equivalent to these systems for sequences of steps up to a certain limit, ℓ. The Lℓ learning algorithm is used. The testing approach is then devised from the inferred X-machines. The method is applied to a case study illustrating the key parts of the approach.

Test infrastructure and environment

This issue contains two papers. These papers focus on test infrastructure and testing environment, respectively. Falling into the category of Software Note, the first paper, “JUGE: An infrastructure for benchmarking Java unit test generators” by Xavier Devroey, Alessio Gambi, Juan Pablo Galeotti, René Just, Fitsum Kifetew, Annibale Panichella and Sebastiano Panichella, presents JUnit Generation Benchmarking Infrastructure (JUGE), which supports test generators (such as search-based, random-based and symbolic execution) by automating the production of unit tests for validation, regression testing, fault localization and so on. JUGE aims to reduce the overall benchmarking effort, ease the comparison of multiple generators and enhance the knowledge transfer between academia and industry. JUGE was used and evolved during multiple editions of a unit testing tool competition since 2013. The authors discuss the expected impact of JUGE in improving the knowledge transfer on tools and approaches for test generation between academia and industry (recommended by Marc Roper). Falling into the typical category of Research Article, the second paper, “Test environments for large-scale software systems—An industrial study of intrinsic and extrinsic success factors” by Torvald Mårtensson, Göran Ancher and Daniel Ståhl, presents a study that investigates the characteristics of a test environment to support an organisation's testing objectives. The study includes a series of interviews with 30 individuals, a series of focus groups with 31 individuals and a cross-company workshop with 30 participants from five large-scale companies in different industry segments. The study findings reveal a list of success factors, including (1) characteristics and capabilities existing within a test environment (intrinsic success factors) and (2) properties not inherent to the test environment, but still vital for a successfully implemented test environment (extrinsic success factors). All the five companies included in the study well acknowledge the reported list of intrinsic and extrinsic success factors (recommended by Per Runeson).

Automatically generating test cases for safety-critical software via symbolic execution

Automated test generation based on symbolic execution can be beneficial for systematically testing safety-critical software, to facilitate test engineers to pursue the strict testing requirements mandated by the certification standards, while controlling at the same time the costs of the testing process. At the same time, the development of safety-critical software is often constrained with programming languages or coding conventions that ban linguistic features which are believed to downgrade the safety of the programs, e.g., they do not allow dynamic memory allocation and variable-length arrays, limit the way in which loops are used, forbid recursion, and bound the complexity of control conditions. As a matter of facts, these linguistic features are also the main efficiency-blockers for the test generation approaches based on symbolic execution at the state of the art.This paper contributes new evidence of the effectiveness of generating test cases with symbolic execution for a significant class of industrial safety critical-systems. We specifically focus on Scade, a largely adopted model-based development language for safety-critical embedded software, and we report on a case study in which we exploited symbolic execution to automatically generate test cases for a set of safety-critical programs developed in Scade. To this end, we introduce an original test generator that we developed in a recent industrial project on testing safety-critical railway software written in Scade, and we report on our experience of using this test generator for testing a set of Scade programs that belong to the development of an on-board signaling unit for high-speed rail. The results provide empirically evidence that symbolic execution is indeed a viable approach for generating high-quality test suites for the safety-critical programs considered in our case study.

EveCheck: An Event-Driven, Scalable Algorithm for Coherent Shared Memory Verification

Cache coherence and relaxed memory consistency challenge the design verification of multicore chips. The well-known self-checking approach based on litmus tests has been successful in uncovering design errors, but it leads to limited coverage. That is why random or directed test generation approaches are used for proper coverage control, but they require a specialized checker to verify shared memory behavior. Unfortunately, the inference-based checkers that provide guarantees against false positives and false negatives are not scalable with core count (unless some memory orderings are known beforehand). On the other hand, scoreboard-based checkers are scalable, but do not offer guarantees against false diagnosis. This article proposes an event-driven algorithm that is scalable without renouncing verification quality, because it relies on direct partial order verification, on-the-fly detection of improper memory orderings, exploitation of extended observability not hampered by design optimizations, and agnostic handling of store atomicity at the implementation level. We compared the new algorithm with two scoreboard-based and one inference-based checker reported in the literature. EveCheck often reaches full error detection with half the test size required by the other checkers, has superior verification quality, and less sensitivity to test generation parameters. Its effort to detect errors was one order of magnitude inferior as compared to the inference-based checker. It did not raise false positives in correct designs, and it was able to discover all the errors studied in faulty designs.

Coloured Petri nets for abstract test generation in software engineering

SummaryModel‐based testing (MBT) relies on models of the system's behaviour to generate abstract tests. Testers can reuse formal models using MBT to increase confidence in critical systems (e.g., medical and avionic systems). In this article, we investigate the current abstract test generation approaches for CPN to provide insights for testers who need to select a suitable one when applying the MBT using CPN. We conduct a systematic literature review to investigate the existing abstract test generation approaches designed for CPN. Subsequently, focusing on specific implementations and advantages/disadvantages, we experiment with formal models of medical systems during our empirical analysis to improve the discussion on existing abstract test generation approaches for CPN. Our study shows that CPN provides reliable tests quickly, depending on the abstract test generation approach applied.

JUGE: An infrastructure for benchmarking Java unit test generators

SummaryResearchers and practitioners have designed and implemented various automated test case generators to support effective software testing. Such generators exist for various languages (e.g., Java, C#, or Python) and various platforms (e.g., desktop, web, or mobile applications). The generators exhibit varying effectiveness and efficiency, depending on the testing goals they aim to satisfy (e.g., unit‐testing of libraries versus system‐testing of entire applications) and the underlying techniques they implement. In this context, practitioners need to be able to compare different generators to identify the most suited one for their requirements, while researchers seek to identify future research directions. This can be achieved by systematically executing large‐scale evaluations of different generators. However, executing such empirical evaluations is not trivial and requires substantial effort to select appropriate benchmarks, setup the evaluation infrastructure, and collect and analyse the results. In this Software Note, we present ourJUnit Generation Benchmarking Infrastructure(JUGE) supporting generators (search‐based, random‐based, symbolic execution, etc.) seeking to automate the production of unit tests for various purposes (validation, regression testing, fault localization, etc.). The primary goal is to reduce the overall benchmarking effort, ease the comparison of several generators, and enhance the knowledge transfer between academia and industry by standardizing the evaluation and comparison process. Since 2013, several editions of a unit testing tool competition, co‐located with the Search‐Based Software Testing Workshop, have taken place whereJUGEwas used and evolved. As a result, an increasing amount of tools (over 10) from academia and industry have been evaluated onJUGE, matured over the years, and allowed the identification of future research directions. Based on the experience gained from the competitions, we discuss the expected impact ofJUGEin improving the knowledge transfer on tools and approaches for test generation between academia and industry. Indeed, theJUGEinfrastructure demonstrated an implementation design that is flexible enough to enable the integration of additional unit test generation tools, which is practical for developers and allows researchers to experiment with new and advanced unit testing tools and approaches.

Автоматическое тестирование LLVM-программ со сложными входными структурами данных

Symbolic execution is a widely used approach for automatic regression test generation and bug and vulnerability finding. The main goal of this paper is to present a practical symbolic execution-based approach for LLVM programs with complex input data structures. The approach is based on the well-known idea of lazy initialization, which frees the user from providing constraints on input data structures manually. Thus, it provides us with a fully automatic symbolic execution of even complex program. Two lazy initialization improvements are proposed for segmented memory models: one based on timestamps and one based on type information. The approach is implemented in the KLEE symbolic virtual machine for the LLVM platform and tested on real C data structures — lists, binomial heaps, AVL trees, red-black trees, binary trees, and tries.

Grammar Based Directed Testing of Machine Learning Systems

The massive progress of machine learning has seen its application over a variety of domains in the past decade. But how do we develop a systematic, scalable and modular strategy to validate machine-learning systems? We present, to the best of our knowledge, the first approach, which provides a systematic test framework for machine-learning systems that accepts grammar-based inputs. Our <sc xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">Ogma</small> approach automatically discovers erroneous behaviours in classifiers and leverages these erroneous behaviours to improve the respective models. <sc xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">Ogma</small> leverages inherent robustness properties present in any well trained machine-learning model to direct test generation and thus, implementing a scalable test generation methodology. To evaluate our <sc xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">Ogma</small> approach, we have tested it on three real world natural language processing (NLP) classifiers. We have found thousands of erroneous behaviours in these systems. We also compare <sc xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">Ogma</small> with a random test generation approach and observe that <sc xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">Ogma</small> is more effective than such random test generation by up to 489 percent.

Cooperative verifier-based testing with CoVeriTest

Testing is a widely applied technique to evaluate software quality, and coverage criteria are often used to assess the adequacy of a generated test suite. However, manually constructing an adequate test suite is typically too expensive, and numerous techniques for automatic test-suite generation were proposed. All of them come with different strengths. To build stronger test-generation tools, different techniques should be combined. In this paper, we study cooperative combinations of verification approaches for test generation, which exchange high-level information. We present CoVeriTest, a hybrid technique for test-suite generation. CoVeriTest iteratively applies different conditional model checkers and allows users to adjust the level of cooperation and to configure individual time limits for each conditional model checker. In our experiments, we systematically study different CoVeriTest cooperation setups, which either use combinations of explicit-state model checking and predicate abstraction, or bounded model checking and symbolic execution. A comparison with state-of-the-art test-generation tools reveals that CoVeriTest achieves higher coverage for many programs (about 15%).

Using search-based techniques for testing executable software models specified through graph transformations

Design by contract is a software development methodology that uses contracts for defining interfaces among interacting components of a software system. Graph transformation system is used to specify the behavioral aspects of software components by defining the pre- and post-conditions of methods as contracts. In this paper, we focus on testing executable software models specified by a graph transformation system. A set of model-specific coverage criteria and a cost-aware search-based test generation approach are introduced. To evaluate the effectiveness of the proposed coverage criteria and the test generation approach, a type of mutation analysis is presented at the model level. Furthermore, a couple of fault-detection methods are used to assess the quality of the generated tests in the model-level mutation analysis. The proposed approach is implemented in GROOVE, a toolset for model checking graph transformation systems. The empirical results based on some well-known case studies demonstrate the efficiency and scalability of each proposed coverage criterion and testing approach. The comparison of the proposed test generation approach with state-of-the-art techniques indicates a significant improvement in terms of fault-detection capability and testing costs.

A Directed Test Generator for Shared-Memory Verification of Multicore Chip Designs

The functional verification of multicore chips requires the generation of parallel test programs able to expose design errors and ensure high coverage in less time. Albeit the coherence hardware can scale gracefully as the number of cores grows, the state space of the coherence protocol increases exponentially. That is why this article describes a directed test generation approach that exploits random test generation (RTG) for avoiding explicit enumeration of the coherence state space while memory consistency is verified. The novel approach was designed for synergy between a data-driven engine that explores neighborhoods toward higher coverage and a model-based engine that exploits constraints while driving RTG toward faster coverage evolution. As compared to a state-of-the-art data-driven generator and to a model-based generator, the proposed approach led to superior coverage evolution with time, when targeting 32-core designs relying on different protocols. For MOESI 2-level, the novel approach was from 4.8 to 18.7 faster to reach the data-driven generator’s maximal coverage, and it was up to 2.7 faster to reach the model-driven generator’s. For MESI 3-level, it found, in 10 to 15 min, a few errors whose detection required the data-driven generator 45 min to 7 h.

A compressive sensing algorithm for hardware trojan detection

Traditionally many fabless companies outsource the fabrication of IC design to the foundries, which may not be trusted always. In order to ensure trusted IC’s it is more significant to develop an efficient technique that detects the presence of hardware Trojan. This malicious insertion causes the logic variation in the nets or leaks some sensitive information from the chip, which reduces the reliability of the system. The conventional testing algorithm for generating test vectors reduces the detection sensitivity due to high process variations. In this work, we present a compressive sensing approach, which can significantly generate optimal test patterns compared to the ATPG vectors. This approach maximizes the probability of Trojan circuit activation, with a high level of Trojan detection rate. The side channel analysis such as power signatures are measured at different time stamps to isolate the Trojan effects. The effect of process noise is minimized by this power profile comparison approach, which provides high detection sensitivity for varying Trojan size and eliminates the requirement of golden chip. The proposed test generation approach is validated on ISCAS benchmark circuits, which achieves Trojan detection coverage on an average of 88.6% reduction in test length when compared to random pattern.

Test Generation and Test Prioritization for Simulink Models with Dynamic Behavior

All engineering disciplines are founded and rely on models, although they may differ on purposes and usages of modeling. Among the different disciplines, the engineering of Cyber Physical Systems (CPSs) particularly relies on models with dynamic behaviors (i.e., models that exhibit time-varying changes). The Simulink modeling platform greatly appeals to CPS engineers since it captures dynamic behavior models. It further provides seamless support for two indispensable engineering activities: (1) automated verification of abstract system models via model simulation, and (2) automated generation of system implementation via code generation. We identify three main challenges in the verification and testing of Simulink models with dynamic behavior, namely incompatibility, oracle and scalability challenges. We propose a Simulink testing approach that attempts to address these challenges. Specifically, we propose a black-box test generation approach, implemented based on meta-heuristic search, that aims to maximize diversity in test output signals generated by Simulink models. We argue that in the CPS domain test oracles are likely to be manual and therefore the main cost driver of testing. In order to lower the cost of manual test oracles, we propose a test prioritization algorithm to automatically rank test cases generated by our test generation algorithm according to their likelihood to reveal a fault. Engineers can then select, according to their test budget, a subset of the most highly ranked test cases. To demonstrate scalability, we evaluate our testing approach using industrial Simulink models. Our evaluation shows that our test generation and test prioritization approaches outperform baseline techniques that rely on random testing and structural coverage.

An automated model‐based approach for unit‐level performance test generation of mobile applications

AbstractMobile devices have limited resources, including memory and processing speed. The performance of mobile applications is an important concern. There are a large number of mobile platforms available with varying operating systems and hardware. Native applications are usually developed and maintained separately for these platforms. The overall performance of native applications may significantly vary across platforms. The current industrial practice is to manually test the performance for each variant, which is not a scalable or efficient approach. We tackled the problem of generating native application variants in our previous work. This paper proposes an automated model‐based approach for performance test generation for native application variants at unit level. We propose a performance profile that allows modeling of domain‐specific performance parameters on UML models, which are used for automated performance test generation for each native variant. The results of applying the approach on two real‐world applications show that the approach evaluates the performance of application variants for two different versions of Android successfully and have potential to reduce the effort and time. A questionnaire‐based study is conducted to evaluate the usefulness of the approach.

Introducing plant features to model-based testing of programmable controllers in automation systems

This paper proposes a model-based test generation approach for programmable controllers that aims at reducing the length of a test sequence by applying plant features. The proposed approach does not require detailed or full knowledge of the plant behavior of a system under test, but it can achieve a remarkable reduction with simple plant features. As a result, the obtained test sequence can be significantly shorter than ones generated by complete testing methods; and meanwhile, it still reaches full coverage of the nominal behavior of the system under test. This makes it feasible to test large-scale systems, or to serve as an early test in the validation of safety critical systems. The proposed approach has been illustrated on a large-scale case study.

A Statistical Test Generation Based on Mutation Analysis for Improving the Hardware Trojan Detection

Hardware Trojan has become a major threat to the security and trustworthiness of integrated circuit (IC) employed in critical applications. Due to the presence of process variations and measurement noises, all existing side-channel Trojan detection approaches suffer from low detection sensitivity or even false negatives with increasing circuit size and decreasing Trojan size. In this paper, we propose a statistical test generation approach based on mutation analysis, which generates a set of test vectors aiming at activating the hardware Trojan inserted into the low activity nodes. Such approach not only enhances the controllability of low activity nodes through increasing the switching activity of it, but also improves the observability by propagating the artificial designed errors introduced by the mutant to the outputs. Simulation results of a set of ISCAS’85 and ISCAS’89 benchmark circuits show that the proposed approach improves the activity of low activity nodes 463% at most compared with the Multiple Excitation of Rare Occurrence (MERO) approach and increases the Trojan coverage with 84.08% reduction in test length. Moreover, the test vectors generated by the proposed approach and the MERO approach, respectively, are exerted to the circuit under test. Experimental results demonstrate that the Mahalanobis distance margin of the proposed approach is much greater than the MERO approach, and thus provide a comparable robustness with decreasing Trojan size.

Decomposition-Based Approach for Model-Based Test Generation

Model-based test generation by model checking is a well-known testing technique that, however, suffers from the state explosion problem of model checking and it is, therefore, not always applicable. In this paper, we address this issue by decomposing a system model into suitable subsystem models separately analyzable. Our technique consists in decomposing that portion of a system model that is of interest for a given testing requirement, into a tree of subsystems by exploiting information on model variable dependency. The technique generates tests for the whole system model by merging tests built from those subsystems. We measure and report effectiveness and efficiency of the proposed decomposition-based test generation approach, both in terms of coverage and time.

Test case generation, selection and coverage from natural language

In Model-based Testing (MBT), test cases are automatically generated from a formal model of the system. A disadvantage of MBT is that developers must deal with formal notations. This limitation is addressed in this paper, where use cases are used to model the system. In previous work, we have proposed an automatic strategy for generating test cases from use cases written in a Controlled Natural Language (CNL), which is an English textual notation with a well-defined grammar. Due to its precise syntax, it can be processed and translated into a formal representation for the purpose of automatic test case generation. This paper extends our previous work by proposing a state-based CNL for describing use case control flows enriched with state and data operations. We translate state-based use case descriptions into CSP processes from which test cases can be automatically generated. In addition, we show how a similar notation can be used to specify test selection via the definition of state-based test purposes, which are also translated into CSP processes. Test generation and selection are mechanised by refinement checking using the CSP tool FDR. Despite the fact that we work at a purely process algebraic level to define a test generation strategy, we are able to address model coverage criteria. Particularly, by using FDR, it is possible to have access to the underlying LTS models; we then implemented algorithms for covering events or transitions, possibly combined with selection using test purposes. We also discuss several ways of improving the efficiency of the test generation strategy. As far as we are aware, this integration between an algebraic approach to test case generation with an operational approach for coverage criteria is an original and promising insight. All steps of the strategy are integrated into a tool that provides a GUI for authoring use cases and test purposes described in the proposed CNL, so the formal CSP notation is completely hidden from the test designer. We illustrate our tool and techniques with a running example and a more elaborate case study taken from an industrial setting. • A process algebraic approach to test generation from use cases in natural language. • Integration of coverage criteria into a purely algebraic test generation approach. • Test generation considering coverage of events or transitions of the model. • Complexity analysis and a provably sound optimisation of the test generation approach. • An industrial case study with metrics on the efficiency of the test generation strategy.