Test Generation Algorithm Research Articles

Personalization of User Experience in Language Learning through Large Language Model Assistants

Language learning benefits from a comprehensive approach, but traditional software often lacks personalization. This study analyzes prompt engineering principles to implement a test generation algorithm using Large Language Models (LLMs). The approach involved examining these principles, exploring related strategies, and creating a unified prompt structure. A test generation script was developed and integrated into an API for an interactive language learning platform. While LLM integration offers highly effective, personalized learning experiences, issues like response time and content diversity need addressing. Future advancements in LLM technology are expected to resolve these limitations.

Test generation algorithm for QCA circuits targeting novel defects and its corresponding fault models

Considering the scaling limitations of current Complementary Metal Oxide Semiconductor (CMOS) technology, Quantum-dot-Cellular Automata (QCA) is emerging as one of the alternatives. QCA being at the molecular scale, defects are more likely to occur in it. Therefore, substantial development of QCA-oriented defects, its corresponding fault models and test generation is required. In this paper, a test generation algorithm for a QCA combinational circuit is proposed. The FAN (A Fanout Oriented) test generation algorithm is extended for QCA. The proposed Automatic Test Pattern Generator (ATPG) for QCA targets Single Stuck at Fault (SSF) set produced by novel Multiple Missing Cells (MMC) defects. The proposed ATPG is based on the QCA-oriented test generation properties and guided by proposed testability measures.The MCNC benchmark circuits are synthesized into QCA using proposed synthesis algorithms to check the effectiveness of the proposed ATPG. The ATPG is developed using C++ and tested on MCNC benchmark circuits. Further, ATPG-generated test vectors are validated at the QCA device level to demonstrate their correctness. The QCADesigner-E tool is used for the device-level implementation of the MCNC benchmark circuit.

Part-X: A Family of Stochastic Algorithms for Search-Based Test Generation With Probabilistic Guarantees

Part-X: A Family of Stochastic Algorithms for Search-Based Test Generation With Probabilistic Guarantees

Semantic matching in GUI test reuse

Reusing test cases across apps that share similar functionalities reduces both the effort required to produce useful test cases and the time to offer reliable apps to the market. The main approaches to reuse test cases across apps combine different semantic matching and test generation algorithms to migrate test cases across Android apps. In this paper we define a general framework to evaluate the impact and effectiveness of different choices of semantic matching with Test Reuse approaches on migrating test cases across Android apps. We offer a thorough comparative evaluation of the many possible choices for the components of test migration processes. We propose an approach that combines the most effective choices for each component of the test migration process to obtain an effective approach. We report the results of an experimental evaluation on 8,099 GUI events from 337 test configurations. The results attest the prominent impact of semantic matching on test reuse. They indicate that sentence level perform better than word level embedding techniques. They surprisingly suggest a negligible impact of the corpus of documents used for building the word embedding model for the Semantic Matching Algorithm. They provide evidence that semantic matching of events of selected types perform better than semantic matching of events of all types. They show that the effectiveness of overall Test Reuse approach depends on the characteristics of the test suites and apps. The replication package that we make publicly available online (https://star.inf.usi.ch/#/software-data/11) allows researchers and practitioners to refine the results with additional experiments and evaluate other choices for test reuse components.

An empirical study of automated unit test generation for Python

Various mature automated test generation tools exist for statically typed programming languages such as Java. Automatically generating unit tests for dynamically typed programming languages such as Python, however, is substantially more difficult due to the dynamic nature of these languages as well as the lack of type information. Our Pynguin framework provides automated unit test generation for Python. In this paper, we extend our previous work on Pynguin to support more aspects of the Python language, and by studying a larger variety of well-established state of the art test-generation algorithms, namely DynaMOSA, MIO, and MOSA. Furthermore, we improved our Pynguin tool to generate regression assertions, whose quality we also evaluate. Our experiments confirm that evolutionary algorithms can outperform random test generation also in the context of Python, and similar to the Java world, DynaMOSA yields the highest coverage results. However, our results also demonstrate that there are still fundamental remaining issues, such as inferring type information for code without this information, currently limiting the effectiveness of test generation for Python.

Effective grey‐box testing with partial FSM models

SummaryFor partial, nondeterministic, finite state machines, a new conformance relation called strong reduction is presented. It complements other existing conformance relations in the sense that the new relation is well suited for model‐based testing of systems whose inputs are enabled or disabled, depending on the actual system state. Examples of such systems are graphical user interfaces and systems with interfaces that can be enabled or disabled in a mechanical way. We present a new test generation algorithm producing complete test suites for strong reduction. The suites are executed according to the grey‐box testing paradigm: it is assumed that the state‐dependent sets of enabled inputs can be identified during test execution, while the implementation states remain hidden, as in black‐box testing. We show that this grey‐box information is exploited by the generation algorithm in such a way that the resulting best‐case test suite size is only linear in the state space size of the reference model. Moreover, examples show that this may lead to significant reductions of test suite size in comparison to true black‐box testing for strong reduction.

Low Power and High Speed Sequential Circuits Test Architecture

Background: Electronic circuits testing and verification are performed to determine faulty devices after IC fabrication and to ensure that the circuit performs its designed functions. The verification process is considered as a test for both sequential and combinational logic circuits. The sequential circuits test is a more complex task than the combinational circuits test. However, dedicated algorithms can be used to test any type of sequential circuit regardless of its complexity. Objective: This paper presented a new design under test (DUT) algorithm for 4-and 8-tap FIR filters. Also, the FIR filter and the proposed DUT algorithm is implemented using field programmable gate arrays (FPGA). Methods: The proposed DUT test generation algorithm is implemented using VHDL and Xilinx ISE V14.5 design suite. The proposed test generation algorithm for the FIR filter utilizes filtering redundant faults to obtain a set of target faults for the DUT. Results: The proposed algorithm reduces time delay for up to 50 % with power consumption reduction of up to 70 % in comparison with the most recent similar work. Conclusions: The implementation results ensured that a high speed and low power consumption architecture can be achieved. Also, the proposed architecture performance is faster than that of the existing techniques.

Minimizing Characterizing sets

• Provides an algorithm that generates compact Characterizing sets from deterministic FSMs . • Shows that constructing minimum sized Characterising sets is PSPACE complete for deterministic FSMs . • Shows that constructing Characterising sets with minimum total number of inputs is NP Complete for deterministic FSMs . • Presents the results of a set of experiment and discussions in a rigorous way. A characterizing set (CS) for a deterministic finite state machine (FSM) M is a set of input sequences that, between them, separate (distinguish) all of the states of M . CSs are used within several test generation techniques that return test suites with guaranteed fault detection power. The number of input sequences in a CS directly affects the cost of applying the resultant test suite. In this paper, we study the complexity of decision problems associated with deriving a smallest CS from an FSM, showing that checking the existence of a CS with K sequences is PSPACE-complete . We also consider the length of a CS, which is the sum of the lengths of the input sequences in the CS. It transpires that the problem of deciding whether there is a CS with length at most K is NP-complete . Motivated by these results, we introduce a heuristic to construct a CS, from a deterministic FSM, with the aim of minimizing the number of input sequences. We evaluated the proposed algorithm by assessing its effect when used within a classical test generation algorithm (the W-method). In the evaluation, we used both randomly generated FSMs and benchmark FSMs. The results are promising, with the proposed algorithm reducing the number of test sequences by 37.3% and decreasing the total length of the test suites by 34.6% on average.

State identification for labeled transition systems with inputs and outputs

For Finite State Machines (FSMs) a rich testing theory has been developed to discover aspects of their behavior and ensure their correct functioning. Although this theory has been frequently used, e.g. to check conformance of protocol implementations, its applicability is limited by restrictions of FSMs, in which inputs and outputs alternate, and outputs are determined by the previous input and state. Labeled Transition Systems with inputs and outputs (LTSs), as studied in ioco testing theory, provide a richer framework for testing component oriented systems, but lack the algorithms for test generation from FSM theory. In this article, we propose an algorithm for the fundamental problem of state identification during testing of LTSs. Our algorithm is a direct generalization of the well-known algorithm for computing adaptive distinguishing sequences for FSMs proposed by Lee and Yannakakis. Our algorithm has to deal with so-called compatible states, states that cannot be distinguished. Analogous to the result of Lee and Yannakakis, we prove that if an adaptive test exists that distinguishes all pairs of (incompatible) states of an LTS, our algorithm will find one. In practice, such perfect adaptive tests typically do not exist. However, in experiments with an implementation of our algorithm on a collection of (both academic and industrial) benchmarks, we find that that the adaptive tests produced by our algorithm still distinguish at least 99% of the incompatible state pairs.

MaxSense

Detection of hardware Trojans is vital to ensure the security and trustworthiness of System-on-Chip (SoC) designs. Side-channel analysis is effective for Trojan detection by analyzing various side-channel signatures such as power, current, and delay. In this article, we propose an efficient test generation technique to facilitate side-channel analysis utilizing dynamic current. While early work on current-aware test generation has proposed several promising ideas, there are two major challenges in applying it on large designs: (i) The test generation time grows exponentially with the design complexity, and (ii) it is infeasible to detect Trojans, since the side-channel sensitivity is marginal compared to the noise and process variations. Our proposed work addresses both challenges by effectively exploiting the affinity between the inputs and rare (suspicious) nodes. The basic idea is to quickly find the profitable ordered pairs of test vectors that can maximize side-channel sensitivity. This article makes two important contributions: (i) It proposed an efficient test generation algorithm that can produce the first patterns in the test vectors to maximize activation of suspicious nodes using an SMT solver, and (ii) it developed a genetic-algorithm based test generation technique to produce the second patterns in the test vectors to maximize the switching in the suspicious regions while minimizing the switching in the rest of the design. Our experimental results demonstrate that we can drastically improve both the side-channel sensitivity (62× on average) and time complexity (13× on average) compared to the state-of-the-art test generation techniques.

Test generation algorithm for the All-Transition-State criteria of Finite State Machines

In the current article a novel test generation algorithm is presented for deterministic finite state machine specifications based on the recently introduced All-Transition-State criteria. The size of the resulting test suite and the time required for test suite generation are investigated through analytical and practical analyses and are also compared to the Transition Tour, Harmonized State Identifiers and random walk test generation methods. The fault detection capabilities of the different approaches are also investigated with simulations applying randomly injected transfer faults.

Scalable Activation of Rare Triggers in Hardware Trojans by Repeated Maximal Clique Sampling

Hardware Trojans are serious threat to security and reliability of computing systems. It is hard to detect these malicious implants using traditional validation methods since an adversary is likely to hide them under rare trigger conditions. While existing statistical test generation methods are promising for Trojan detection, they are not suitable for activating extremely rare trigger conditions in stealthy Trojans. To address the fundamental challenge of activating rare triggers, we propose a new test generation paradigm for trigger activation by repeated maximal clique sampling (TARMAC). The basic idea is to utilize a satisfiability modulo theories (SMTs) solver to construct a test corresponding to each maximal clique. This article makes three fundamental contributions: 1) it proves that the trigger activation problem can be mapped to clique cover problem, and the test vectors generated by covering maximal cliques are complete and compact; 2) it proposes efficient test generation algorithms to activate trigger conditions by repeated maximal clique sampling; and 3) it outlines an efficient mechanism to run the clique sampling in parallel to significantly improve the scalability of our test generation framework. The experimental results demonstrate that our proposed approach is scalable and it outperforms state-of-the-art approaches by several orders-of-magnitude in detecting stealthy Trojans.

Using memetic algorithm for robustness testing of contract-based software models

Graph Transformation System (GTS) can formally specify the behavioral aspects of complex systems through graph-based contracts. Test suite generation under normal conditions from GTS specifications is a task well-suited to evolutionary algorithms such as Genetic and Particle Swarm Optimization (PSO) metaheuristics. However, testing the vulnerabilities of a system under unexpected events such as invalid inputs is essential. Furthermore, the mentioned global search algorithms tend to make big jumps in the system’s state-space that are not concentrated on particular test goals. In this paper, we extend the HGAPSO approach into a cost-aware Memetic Algorithm (MA) by making small local changes through a proposed local search operator to optimize coverage score and testing costs. Moreover, we test GTS specifications not only under normal events but also under unexpected situations. So, three coverage-based testing strategies are investigated, including normal testing, robustness testing, and a hybrid strategy. The effectiveness of the proposed test generation algorithm and the testing strategies are evaluated through a type of mutation analysis at the model-level. Our experimental results show that (1) the hybrid testing strategy outperforms normal and robustness testing strategies in terms of fault-detection capability, (2) the robustness testing is the most cost-efficient strategy, and (3) the proposed MA with the hybrid testing strategy outperforms the state-of-the-art global search algorithms.

Higher-order demand-driven symbolic evaluation

Symbolic backwards execution (SBE) is a useful variation on standard forward symbolic evaluation; it allows a symbolic evaluation to start anywhere in the program and proceed by executing in reverse to the program start. SBE brings goal-directed reasoning to symbolic evaluation and has proven effective in e.g. automated test generation for imperative languages. In this paper we define DDSE, a novel SBE which operates on a functional as opposed to imperative language; furthermore, it is defined as a natural extension of a backwards-executing interpreter. We establish the soundness of DDSE and define a test generation algorithm for this toy language. We report on an initial reference implementation to confirm the correctness of the principles.

CTFTP: A Test Case Generation Strategy for General Boolean Expressions Based on Ordered Binary Label-Driven Petri Nets

Boolean expression testing requires certain types of tests for each Boolean expression in program specification or implementation. Fault-based testing essentially uses a subset of the exhaustive test set to detect certain special types of faults. A fault-based Boolean expression testing strategy called constraint true and false test point (CTFTP) is proposed. The test consists of two test case generation strategies, namely a unique constraint true point (UCTP) strategy and a near constraint false point (NCFP) strategy. An ordered binary label-driven Petri net model is presented to analyze the interaction between Boolean transitions and Boolean literals and yield the test paths of a singular term for the irredundant disjunctive normal forms (IDNFs). On the basis of the test paths, we develop a configuration-based IDNF test generation algorithm, which is employed to obtain the UCTP, NCFP, and CTFTP test sets for the IDNFs. The proposed test generation algorithm based on literal substitution is applied to extend the CTFTP strategy and generate a test suite for general Boolean expressions, which are evaluated using TCAS II specifications. Experimental results show that the CTFTP strategy can detect the same seven types of faults similar to the MUMCUT strategy when testing IDNFs, but only a subset of the MUMCUT test set is required. Five types of faults of general Boolean expressions can also be detected using CTFTP strategies.

Comparing Graph-Based Algorithms to Generate Test Cases from Finite State Machines

Model-Based Testing (MBT) is a well-known technique that employs formal models to represent reactive systems’ behavior and generates test cases. Such systems have been specified and verified using mostly Finite State Machines (FSMs). There is a plethora of test generation algorithms in the literature; most of them are based on graphs once an FSM can be formally defined as a graph. Nevertheless, there is a lack of studies on analyzing cost and efficiency of FSM-based test generation algorithms. This study compares graph-based algorithms adopted to generate test cases from FSM models. In particular, we compare the Chinese Postman Problem (CPP) and H-Switch Cover (HSC) algorithms with the well-known Depth-First Search (DFS) and Breadth-First Search (BFS) algorithms in the context of covering all-transitions and all-transition-pairs criteria in an FSM. First, a systematic literature mapping was conducted to summarize the methods that have been adopted in MBT, considering FSMs. Second, the main methods found were implemented and analyzed on randomly-generated FSMs, as well as real-world models that represent embedded systems of space applications. To make comparisons, we considered analyses in terms of cost (time), efficiency (mutant analysis) and characteristics of the generated test suites (number of test cases, average length of test cases, largest and smallest test cases, standard deviation and distribution of test cases). In general, CPP presented the best results in terms of number of test cases and test suite size. In addition, CPP also presented low distribution of average length compared to other algorithms.

Test Pattern Generator for MV-Based QCA Combinational Circuit Targeting MMC Fault Models

Quantum-dot-Cellular Automata (QCA) is emerging as one of the alternatives for Integrated Circuit Technology considering the scaling limitations of current Complementary Metal Oxide Semiconductor (CMOS) technology. Being at molecular scale, defects are more likely to occur in QCA devices. Therefore, the substantial development of QCA-oriented defects, its corresponding fault models and test generation is required. This paper addresses the fault model caused by multiple missing cells defect and test generation for these faults for QCA circuits. We have shown that the single missing cell considerations are not enough. Correspondingly, in this paper, testing properties for the detection of fault caused by multiple missing cell defects in QCA devices mainly Majority Voter (MV) are proposed. Even though there are many advanced ATPGs available for CMOS technology, to demonstrate very special MV-based properties with reference to the test pattern generation process and to open the path for advanced combinational and sequential ATPGs specific to MV-based QCA circuits, the extension of basic Automatic Test Pattern Generator is proposed here. The proposed test generation algorithm is guided by the extended Sandia Controllability Observability Analysis Program (SCOAP) testability measures especially for QCA logic primitives.

Design, implementation, and application of GPU-based Java bytecode interpreters

We present the design and implementation of GVM, the first system for executing Java bytecode entirely on GPUs. GVM is ideal for applications that execute a large number of short-living tasks, which share a significant fraction of their codebase and have similar execution time. GVM uses novel algorithms, scheduling, and data layout techniques to adapt to the massively parallel programming and execution model of GPUs. We apply GVM to generate and execute tests for Java projects. First, we implement a sequence-based test generation on top of GVM and design novel algorithms to avoid redundant test sequences. Second, we use GVM to execute randomly generated test cases. We evaluate GVM by comparing it with two existing Java bytecode interpreters (Oracle JVM and Java Pathfinder), as well as with the Oracle JVM with just-in-time (JIT) compiler, which has been engineered and optimized for over twenty years. Our evaluation shows that sequence-based test generation on GVM outperforms both Java Pathfinder and Oracle JVM interpreter. Additionally, our results show that GVM performs as well as running our parallel sequence-based test generation algorithm using JVM with JIT with many CPU threads. Furthermore, our evaluation on several classes from open-source projects shows that executing randomly generated tests on GVM outperforms sequential execution on JVM interpreter and JVM with JIT.

Test Generation and Test Prioritization for Simulink Models with Dynamic Behavior

All engineering disciplines are founded and rely on models, although they may differ on purposes and usages of modeling. Among the different disciplines, the engineering of Cyber Physical Systems (CPSs) particularly relies on models with dynamic behaviors (i.e., models that exhibit time-varying changes). The Simulink modeling platform greatly appeals to CPS engineers since it captures dynamic behavior models. It further provides seamless support for two indispensable engineering activities: (1) automated verification of abstract system models via model simulation, and (2) automated generation of system implementation via code generation. We identify three main challenges in the verification and testing of Simulink models with dynamic behavior, namely incompatibility, oracle and scalability challenges. We propose a Simulink testing approach that attempts to address these challenges. Specifically, we propose a black-box test generation approach, implemented based on meta-heuristic search, that aims to maximize diversity in test output signals generated by Simulink models. We argue that in the CPS domain test oracles are likely to be manual and therefore the main cost driver of testing. In order to lower the cost of manual test oracles, we propose a test prioritization algorithm to automatically rank test cases generated by our test generation algorithm according to their likelihood to reveal a fault. Engineers can then select, according to their test budget, a subset of the most highly ranked test cases. To demonstrate scalability, we evaluate our testing approach using industrial Simulink models. Our evaluation shows that our test generation and test prioritization approaches outperform baseline techniques that rely on random testing and structural coverage.

Enhancement of Adaptive Software-Based Self Test Generation of Embedded Processors Cores

The paper deals with automatic software-based test generation for processors as basic blocks of current complex systems on chip and embedded systems. Testing processors needs continually new test generation methods, algorithms and test application techniques for their verification, manufacture and infield testing and reliable life-time run. The functional tests of processors are mainly generated over an instruction set architecture and processor model description. Such types of tests are serving as additional tests to structural testing or as tests used in verification. They run in similar form and frequency as real programmes. Such tests are categorized as the software-based self tests. A metric for quality evaluation of the software-based tests is obviously provided by code coverage of a processor model. A functional test generation method is based on VHDL model of processors and genetic algorithms with using various evolutionary strategies. The contribution to the SBST methods based on GAs using the latest defined ES was identified. Functionality and effectiveness of the developed methods were evaluated in the implemented system AGenMIX with enhanced function of genetic algorithms computation over three types of RISC processor.