Test Sequence Generation Research Articles

Using expression parsing and algebraic operations to generate test sequences.

It has become a popular trend to build software’s regular expression or extended regular expression models in order to generate test sequences from these models. Such test sequences tend to have promising test coverage and fault detection capability. During this process, one critical step is expression parsing based on algebraic operations. However, the parsing can be very challenging as different algebraic systems have different algebraic operators and algebraic operations. Besides, the parsing difficulty continues to grow as software complexity increases. To address the above challenges, this paper proposes a general expression parsing framework for test sequence generation. The proposed framework consists of three stages, expression decomposition, algebraic operations, and subexpression combination. To implement the framework, an expression parsing algorithm based on abstract syntax tree is developed. Case studies based on 117 expressions collected from the literature over the past 30 years as well as 13 software systems are conducted to evaluate the effectiveness of the proposed algorithm. The results indicate that our algorithm is superior to three existing and commonly used algorithms with respect to expression parsing and software fault detection.

Autotesting an Embedded Reconfigurable Computing System

Purpose of research. The main idea is to build a mathematical testing model that integrates different aspects of an embedded reconfigurable computing system and its interactions. This model provides an efficient representation of test scenarios and allows to analyse the dynamics of the reconfigurable computing system during testing. The paper also discusses methods for generating test sequences based on the properties of a finite state machine.Methods. The authors propose to represent the autotest as a finite state Mile machine, where states serve to store information about the current state of an embedded reconfigurable computing system. The input signal is the interaction with the system and the output signal is the system's response to the input signal. This approach allows to formalize the testing process and simplify the analysis of possible problems.Results. This paper discusses the application of automata theory in the context of automated testing of embedded reconfigurable computing systems. Automata theory provides effective methods and tools for analysis and simulation of discrete dynamic systems, which makes it suitable for automated testing tasks.Conclusion. The results show that the use of automata theory can significantly improve the quality and efficiency of automated testing of embedded reconfigurable computing systems. This approach provides a deeper analysis of the system and allows to detect and prevent potential problems that may arise during its operation.

A measure of the difference between test sets for generating controlled random tests

Objectives . The problem of constructing the characteristics of the difference between test sequences is solved. Its relevance for generating controlled random tests and the complexity of finding measures of difference for symbolic tests are substantiated. The limitations of using the Hamming and Damerau – Levenshtein distances to obtain a measure of the difference between test sets are shown.Methods . Based on the characteristic of the interval used in the theory of the chain of successive events, a new measure of the difference between two symbolic test sets is determined. As a difference measure, the distance AD(Ti, Tk) between the test sets Ti and Tk is calculated using the interval characteristic, which is based on determining independent pairs of same (identical) symbols belonging to two sets and calculating the intervals between them.Results. The combinatorial nature of the calculation of the proposed difference measure for symbolic test sets of an arbitrary alphabet and dimension is shown. An example of calculating this measure for various types of test sets, including such as address test sets, is given. Possible modifications are shown and some properties and limitations are determined. The application of the measure of difference is considered for the case of repeated testing of storage devices based on address sequences pA with even p repetition of addresses. For the case p = 2, mathematical relations are given for calculating the intervals and distances AD(Ti, Tk) for address sequences 2A used for controlled random testing of storage devices. The main attention is paid to binary test sets, when the task of calculating given difference metric is reduced to the classical assignment problem using the Hungarian algorithm. The computational complexity of the Hungarian algorithm is estimated by the relation O(n4). As an alternative to the Hungarian algorithm, an algorithm for calculating the considered difference measure is proposed, the complexity of which is much less and has an estimate equal to O(n2). The experimental studies confirm the effectiveness of the proposed algorithm.Conclusion. The proposed difference measure extends the possibilities of generating test sequences when generating controlled random tests. It is shown that test sets, which are indistinguishable when Hamming distance is used as a measure of difference, have different values of AD(Ti, Tk) that allows to make more accurate classification of randomly generated sets as candidates for test sets.

AUTOMATION OF TESTING AND FAULT DETECTION FOR ROCKET ENGINE TEST FACILITIES WITH MACHINE LEARNING

The German Aerospace Center (DLR) Institute of Space Propulsion has unique expertise in operating test facilities for rocket engine testing and development in Europe since 1959. However, essential elements of the test site were designed up to half a century ago. In order to ensure a futureproof and intelligent digital test infrastructure, the potential of test automation, advanced control, and monitoring systems is investigated based on machine learning. Such intelligent control systems are expected to reduce engine development and test preparation times, thereby lowering the associated costs. Additionally, advanced monitoring systems are anticipated to increase the safety and reliability of the test infrastructure. This paper presents the results of two pilot projects: the first project uses reinforcement learning to automatically generate test sequences based on test requirements, while the second project develops a feed-forward forecasting model to predict deviations from expected behavior in the feed-line of a rocket engine test facility.

RATE: A model‐based testing approach that combines model refinement and test execution

AbstractIn this paper, we present an approach to conformance testing based on abstract state machines (ASMs) that combines model refinement and test execution (RATE) and its application to three case studies. The RATE approach consists in generating test sequences from ASMs and checking the conformance between code and models in multiple iterations. The process follows these steps: (1) model the system as an abstract state machine; (2) validate and verify the model; (3) generate test sequences automatically from the ASM model; (4) execute the tests over the implementation and compute the code coverage; (5) if the coverage is below the desired threshold, then refine the abstract state machine model to add the uncovered functionalities and return to step 2. We have applied the proposed approach in three case studies: a traffic light control system (TLCS), the IEEE 11073‐20601 personal health device (PHD) protocol, and the mechanical ventilator Milano (MVM). By applying RATE, at each refinement level, we have increased code coverage and identified some faults or conformance errors for all the case studies. The fault detection capability of RATE has also been confirmed by mutation analysis, in which we have highlighted that, many mutants can be killed even by the most abstract models.

A new approach to event- and model-based feature-driven software testing and comparison with similar approaches

A software can be thought as a composition of features. Feature-oriented software development (FOSD) builds the development process on features. Part of the FOSD process is testing, and accordingly, it should be feature-driven. In model-based testing, test cases are systematically generated using the model. This research concentrates on event-based graphical models and utilizes event sequence graphs (ESGs). We develop a new test sequence generation algorithm for ESGs and named it short and frequent test sequences (SFT). Then we compare it with the existing test sequence generation algorithm called TSD. Moreover, we introduce two model-building approaches, namely daisy and swim lane, for ESGs and analyze their effects on feature-driven testing. For the evaluation, we use five different feature-driven software models. The evaluation results shows that both modeling approaches are advantageous in certain test objectives. For testing the software product as a whole, test sequence(s) should be generated by TSD from daisy modeled ESG. If a certain feature within the software product or its interaction with another feature is to be tested, then test sequence(s) should be generated by SFT from swim lane modeled ESG.

Research on conformance test sequence generation of the communication unit adaptation specification protocol

The communication unit adaptation specification protocol of the customer-side energy measuring equipment and its conformance test is important for realizing massive equipment access to the energy internet flexibly and efficiently. The conformance test of communication protocol is the key to generating complete and efficient test sequences. In allusion to the problem of conformance test sequence generation of communication unit adaptation specification protocol, the paper first formally describes the protocol based on a finite state machine, and then further carries out the research on conformance test sequence generation based on T method, U method and U+ method. The results show that the test sequence generated by U+ method has the ability of state verification and the length is 22.1% shorter than that of U method. Its redundancy is slightly less than that of T method and the error detection ability is much better. Therefore, U+ method is more suitable for the conformance test sequence generation of communication unit adaptation specification protocol.

Model-Based Test Sequence Generation and Prioritization Using Ant Colony Optimization

The paper presents an approach to generate and optimize test sequences from the input UML activity diagram. For this, an algorithm is proposed called Unified Modelling Language for Test Sequence Generation (UMLTSG) that uses a search-based algorithm, named Test Sequence Prioritization using Ant Colony Optimization (TSP ACO) to generate and optimize test sequences. The algorithms overcome the existing limitations of handling complex decision-making activity such as conditional activity, fork activity, and join the activity. The optimization process helps to reduce the number of processing nodes that leads to minimizing the time and cost. The proposed approach experiments on a well-known application Railway Ticket Reservation System (RTRS). APFD metric measures the effectiveness of our approach and found that the prioritized order of test sequences achieved 20% higher APFD score. Apart from this, the authors have also experimented on six real life case studies and obtained an average of 52.16% reduction in redundant test paths.

A general enhancement method for test strategy generation for the sequential fault diagnosis of complex systems

In order to improve the reliability, operational readiness and system safety of equipment, testability should be seriously considered in the design stage. As an important part of design for testability, test sequence generation is a binary identification problem because a minimal expected cost testing procedure must be developed in order to determine the amount of possible failure sources, if any, are present. Many algorithms have been proposed, but the generation time is long or the test cost is high when dealing with a large-scale dependency matrix. To address this issue, we propose a general enhancement method based on the SVM, the ECA* and the Monte Carlo. It can be applied to any existing algorithm and can effectively improve the performance. The available tests are classed based on the SVM according to the information of nodes, the ECA* is used to cluster states, and the morphological function of the test sequence is obtained through the Monte Carlo simulation. All this information is fused to dynamically adjust the scale of the dependency matrix and selected to modify the parameters. Experiments show that the existing algorithms have shorter calculation time and lower costs because the information is considered more comprehensively after enhancement.

MuFBDTester: A mutation‐based test sequence generator for FBD programs implementing nuclear power plant software

SummaryFunction block diagram (FBD) is a standard programming language for programmable logic controllers (PLCs). PLCs have been widely used to develop safety‐critical systems such as nuclear reactor protection systems. It is crucial to test FBD programs for such systems effectively. This paper presents an automated test sequence generation approach using mutation testing techniques for FBD programs and the developed tool, MuFBDTester. Given an FBD program, MuFBDTester analyses the program and generates mutated programs based on mutation operators. MuFBDTester translates the given program and mutants into the input language of a satisfiability modulo theories (SMT) solver to derive a set of test sequences. The primary objective is to find the test data that can distinguish between the results of the given program and mutants. We conducted experiments with several examples including real industrial cases to evaluate the effectiveness and efficiency of our approach. With the control of test size, the results indicated that the mutation‐based test suites were statistically more effective at revealing artificial faults than structural coverage‐based test suites. Furthermore, the mutation‐based test suites detected more reproduced faults, found in industrial programs, than structural coverage‐based test suites. Compared to structural coverage‐based test generation time, the time required by MuFBDTester to generate one test sequence from industrial programs is approximately 1.3 times longer; however, it is considered to be worth paying the price for high effectiveness. Using MuFBDTester, the manual effort of creating test suites was significantly reduced from days to minutes due to automated test generation. MuFBDTester can provide highly effective test suites for FBD engineers.

Multiobjective Genetic Algorithm for Class Testing using OCL Class Contract Specifications: A Framework

It has been a software trend to build large-scale complex systems with high reliability. Due to the size of the software and the dynamic requirements of the stakeholders, it becomes hard to test those software systems manually. This may lead the software to fatal failures and cause irrecoverable catastrophic damage. To be safe, the software system must be investigated thoroughly before it is too late. Test sequence generation for Unified Modeling Language (UML) class models from their semiformal Object Constraint Language specifications can be helpful in identifying the defects in the early phase of the software life cycle. The existing approaches suffer from inherent problems of exhaustive exploration of finite state machines (infeasible paths, exponential number of test sequences, and uncertainty of completion of testing). Evolutionary algorithms can greatly help by optimizing the test sequences to get optimal coverage, minimal cost, and higher quality. The proposed approach helps us to improve the testing of Unified Modeling Language (UML) model-based software, by testing the conformance to semiformal class operation contract specifications (specified in the form of Object Management Group (OMG) standard and Object Constraint Language (OCL) semiformal language). The presented research achieved two main goals: (1) automation of testing process and conformance to standards of the current technique of test sequence generation, bridging the gap between the research and industry; (2) improvement in the state of the art approach through the application of multiobjective genetic algorithms (MOGAs). A case study along with the results achieved through the proposed technique is presented as well, clearly reflecting the significance of the proposed research.

Research on automatic test sequence generation method of computer interlocking test

Railway computer interlocking system is the core equipment of station signal control system, which directly affects the reliability and safety of the whole system. It is essential to test the interlocking software comprehensively and strictly [1]. The traditional interlocking test is conducted by testers to control the quality and efficiency of the test. According to the reference railway transportation “Interim Measures for railway signal interlocking test”, combined with the signal interlocking relationship test checklist, compared with the actual station yard of railway station, relying on manual preparation and manual implementation of test items, the test efficiency is relatively low. Therefore, the realization of automatic interlocking test has become a research hotspot at present. Interlocking function test belongs to the research field of black box test. On the premise of ensuring the comprehensiveness of test, generating efficient test sequence is the research focus of this subject.

Generating and Employing Witness Automata for ACTLW Formulae

When verifying the validity of a formula in a system model by a model checker, a common feature is the generation of a linear witness or counterexample, which is a computation path usually showing a single reason why the formula is valid or, respectively, not. For systems represented with Labeled Transition Systems (LTS) and a subset of ACTLW (Action-based Computation Tree Logic with Unless operator) formulae, a procedure exists for the generation of witness automata, which contain all the interesting finite linear witnesses, thus revealing all the reasons of the validity of a formula. Although this procedure uses a symbolic representation of LTSs, transitions of a given LTS are traversed one by one. In this paper, we propose a procedure which exploits the symbolic representation efficiently to traverse several transitions at once. We evaluate the procedure on models of a communication protocol from industry and a biological system. The results show it to be at least several times faster than the former one. Witness automata were first introduced to allow for compositional generation of test sequences. We propose two more possible uses. One is for the detection of multiple errors in a model by exploring the witness automaton for a formula, instead of only one, which is usually the case with a single witness. The other one is for the detection of previously unknown system properties. As witness automata can be rather large, we show how some existing tools could help in examining them through visualization and simulation.

Memory-Efficient LFSR Encoding and Weightage Driven Bit Transition for Improved Fault Coverage

Linear Feedback Shift Registers (LFSRs) have been employed as test pattern generators in BIST for decades; however, an emerging problem with design constraints leads to a lot of improvements in this field. This paper presents a memory-efficient encoding-based method to generate test patterns for a given primitive polynomial LFSR TPG. Here, test patterns generated from LFSR are divided into groups and follow encoding to transform into multiple test patterns. These newly generated encoded test patterns are further divided into transitional and non-transitional blocks which control the bit transitions over encoded values. This weighted driven bit transition also prevents certain bit transitions that reduce the dynamic power as well during the testing process. This TPG technique can be used for generating both pseudo-random test sequence and deterministic pattern generation by allowing fine control over bit transition and appropriate encoder design with the least hardware complexity overhead. The proposed technique has experimented over ISCAS ‘85 and some sequential part of ISCAS ‘89 benchmark circuits to validate the superiority in terms of memory efficiency compared to some well-known LFSR reseeding techniques and the hardware complexity reduction during BIST implementation. The proposed encoded test pattern also achieves a minimum of 14% test data volume and toggle control scheme in power reduction.

Optimization of Favourable Test Path Sequences Using Bio-Inspired Natural River System Algorithm

Testing of software requires a great amount of time and effort. The tester's main aim is to design optimized test sequences with a minimum amount of time, effort, and with less redundancy. Testers have used artificial intelligence meta-heuristic algorithms for optimization of test sequences. The model-driven approach is helpful in the generation of test sequences at early designing phase only. The model-driven approach uses UML diagram to represent the system's behavior and design test cases for the system at design stage of software development life cycle. The proposed approach uses natural river system for optimizing favourable non-redundant test path sequences using UML activity diagrams and sequence diagrams. The implementation of proposed approach has been done using python and results show that the proposed approach provides full coverage of test paths with less redundant test nodes compared to other meta heuristic algorithms.

Generating feasible protocol test sequences from EFSM models using Monte Carlo tree search

Feasible test sequences generation is a key step in protocol conformance testing based on the Extended Finite State Machine (EFSM) model. To guarantee the feasibility of generated test sequences, transition executability analysis (TEA) technique is widely applied in automatic test derivation. However, the TEA method often suffers from the famous state explosion problem, which has become a major obstacle to its efficient application.

Test Compaction by Backward and Forward Extension of Multicycle Tests

Multicycle tests are useful for test compaction even when full scan allows single- or two-cycle tests to be used. To avoid sequential test generation, multicycle tests can use the test data (scan-in states and primary input vectors) from a single- or two-cycle test set, possibly modified to make it more suitable for multicycle tests. However, the modification process requires repeated fault simulation to prevent a loss of fault coverage. This brief observes that the need to modify test data results from the fact that tests are only extended forward to increase their numbers of clock cycles starting from their (modified) scan-in states. The brief observes further that extending tests backward is a more computationally efficient way of obtaining multicycle tests with modified test data. The brief defines a new type of multicycle test to increase the effectiveness of backward extension and describes a test compaction procedure based on backward and forward extension. It presents experimental results for gate-exhaustive faults, requiring large numbers of tests, in benchmark circuits.

Runtime testing of context-aware variability in adaptive systems

Abstract Context: A Dynamically Adaptive System (DAS) supports runtime adaptations to handle changes in the operational environment. These adaptations can change the system’s structure or behavior and even the logic of its adaptation mechanism. However, these adaptations may insert defects, leading the system to fail at runtime. Objective: Aiming to identify these failures, testing can be executed to verify the system at runtime. Studies in the literature mostly focus on testing to verify the adaptations at design-time or functionalities at runtime, rather than exercising the adaptation mechanism at runtime. So, we propose RETAkE (RuntimE Testing of dynamically Adaptive systEms). Method: RETAkE is an approach to perform the runtime testing based on the system’s context variability and feature modeling. RETAkE tests the adaptation mechanism, enabling the verification of its adaptation rules with the system’s variability model. The runtime testing is supported by the verification of behavioral properties. For the evaluation, we used the mutation testing technique with two DAS. We also conducted an evaluation to measure the overhead introduced when RETAkE is integrated to the DAS. Results: RETAkE identified the mutants in the two mobile DAS, but the results vary due to the probabilistic nature of the approach to generate test sequences. Regarding the overhead, test sequences of size 30 had a low impact. However, bigger test sequences increase the overhead. Conclusion: The integration of RETAkE to the DAS adaptation mechanism can support the discovery of adaptation failures that occur at runtime. Furthermore, the results of the evaluation suggest its feasibility to perform runtime testing.

Automated Test Case Generation for Digital System Designs: A Mapping Study on VHDL, Verilog, and SystemVerilog Description Languages

Researchers have proposed different methods for testing digital systems and circuits in the last couple of decades. The need for testing digital logic circuits has become more important than ever due to the growing complexity of such systems. During the design phase, testing is focusing on design defects, as well as manufacturing and wear out type of defects. Failures in digital systems could be caused, for example, by design errors, the use of inherently probabilistic devices, and manufacturing variability. As a way to test digital systems in a more efficient way, automated test generation has been proposed to automatically create tests that can quickly and accurately identify faulty components. Examples of such techniques are the sequential test generation, the scan path testing, and the random test generation techniques. With the research domain becoming more mature and growing, it is essential to systematically identify, analyze, and classify these contributions. We performed a systematic mapping study of automated test generation for digital circuits aimed at providing an overview of the application of these techniques. We focused on three of the most widely-used and well-supported hardware description languages (HDLs) for digital systems: Verilog, SystemVerilog, and VHDL. Our results suggest that the majority of the test generation methods for digital circuits are focused on the behavioral and register-transfer design levels. Fault-independent and fault-oriented test generation are the most frequently reported types of test generation methods, while HDL model simulation is the most common test generation technology used to search for test cases in these academic studies. While the results are suggesting a growing interest in this area, the majority of articles are published as conferences papers. Our results show that only 31% of the methods are implemented as software tools and only 63% of all contributions are actually generating executable test cases. This study makes three important contributions, (i) a state-of-the-art of test generation for digital system designs research is provided, (ii) the reported characteristics are identified in both the primary papers and experimental reports, (iii) gaps and opportunities for future test generation for digital system designs research are identified.

Broad-Brush Compaction for Sequential Test Generation

Test sequences that do not use scan chains have several advantages as manufacturing tests. Test sequences can be produced by sequential test generation procedures, and static test compaction can be applied to reduce the length of the sequence without losing fault coverage. A typical sequential test generation procedure that produces a single test sequence concatenates test subsequences to form the test sequence. Existing static test compaction procedures ignore the construction of the sequence from subsequences and consider every test vector individually for omission. This brief introduces a broad-brush static test compaction procedure that considers entire subsequences for omission to achieve test compaction. As additional subsequences are concatenated, subsequences concatenated earlier may become unnecessary. The procedure described in this brief identifies this situation. The advantage of a broad-brush approach is a low computational effort, which is demonstrated by experimental results for benchmark circuits.