Implementation Under Test Research Articles

An Adaptive Approach to Optimize Probabilistic Distributed Testing

Typically, conformance testing consists of placing a set of parallel testers at each port of an implementation to ensure its conformance to the specifification. However, a number of common fault detections occur if no coordination is made between these parallel testers and the implementation under test (IUT). Therefore, the test process must support mechanisms of coordination between these distributed components, particularly for implementation with stochastic behaviour. To this end, as well as to analyse the stochastic behaviour of the implementation under test, we propose in this paper an algorithm to generate for each tester a probabilistic local test sequence (PLTS) aiming to avoid both synchronization and observation issues. Finally, we suggest a new architecture based on Markov decision processes with an adaptive controller to control and optimize the whole testing process.

EVEREST: An Automatic Model-Based Testing Tool for Asynchronous Reactive Systems: An IOLTS Model-Based Testing Tool

Reactive systems are characterized by the interaction with the environment, where the exchange of the input and output stimuli, usually, occurs asynchronously. Systems of this nature, in general, require a rigorous testing activity over their developing process. Therefore model-based testing has been successfully applied over asynchronous reactive systems using Input Output Labeled Transition Systems (IOLTSs) as the basis. In this work we present a reactive testing tool to check conformance, generate test suites and run test cases using IOLTS models. Our tool can check whether the behavior of an implementation under test (IUT) complies with the behavior of its respective specification. We have implemented a classical conformance relation ioco and a more general notion of conformance based on regular languages. Further, the tool provides a test suite generation in a black-box testing setting for finding faults over IUTs according to a specific domain. We have also described some case studies to probe the tool's functionalities and also to highlight a comparative analysis on both conformance approaches. Finally, we offer experiments to evaluate the performance of our tool using several scenarios.

A temporal based approach for MapReduce distributed testing

Over the last few years, there has been a rising trend towards the field of distributed testing where the implementation under test (IUT) has physical distributed ports. However, running several testers in such ports can lead to a number of coordination and synchronisation issues between these remote testers. We suggest in previous works to cope with these problems by enhancing the test architecture with some timing constraints. In this paper, we introduce a new algorithm to generate local timed test sequences so as to describe the behaviour of the test at each port. We aim through this algorithm to reduce the number of messages exchanged among the testers. Moreover, we illustrate our approach through a case study using MapReduce architecture with several worker components. These systems commonly encounter failures as a result of several circumstances, such as hardware faults, power outages, or network connection delays. To this end, we set the temporal properties in the specification of these systems. Then, we apply our algorithm to derive the related timed local test sequences. Furthermore, we explain how to apply our approach in the case of MapReduce testing to identify faulty workers, in order to reschedule their tasks to a healthy worker.

Test Case Generation Method for Increasing Software Reliability in Safety-Critical Embedded Systems

Finite-state machines (FSMs) and the W method have been widely used in software testing. However, the W method fails to detect post-processing errors in the implementation under test (IUT) because it ends testing when it encounters a previously visited state. To alleviate this issue, we propose an enhanced fault-detection W method. The proposed method does not stop the test, even if it has reached a previously visited state; it continues to test and check the points that the W method misses. Through various case studies, we demonstrated software testing using the W method and the proposed method. From the results, it can be inferred that the proposed method can more explicitly determine the consistency between design and implementation, and it is a better option for testing larger software. Unfortunately, the testing time of the proposed method is approximately 1.4 times longer than that of the W method because of the added paths. However, our method is more appropriate than the W method for software testing in safety-critical systems, even if this method is time consuming. This is because the error-free characteristics of a safety-critical system are more important than anything else. As a result, our method can be used to increase software reliability in safety-critical embedded systems.

Non-Deterministic Delay Behavior Testing of Chinese Train Control System Using UPPAAL-TRON

The Chinese Train Control System level 3 (CTCS-3) is an open and real-time safety-critical system. Due to the non-deterministic delay behavior in the environment, there is a large number of stimuli to the train control system with different patterns of arrival times. Control strategies must be well considered, as either the logic of the function fails or the real-time constraints are dissatisfied, may directly lead to significant human injury or financial loss. In this paper, we introduce the notion of test specification and the relativized timed input/output conformance based on timed automata theory. A new test case generation and execution algorithm has been proposed, by which the tester can reset Implementation Under Test (IUT) whenever they want, make it more sense in specific domain fields. We apply UPPAAL-TRON based online conformance testing framework to test nondeterministic delay behavior of Radio Block Center (RBC) handover scenario of CTCS-3 against critical safety, real-time, and liveness properties (defined in the system requirement specification (SRS) documents). Our experiments show that assurance of collision avoidance and train non-derailment can be guaranteed, but emergency brake intervention may happen with low probability in this scenario under the requirements specified in the SRS document.

A Synchronized Test Control Execution Model of Distributed Systems

Conformance testing may be seen as mean to execute an IUT (implementation under test) by carrying out test cases in order to observe whether the behavior of the IUT is conforming to its specifications. However, the development of distributed testing frameworks is more complex and the implementation of the parallel testing components (PTCs) should take into consideration the mechanisms and functions required to support interaction during PTC communication. In this article, the authors present another way to control the test execution of PTCs by introducing synchronization messages into the local test sequences. Then, they suggest an agent-based simulation to implement synchronized local test sequences and resolve the problem of control and synchronization.

Introducing complexity to formal testing

A general theory introducing asymptotic complexity to testing is presented. Our goal is measuring how fast the effort of testing must increase to reach higher levels of partial certainty on the correctness of the implementation under test (IUT). By recent works it is known that, for many practical testing scenarios, any partial level of correctness certainty less than 1 (where 1 means full certainty) can be reached by some finite test suite. In this paper we address the problem of finding out how fast must these test suites grow as long as the target level gets closer to 1. More precisely, we want to study how test suites grow with α, where α is the inverse of the distance to 1 (e.g., if α=4 then our target level is 0.75=1−14). A general theory to measure this testing complexity is developed. We use this theory to analyze the testing complexity of some general testing problems, as well as the complexity of some specific testing strategies for these problems, and discover that they are within e.g., O(logα), O(log2α), O(α), O(αlogα), or O(α). Similarly as the computational complexity theory conceptually distinguishes between the complexity of problems and algorithms, tightly identifying the complexity of a testing problem will require reasoning about any testing strategy for the problem. The capability to identify testing complexities will provide testers with a measure of the productivity of testing, that is, a measure of the utility of applying the (n+1)-th planned test case (after having passed the n previous ones) in terms of how closer would that additional test case get us to the (ideal) complete certainty on the IUT (in-)correctness.

Recent Advances in the Theory of Checking Experiments with State-oriented Models

We consider the problem of model-based testing of reactive systems with state-oriented models, such as finite state machines and transition systems, both distinguishing inputs and outputs. Inputs to an Implementation under Test (IUT) are generated by a tester, while outputs are analyzed by the latter to conclude whether the IUT is a conforming implementation of a given specification. This problem has its roots in the theory of checking experiments for finite state machines. It continues to attract a lot of attention of research community. In this paper, we provide a short overview of directions in which this theory has evolved. We report on our recent contributions and point to several unsolved tasks.

A Quantitative Approach to Input Generation in Real-Time Testing of Stochastic Systems

In the process of testing of concurrent timed systems, input generation identifies values of temporal parameters that let the Implementation Under Test (IUT) execute selected cases. However, when some parameters are not under control of the driver, test execution may diverge from the selected input and produce an inconclusive behavior. We formulate the problem on the basis of an abstraction of the IUT which we call partially stochastic Time Petri Net (psTPN), where controllable parameters are modeled as nondeterministic values and noncontrollable parameters as random variables with general (GEN) distribution. With reference to this abstraction, we derive the analytical form of the probability that the IUT runs along a selected behavior as a function of choices taken on controllable parameters. In the applicative perspective of real-time testing, this identifies a theoretical upper limit on the probability of a conclusive result, thus providing a means to plan the number of test repetitions that are necessary to guarantee a given probability of test-case coverage. It also provides a constructive technique for an optimal or suboptimal approach to input generation and a way to characterize the probability of conclusive testing under other suboptimal strategies.

Generating asynchronous test cases from test purposes

Context Input/output transition system (IOTS) models are commonly used when next input can arrive even before outputs are produced. The interaction between the tester and an implementation under test (IUT) is usually assumed to be synchronous. However, as the IUT can produce outputs at any moment, the tester should be prepared to accept all outputs from the IUT, or else be able to block (refuse) outputs of the implementation. Testing distributed, remote applications under the assumptions that communication is synchronous and actions can be blocked is unrealistic, since synchronous communication for such applications can only be achieved if special protocols are used. In this context, asynchronous tests can be more appropriate, reflecting the underlying test architecture which includes queues. Objective In this paper, we investigate the problem of constructing test cases for given test purposes and specification input/output transition systems, when the communication between the tester and the implementation under test is assumed to be asynchronous, performed via multiple queues. Method When issuing verdicts, asynchronous tests should take into account a distortion caused by the queues in the observed interactions. First, we investigate how the test purpose can be transformed to account for this distortion when there are a single input queue and a single output queue. Then, we consider a more general problem, when there may be multiple queues. Results We propose an algorithm which constructs a sound test case, by transforming the test purpose prior to composing it with the specification without queues. Conclusion The proposed algorithm mitigates the state explosion problem which usually occurs when queues are directly involved in the composition. Experimental results confirm the resulting state space reduction.

Checking experiments for stream X-machines

Stream X-machines are a state based formalism that has associated with it a particular development process in which a system is built from trusted components. Testing thus essentially checks that these components have been combined in a correct manner and that the orders in which they can occur are consistent with the specification. Importantly, there are test generation methods that return a checking experiment: a test that is guaranteed to determine correctness as long as the implementation under test (IUT) is functionally equivalent to an unknown element of a given fault domain Ψ . Previous work has show how three methods for generating checking experiments from a finite state machine (FSM) can be adapted to testing from a stream X-machine. However, there are many other methods for generating checking experiments from an FSM and these have a variety of benefits that correspond to different testing scenarios. This paper shows how any method for generating a checking experiment from an FSM can be adapted to generate a checking experiment for testing an implementation against a stream X-machine. This is the case whether we are testing to check that the IUT is functionally equivalent to a specification or we are testing to check that every trace (input/output sequence) of the IUT is also a trace of a nondeterministic specification. Interestingly, this holds even if the fault domain Ψ used is not that traditionally associated with testing from a stream X-machine. The results also apply for both deterministic and nondeterministic implementations.

Estimating the feasibility of transition paths in extended finite state machines

There has been significant interest in automating testing on the basis of an extended finite state machine (EFSM) model of the required behaviour of the implementation under test (IUT). Many test criteria require that certain parts of the EFSM are executed. For example, we may want to execute every transition of the EFSM. In order to find a test suite (set of input sequences) that achieves this we might first derive a set of paths through the EFSM that satisfy the criterion using, for example, algorithms from graph theory. We then attempt to produce input sequences that trigger these paths. Unfortunately, however, the EFSM might have infeasible paths and the problem of determining whether a path is feasible is generally undecidable. This paper describes an approach in which a fitness function is used to estimate how easy it is to find an input sequence to trigger a given path through an EFSM. Such a fitness function could be used in a search-based approach in which we search for a path with good fitness that achieves a test objective, such as executing a particular transition, and then search for an input sequence that triggers the path. If this second search fails then we search for another path with good fitness and repeat the process. We give a computationally inexpensive approach (fitness function) that estimates the feasibility of a path. In order to evaluate this fitness function we compared the fitness of a path with the ease with which an input sequence can be produced using search to trigger the path and we used random sampling in order to estimate this. The empirical evidence suggests that a reasonably good correlation (0.72 and 0.62) exists between the fitness of a path, produced using the proposed fitness function, and an estimate of the ease with which we can randomly generate an input sequence to trigger the path.

Exploring alternatives for transition verification

When an implementation under test (IUT) is state-based, and its expected abstract behavior is given in terms of a finite state machine (FSM), a checking sequence generated from a specification FSM and applied to an IUT for testing can provide us with high-level confidence in the correct functional behavior of our implementation. One of the issues here is to generate efficient checking sequences in terms of their lengths. As a major characteristics, a checking sequence must contain all β-sequences for transition verification. In this paper, we discuss the possibility of reducing the lengths of checking sequences by making use of the invertible transitions in the specification FSM to increase the choice of β-sequences to be considered for checking sequence generation. We present a sufficient condition for adopting alternative β-sequences and illustrate typical ways of incorporating these alternative β-sequences into existing methods for checking sequence generation to reduce the lengths. Compared to the direct use of three existing methods, our experiments show that most of the time the saving gained by adopting alternative β-sequences falls in the range of 10–40%.

Fault masking by multiple timing faults in timed EFSM models

Detection of multiple timing faults is a challenging task because these faults, although may be detectable individually, can mask each other's faulty behavior, making a faulty implementation under test (IUT) indistinguishable from a non-faulty one during testing. This phenomenon, called fault masking, is formally defined in this paper. It is proven that graph augmentation algorithms proposed for timed Extended Finite State Machines (EFSMs) with multiple timers can detect pairwise occurrences of classes of timing faults in an IUT and, hence, detects fault masking.

Multiple UIO-based test sequence generation for distributed systems

In developing distributed systems, conformance testing is required to determine whether an implementation under test (IUT) conforms to its specification. With distributed test architectures involving multiple remote testers, testing approaches may become more complicated because of issues known as controllability and observability problems. Based on a finite state machine (FSM) representation of the system’s specification, this paper proposes a new method to generate a test sequence utilizing multiple UIO sequences. The method is essentially guided by the way of minimizing the use of external coordination messages and input/output operations. Experiments are given to evaluate the proposed method.

Testing data processing-oriented systems from stream X-machine models

One of the great benefits of using a stream X-machine to specify a system is its associated testing method. Under certain design for test conditions, this method produces a test suite that can determine the correctness of the implementation under test ( IUT), provided that the processing functions of the stream X-machine specification have been correctly implemented. The method was originally developed for controllable stream X-machines. A recent paper generalizes the original method by considering specifications that do not meet the controllability requirement. However, it is still required that a controllable stream X-machine model of the IUT exists and the size of the test suite produced strongly depends on the (estimated) upper bound on the number of states of this controllable model. While this assumption is in general reasonable for most interactive systems, it may produce unmanageable test suites for even simple data processing-oriented applications. This paper provides a new variant of the stream X-machine based testing method that no longer depends on the size of a controllable model of the IUT. In data processing-oriented applications, the new method can drastically reduce the size of the test suite produced at the expense of a (possibly) more complex generation process.

Algorithms for Modeling a Class of Single Timing Faults in Communication Protocols

A set of graph augmentation algorithms is introduced to model a class of timing faults in timed-EFSM models. It is shown that the test sequences generated based on our models can detect 1 -clock and n-clock timing faults and incorrect timer setting faults in an implementation under test (IUT). It is proven that the size of the augmented graph resulting from our augmentation algorithms is on the same order of magnitude as that of the original specification.

Timing issues in distributed testing

The objective of conformance testing is to determine whether an implementation under test (IUT) conforms to its specification. In distributed test architecture where there are multiple remote testers, the objective can be complicated by the fact that testers may encounter controllability and observability problems during the application of a test sequence. A certain amount of work has been done in the area of generating test sequence that is free from these problems. However, few researchers investigate them from the aspect of test execution. This work studies the test execution phase when test sequences are applied to the implementation and it is pointed out that controllability and observability problems can be resolved if and only if the test system implements some timing constraints. When determining these constraints, the dynamic time information during test is taken into account, which reduces the test execution time and improves test efficiency further.

Heuristics for fault diagnosis when testing from finite state machines

AbstractWhen testing from finite state machines, a failure observed in the implementation under test (IUT) is called a symptom. A symptom could have been caused by an earlier state transfer failure. Transitions that may be used to explain the observed symptoms are called diagnosing candidates. Finding strategies to generate an optimal set of diagnosing candidates that could effectively identify faults in the IUT is of great value in reducing the cost of system development and testing. This paper investigates fault diagnosis when testing from finite state machines and proposes heuristics for fault isolation and identification. The proposed heuristics attempt to lead to a symptom being observed in some shorter test sequences, which helps to reduce the cost of fault isolation and identification. The complexity of the proposed method is analysed. A case study is presented, which shows how the proposed approach assists in fault diagnosis. Copyright © 2006 John Wiley & Sons, Ltd.

Reducing the cost of applying adaptive test cases

The testing of a state-based system may involve the application of a number of adaptive test cases. Where the implementation under test (IUT) is deterministic, the response of the IUT to some adaptive test case γ1 could be capable of determining the response of the IUT to another adaptive test case γ2. Thus, the expected cost of applying a set of adaptive test cases depends upon the order in which they are applied. This paper explores properties of adaptive test cases and considers the problem of finding an order of application of the elements from some set of adaptive test cases, which minimises the expected cost of testing.