Distributed Test Architecture Research Articles

Review of Live-Virtual-Constructive Simulation Technology

With the improvement of network communication technology, creative computer simulation architectures and today’s cutting-edge hardware, the live-virtual-constructive (LVC) simulation technology gradually matures adequately to synthesize joint heterogeneous systems. Firstly, the development of typical LVC architectures and their key technologies are analysed, such as Advanced Distributed Simulation (ADS), High-Level Architecture (HLA), Test and Train Enabling Architecture (TENA), Joint Simulation Supporting Platform (JoSim) and so force. Then, applications in the field of US and Chinese military are reviewed, as well as cases in air combat training and other scholarly research concept. Finally, a summary of LVC technology is given, and the development trend of military LVC simulation technology is predicted to explore the specific content that needs to be studied.

Bounded Reordering in the Distributed Test Architecture

In the distributed test architecture, the system under test (SUT) interacts with its environment at multiple physically distributed ports and the local testers at these ports do not synchronize their actions. This presents many challenges and, in particular, apparently incorrect behaviors can be the consequence of an erroneous assumption about the exact order in which actions were performed at different ports. In previous work, we defined a conformance relation for the distributed test architecture. Essentially, the SUT is faulty if we observe a trace $\sigma$ such that no admissible reordering of the actions in $\sigma$ could have been produced by the specification. However, this notion can be weak if the compared traces might be too different. This paper introduces conformance relations where, for a given metric, a reordering is only considered if the distance between the two traces is at most a certain bound $k$ . We introduce two different metrics and provide algorithms to construct finite automata accepting these close , with respect to each metric, sequences. We also study the computational complexity of the two main problems associated with the new framework: deciding whether a trace is accepted by the new automaton and deciding whether one system conforms to a specification with respect to the new conformance relation.

Implementation relations and probabilistic schedulers in the distributed test architecture

Research partially supported by the MINECO/FEDER project DArDOS (TIN2015- 65845-C3-1-R) and the Comunidad de Madrid project SICOMORo-CM (S2013/ICE-3006).

Distinguishing Sequences for Distributed Testing: Preset Distinguishing Sequences

This paper concerns the problem of testing from a finite state machine (FSM) $M$ modelling a system that interacts with its environment at multiple physically distributed interfaces, called ports. We assume that the distributed test architecture is used: there is a local tester at each port, the tester at port $p$ only observes events at $p$ and the testers do not interact during testing. This paper formalizes the notion of an adaptive test strategy and what it means for an adaptive test strategy to be controllable. We provide algorithms to check whether a global strategy is controllable and to generate a controllable adaptive distinguishing sequence (ADS). We prove that controllable ADS existence is PSPACE-Hard and that the problem of deciding whether $M$ has a controllable ADS with length $\ell $ is NP-Hard. In practice, there is likely to be a polynomial upper bound on the length of ADS in which we are interested and for this case the decision problem is NP-Complete.

Distinguishing Sequences for Distributed Testing: Adaptive Distinguishing Sequences

This paper concerns the problem of testing from a finite state machine (FSM) | $M$ | modelling a system that interacts with its environment at multiple physically distributed interfaces, called ports. We assume that the distributed test architecture is used: there is a local tester at each port, the tester at port | $p$ | only observes events at | $p$ | and the testers do not interact during testing. This paper formalizes the notion of an adaptive test strategy and what it means for an adaptive test strategy to be controllable. We provide algorithms to check whether a global strategy is controllable and to generate a controllable adaptive distinguishing sequence (ADS). We prove that controllable ADS existence is PSPACE-Hard and that the problem of deciding whether | $M$ | has a controllable ADS with length | $\ell $ | is NP-Hard . In practice, there is likely to be a polynomial upper bound on the length of ADS in which we are interested and for this case the decision problem is NP-Complete .

최적화된 동기적 적합성시험 항목의 발견적 생성 방법

This paper presents a new technique for generating an optimum synchronizable test sequence that can be applied in the distributed test architecture where both external synchronization and input/output operation costs are taken into consideration. The method defines a set of phases that constructs a tester-related digraph from a given finite state machine representation of a protocol specification such that a minimum cost tour of the digraph with intrinsically synchronizable transfer sequences can be used to generate an optimum synchronizable test sequence using synchronizable state identification sequences as the state recognition sequence for each state of the given finite state machine. This hybrid approach with a heuristic and optimization technique provides a simple and elegant solution to the synchronization problem that arises during the application of a predetermined test sequence in some protocol test architectures that utilize remote testers.

Model-based testing of global properties on large-scale distributed systems

ContextLarge-scale distributed systems are becoming commonplace with the large popularity of peer-to-peer and cloud computing. The increasing importance of these systems contrasts with the lack of integrated solutions to build trustworthy software. A key concern of any large-scale distributed system is the validation of global properties, which cannot be evaluated on a single node. Thus, it is necessary to gather data from distributed nodes and to aggregate these data into a global view. This turns out to be very challenging because of the system’s dynamism that imposes very frequent changes in local values that affect global properties. This implies that the global view has to be frequently updated to ensure an accurate validation of global properties. ObjectiveIn this paper, we present a model-based approach to define a dynamic oracle for checking global properties. Our objective is to abstract relevant aspects of such systems into models. These models are updated at runtime, by monitoring the corresponding distributed system. MethodWe conduce real-scale experimental validation to evaluate the ability of our approach to check global properties. In this validation, we apply our approach to test two open-source implementations of distributed hash tables. The experiments are deployed on two clusters of 32 nodes. ResultsThe experiments reveal an important defect on one implementation and show clear performance differences between the two implementations. The defect would not be detected without a global view of the system. ConclusionTesting global properties on distributed software consists of gathering data from different nodes and building a global view of the system, where properties are validated. This process requires a distributed test architecture and tools for representing and validating global properties. Model-based techniques are an expressive mean for building oracles that validate global properties on distributed systems.

Timed implementation relations for the distributed test architecture

In order to test systems that have physically distributed interfaces, called ports, we might use a distributed approach in which there is a separate tester at each port. If the testers do not synchronise during testing then we cannot always determine the relative order of events observed at different ports and this leads to new notions of correctness that have been described using corresponding implementation relations. We study the situation in which each tester has a local clock and timestamps its observations. If we know nothing about how the local clocks relate then this does not affect the implementation relation while if the local clocks agree exactly then we can reconstruct the sequence of observations made. In practice, however, we are likely to be between these extremes: the local clocks will not agree exactly but we have some information regarding how they can differ. We start by assuming that a local tester interacts synchronously with the corresponding port of the system under test and then extend this to the case where communications can be asynchronous, considering both the first-in-first-out (FIFO) case and the non-FIFO case. The new implementation relations are stronger than implementation relations for distributed testing that do not use timestamps but still reflect the distributed nature of observations. This paper explores these alternatives and derives corresponding implementation relations.

Controllable Testing from Nondeterministic Finite State Machines with Multiple Ports

Some systems have physically distributed interfaces, called ports, at which they interact with their environment. We place a tester at each port and if the testers cannot directly communicate and there is no global clock then we are using the distributed test architecture. It is known that this test architecture introduces controllability problems when testing from a deterministic finite state machine. This paper investigates the problem of testing from a nondeterministic finite state machine in the distributed test architecture and explores controllability. It shows how we can decide in polynomial time whether an input sequence is controllable. It also gives an algorithm for generating such an input sequence \bar{x} and shows how we can produce testers that implement \bar{x}.

Scenarios‐based testing of systems with distributed ports

SUMMARYDistributed systems are usually composed of several distributed components that communicate with their environment through specific ports. When testing such a system we separately observe sequences of inputs and outputs at each port rather than a global sequence and potentially cannot reconstruct the global sequence that occurred. Typically, the users of such a system cannot synchronize their actions during use or testing. However, the use of the system might correspond to a sequence of scenarios, where each scenario involves a sequence of interactions with the system that, for example, achieves a particular objective. When this is the case there is the potential for a significant delay between two scenarios and this effectively allows the users of the system to synchronize between scenarios. If we represent the specification of the global system by using a state‐based notation, we say that ait scenario is any sequence of events that happens between two of these operations. We can encode scenarios in two different ways. The first approach consists of marking some of the states of the specification to denote these synchronization points. It transpires that there are two ways to interpret such models and these lead to two implementation relations. The second approach consists of adding a set of traces to the specification to represent the traces that correspond to scenarios. We show that these two approaches have similar expressive power by providing an encoding from marked states to sets of traces. In order to assess the appropriateness of our new framework, we show that it represents a conservative extension of previous implementation relations defined in the context of the distributed test architecture: if we consider that all the states are marked then we simply obtain ioco (the classical relation for single‐port systems) while if no state is marked then we obtain dioco (our previous relation for multi‐port systems). Finally, we concentrate on the study of controllable test cases, that is, test cases such that each local tester knows exactly when to apply inputs. We give two notions of controllable test cases, define an implementation relation for each of these notions and relate them. We also show how we can decide whether a test case satisfies these conditions. Copyright © 2011 John Wiley & Sons, Ltd.

Canonical finite state machines for distributed systems

There has been much interest in testing from finite state machines (FSMs) as a result of their suitability for modelling or specifying state-based systems. Where there are multiple ports/interfaces a multi-port FSM is used and in testing, a tester is placed at each port. If the testers cannot communicate with one another directly and there is no global clock then we are testing in the distributed test architecture. It is known that the use of the distributed test architecture can affect the power of testing and recent work has characterised this in terms of local s-equivalence: in the distributed test architecture we can distinguish two FSMs, such as an implementation and a specification, if and only if they are not locally s-equivalent. However, there may be many FSMs that are locally s-equivalent to a given FSM and the nature of these FSMs has not been explored. This paper examines the set of FSMs that are locally s-equivalent to a given FSM M . It shows that there is a unique smallest FSM χ min ( M ) and a unique largest FSM χ max ( M ) that are locally s-equivalent to M . Here smallest and largest refer to the set of traces defined by an FSM and thus to its semantics. We also show that for a given FSM M the set of FSMs that are locally s-equivalent to M defines a bounded lattice. Finally, we define an FSM that, amongst all FSMs locally s-equivalent to M , has fewest states. We thus give three alternative canonical FSMs that are locally s-equivalent to an FSM M : one that defines the smallest set of traces, one that defines the largest set of traces, and one with fewest states. All three provide valuable information and the first two can be produced in time that is polynomial in terms of the number of states of M . We prove that the problem of finding an s-equivalent FSM with fewest states is NP-hard in general but can be solved in polynomial time for the special case where there are two ports.

Using status messages in the distributed test architecture

If the system under test has multiple interfaces/ports and these are physically distributed then in testing we place a tester at each port. If these testers cannot directly communicate with one another and there is no global clock then we are testing in the distributed test architecture. If the distributed test architecture is used then there may be input sequences that cannot be applied in testing without introducing controllability problems. Additionally, observability problems can allow fault masking. In this paper we consider the situation in which the testers can apply a status message: an input that causes the system under test to identify its current state. We show how such a status message can be used in order to overcome controllability and observability problems.

Multiple UIO-based test sequence generation for distributed systems

In developing distributed systems, conformance testing is required to determine whether an implementation under test (IUT) conforms to its specification. With distributed test architectures involving multiple remote testers, testing approaches may become more complicated because of issues known as controllability and observability problems. Based on a finite state machine (FSM) representation of the system’s specification, this paper proposes a new method to generate a test sequence utilizing multiple UIO sequences. The method is essentially guided by the way of minimizing the use of external coordination messages and input/output operations. Experiments are given to evaluate the proposed method.

Checking sequences for distributed test architectures

Controllability and observability problems may manifest themselves during the application of a checking sequence in a test architecture where there are multiple remote testers. These problems often require the use of external coordination message exchanges among testers during testing. However, the use of coordination messages requires the existence of an external network that can increase the cost of testing and can be difficult to implement. In addition, the use of coordination messages introduces delays and this can cause problems where there are timing constraints. Thus, sometimes it is desired to construct a checking sequence from the specification of the system under test that will be free from controllability and observability problems without requiring the use of external coordination message exchanges. This paper gives conditions under which it is possible to produce such a checking sequence, using multiple distinguishing sequences, and an algorithm that achieves this.

The Effect of the Distributed Test Architecture on the Power of Testing

There has been much interest in testing from finite-state machines (FSMs). If the system under test can be modelled by the (minimal) FSM N then testing from an (minimal) FSM M is testing to check that N is isomorphic to M. In the distributed test architecture, there are multiple interfaces/ports and there is a tester at each port. This can introduce controllability/synchronization and observability problems. This paper shows that the restriction to test sequences that do not cause controllability problems and the inability to observe the global behaviour in the distributed test architecture, and thus relying only on the local behaviour at remote testers, introduces fundamental limitations into testing. There exist minimal FSMs that are not equivalent, and so are not isomorphic, and yet cannot be distinguished by testing in this architecture without introducing controllability problems. Similarly, an FSM may have non-equivalent states that cannot be distinguished in the distributed test architecture without causing controllability problems: these are said to be locally s-equivalent and otherwise they are locally s-distinguishable. This paper introduces the notion of two states or FSMs being locally s-equivalent and formalizes the power of testing in the distributed test architecture in terms of local s-equivalence. It introduces a polynomial time algorithm that, given an FSM M, determines which states of M are locally s-equivalent and produces minimal length input sequences that locally s-distinguish states that are not locally s-equivalent. An FSM is locally s-minimal if it has no pair of locally s-equivalent states. This paper gives an algorithm that takes an FSM M and returns a locally s-minimal FSM M′ that is locally s-equivalent to M.

Timing issues in distributed testing

The objective of conformance testing is to determine whether an implementation under test (IUT) conforms to its specification. In distributed test architecture where there are multiple remote testers, the objective can be complicated by the fact that testers may encounter controllability and observability problems during the application of a test sequence. A certain amount of work has been done in the area of generating test sequence that is free from these problems. However, few researchers investigate them from the aspect of test execution. This work studies the test execution phase when test sequences are applied to the implementation and it is pointed out that controllability and observability problems can be resolved if and only if the test system implements some timing constraints. When determining these constraints, the dynamic time information during test is taken into account, which reduces the test execution time and improves test efficiency further.

DOS Middleware Instrumentation for Ensuring Reproducibility of Testing Procedures

One of the important issues in testing distributed applications is to provide automated control of the nondeterminism by guarding the interprocess communications (IPC) in the presence of multiple execution choices, so that we can force the execution along the expected path and make the testing reproducible. As distributed object system (DOS) middlewares are now widely recognized and adopted, it is desirable to build up the distributed test architecture with an instrumental layer in the middlewares to gain the control over the IPC. Adding this layer transparently from the existing middleware implementations is enabled by some special interfaces provided by the middlewares for interception services. Here, we discuss the design issues on developing such an instrumental layer in DOS middlewares as part of our testing environment for reproducible testing of distributed applications

Overcoming observability problems in distributed test architectures

The introduction of multiple remote testers to apply a test or checking sequence in a test architecture brings out the possibility of controllability and observability problems. These problems often require the use of external coordination message exchanges among testers. In this paper, we consider constructing a test or checking sequence from the specification of the system under test such that it will be free from these problems and will not require the use of external coordination messages. We give an algorithm that can check whether it is possible to construct subsequences from a given specification that eliminate the need for using external coordination message exchanges, and when it is possible actually produces such subsequences.

Constructing checking sequences for distributed testing

Abstract The objective of testing is to determine whether an implementation under test conforms to its specification. In distributed test architectures involving multiple remote testers, this objective can be complicated by the fact that testers may encounter coordination problems relating to controllability (synchronization) and observability during the application of tests. Based on a finite state machine (FSM) specification of the externally observable behaviour of a distributed system and a distinguishing sequence, this paper proposes a method for constructing a checking sequence where there is no potential controllability or observability problems, and where the use of external coordination message exchanges among testers is minimized. The proposed method does not assume a reliable reset feature in the implementations of the given FSM to be tested by the resulting checking sequence.

Distributed testing without encountering controllability and observability problems

The objective of testing is to determine whether a system under test conforms to its specification. In distributed test architectures that utilize remote testers, this objective can be complicated by the fact that testers may encounter problems relating to controllability and observability during the application of a test sequence. Existing solutions to these problems involve first constructing a test sequence from the specification of an implementation under test, and then inserting coordination messages or appending selected test subsequences that prevent the occurrences of controllability and observability problems during the application of the resulting test sequence. This paper proposes a method that utilizes a set of transformation rules to construct an auxiliary directed graph from a given specification, and constructs a rural Chinese postman tour in this graph to yield a minimum-length test sequence where there is no potential controllability or observability problems, and where the use of coordination messages is minimized.