Observation inaccuracy in conformance testing with multiple testers

Abstract

In the conformance testing of a software system, independent multiple testers (or observers) are often used, each providing input messages to and receiving output messages from the software system. For an execution of the software system, if all testers have observed correct results, the intuitive conclusion is that this execution is correct. However, this conclusion may be wrong, and this problem is referred to as multi-tester observation inaccuracy. In this paper, for a specification written as a deterministic or nondeterministic finite state machine (FSM), we present a necessary and sufficient condition for an incorrect test observation by multiple testers. For a specification written as a deterministic FSM, we define three types of implementation faults that can cause incorrect test observations: input exchange, forward output shifting and backward output shifting faults. We also propose a strategy for solving the problem of multi-tester observation inaccuracy.