Model-based testing of stochastically timed systems

Mariëlle Stoelinga,Marcus Gerhold,Arnd Hartmanns

doi:10.1007/s11334-019-00349-z

Mariëlle Stoelinga, Marcus Gerhold + Show 1 more

Open Access

https://doi.org/10.1007/s11334-019-00349-z

Copy DOI

Abstract

Many systems are inherently stochastic: they interact with unpredictable environments or use randomised algorithms. Classical model-based testing is insufficient for such systems: it only covers functional correctness. In this paper, we present two model-based testing frameworks that additionally cover the stochastic aspects in hard and soft real-time systems. Using the theory of Markov automata and stochastic automata for specifications, test cases, and a formal notion of conformance, they provide clean mechanisms to represent underspecification, randomisation, and stochastic timing. Markov automata provide a simple memoryless model of time, while stochastic automata support arbitrary continuous and discrete probability distributions. We cleanly define the theoretical foundations, outline practical algorithms for statistical conformance checking, and evaluate both frameworks’ capabilities by testing timing aspects of the Bluetooth device discovery protocol. We highlight the trade-off of simple and efficient statistical evaluation for Markov automata versus precise and realistic modelling with stochastic automata.

Highlights

Model-based testing (MBT) [50] is a technique to automatically generate, execute, and evaluate test suites on black-box implementations under test (IUT)
We present two MBT frameworks based on input–output Markov automata [17] (IOMA) and stochastic automata [11,12] (IOSA), which are transition systems augmented with discrete probabilistic choices and stochastic delays
This paper provides a new integrated presentation of our previous papers on model-based testing for Markov automata [21] and stochastic automata [19]

Summary

Introduction

Model-based testing (MBT) [50] is a technique to automatically generate, execute, and evaluate test suites on black-box implementations under test (IUT). The theoretical ingredients of an MBT framework are a formal model that specifies the desired system behaviour, often in terms of (some extension of) input–output transition systems; a notion of conformance that specifies when an IUT is considered a valid implementation of the model; and a precise definition of what a test case is. For the framework to be applicable in practice, we need algorithms to derive test cases from the model, execute them on the IUT, and evaluate the results, i.e. decide conformance. They need to be sound (i.e. every implementation that fails a test case does not conform to the model), and ideally complete In the presence of nondeterminism, the IUT is allowed to use any deterministic or randomised policy to decide between the specified alternatives

Objectives

Methods

Results

Discussion

Conclusion