Test criteria for model-checking-assisted test case generation: A computational study

Abstract

Test case generation is often cited as one of the most challenging tasks in testing dependable systems [9]. Besides benefits as a verification technique by its own right, model checking is emerging as an efficient method for automating test case generation. Existing testing criteria and a range of new criteria, namely the vacuity-based ones, inspired by formal requirements have been used in model-checking-assisted test generation. This paper reviews some of these existing and new test criteria. We developed a unified framework for evaluating the effectiveness of these test criteria and the efficiency of model-checking-assisted test generation for these criteria. The benefits of this work are three-fold: first, the computational study carried out in this work assesses the practical effectiveness and efficiency of model-checking-assisted test case generation, which are important metrics to consider for selecting the right test criteria and test generation approach. Second, we propose a unified test generation framework based on generalized Büchi automata. The framework uses the same off-the-shelf model checker, in this case, SPIN model checker [10], to generate test cases for different criteria and compare them on a consistent basis. Last but not least, we describe in great details the methodology and automated test generation environment that we developed on the basis of our unified framework. Such details are of interest to researchers who needs to carry out their own experimental study on test criteria, and to practitioners who want to integrate model-checking-assisted test generation into their testing process.