Test Template Framework Research Articles

Coverage Criteria for Set-Based Specifications

Model-based testing (MBT) studies how test cases are generated from a model of the system under test (SUT). Many MBT methods rely on building an automaton from the model and then they generate test cases by covering the automaton with different path coverage criteria. However, if a model of the SUT is a logical formula over some complex mathematical theories (such as set theory) it may be more natural or intuitive to apply coverage criteria directly over the formula. On the other hand, domain partition, i.e. the partition of the input domain of model operations, is one of the main techniques in MBT. Partitioning is conducted by applying different rules or heuristics. Engineers may find it difficult to decide what, where and how these rules should be applied. In this paper we propose a set of coverage criteria based on domain partition for set-based specifications. We call them testing strategies. Testing strategies play a similar role to path- or data-based coverage criteria in structural testing. Furthermore, we show a partial order of testing strategies as is done in structural testing. We also describe an implementation of testing strategies for the Test Template Framework, which is a MBT method for the Z notation; and a scripting language that allows users to implement testing strategies.

Tool support for the Test Template Framework

This paper describes tool support that has been implemented for the Test Template Framework (TTF). The TTF is a model-based testing (MBT) method that is especially well suited for unit testing from Z specifications. Although the TTF is a sound MBT method and it has been widely referenced since its first publication, attention in recent years has decayed. In fact, some have argued that generating abstract test cases following the TTF is a manual task requiring its users to perform complex predicate manipulations. This paper shows that these observations are dubious by describing Fastest, a tool that implements solutions for all these issues and, according to many experiments, produces abstract test cases for more than 80% of the satisfiable test specifications. Furthermore, it is claimed that Fastest fulfils the needs of the Z user community regarding MBT tools, which is supported with a range of case studies. Copyright © 2012 John Wiley & Sons, Ltd.

Applying SMT Solvers to the Test Template Framework

The Test Template Framework (TTF) is a model-based testing method for the Z notation. In the TTF, test cases are generated from test specifications, which are predicates written in Z. In turn, the Z notation is based on first-order logic with equality and Zermelo-Fraenkel set theory. In this way, a test case is a witness satisfying a formula in that theory. Satisfiability Modulo Theory (SMT) solvers are software tools that decide the satisfiability of arbitrary formulas in a large number of built-in logical theories and their combination. In this paper, we present the first results of applying two SMT solvers, Yices and CVC3, as the engines to find test cases from TTF's test specifications. In doing so, shallow embeddings of a significant portion of the Z notation into the input languages of Yices and CVC3 are provided, given that they do not directly support Zermelo-Fraenkel set theory as defined in Z. Finally, the results of applying these embeddings to a number of test specifications of eight cases studies are analysed.

A framework for specification-based testing

Test templates and a test template framework are introduced as useful concepts in specification-based testing. The framework can be defined using any model-based specification notation and used to derive tests from model-based specifications-in this paper, it is demonstrated using the Z notation. The framework formally defines test data sets and their relation to the operations in a specification and to other test data sets, providing structure to the testing process. Flexibility is preserved, so that many testing strategies can be used. Important application areas of the framework are discussed, including refinement of test data, regression testing, and test oracles.