The power of single and error annotations in category partition testing

Abstract

Category Partition (CP) is a black box testing technique that formalizes the specification of the input domain in a CP specification for the system under test. A CP specification is driven by tester's expertise and bundles parameters, categories (characteristics of parameters) and choices (acceptable values for categories) required for extensively testing the system. For completeness the choices correspond to permitted input values as well as some values to account for boundaries or robustness. These choices are then combined to form test frames on the basis of various criteria such as each choice or pairwise. To ensure that the combinations of choices are feasible and account for valid sets of user requirements, constraints are introduced to specify permitted combinations among choices, and to specify single or error choices. In a typical development environment where testing is driven by stringent deadlines a tester might have to decide how many constraints are enough to attain the maximum level of test completeness. The present work will assist a test engineer in making this decision. We conclude, on the basis of our experimental evaluation on academic and industrial case studies, that an equally effective test suite can be attained by meticulously defining error and single annotations in a CP specification while ignoring other constraints among choices.