Introducing complexity to formal testing

Abstract

A general theory introducing asymptotic complexity to testing is presented. Our goal is measuring how fast the effort of testing must increase to reach higher levels of partial certainty on the correctness of the implementation under test (IUT). By recent works it is known that, for many practical testing scenarios, any partial level of correctness certainty less than 1 (where 1 means full certainty) can be reached by some finite test suite. In this paper we address the problem of finding out how fast must these test suites grow as long as the target level gets closer to 1. More precisely, we want to study how test suites grow with α, where α is the inverse of the distance to 1 (e.g., if α=4 then our target level is 0.75=1−14). A general theory to measure this testing complexity is developed. We use this theory to analyze the testing complexity of some general testing problems, as well as the complexity of some specific testing strategies for these problems, and discover that they are within e.g., O(logα), O(log2α), O(α), O(αlogα), or O(α). Similarly as the computational complexity theory conceptually distinguishes between the complexity of problems and algorithms, tightly identifying the complexity of a testing problem will require reasoning about any testing strategy for the problem. The capability to identify testing complexities will provide testers with a measure of the productivity of testing, that is, a measure of the utility of applying the (n+1)-th planned test case (after having passed the n previous ones) in terms of how closer would that additional test case get us to the (ideal) complete certainty on the IUT (in-)correctness.