The Practical Assessment of Test Sets with Inductive Inference Techniques

Abstract

Inductive inference is the process of hypothesizing a model from a set of examples. It can be considered to be the inverse of program testing, which is the process of generating a finite set of tests that are intended to fully exercise a software system. This relationship has been acknowledged for almost 30 years, and has led to the emergence of several induction-based techniques that aim either to generate suitable test sets or assess the adequacy of existing test sets. Unfortunately these techniques are usually deemed to be too impractical, because they are based on exact inference, requiring a vast set of examples or tests. In practice a test set can still be adequate if the inferred model contains minor errors. This paper shows how the Probably Approximately Correct (PAC) framework, a well-established approach in the field of inductive inference, can be applied to inductive testing techniques. This facilitates a more pragmatic assessment of these techniques by allowing for a degree of error. This evaluation framework gives rise to a challenge: To identify the best combination of testing and inference techniques that produce practical and (approximately) adequate test sets.