Metamorphic Runtime Checking of Non-Testable Programs

Abstract

Challenges arise in assuring the quality of applications that do not have test oracles, i.e., for which it is difficult or impossible to know what the correct output should be for arbitrary input. Recently, metamorphic testing [7] has been shown to be a simple yet effective technique in addressing the quality assurance of these so-called “non-testable programs” [51]. In metamorphic testing, existing test case input is modified to produce new test cases in such a manner that, when given the new input, the function should produce an output that can easily be computed based on the original output. That is, if input x produces output f (x), then we create input x’ such that we can predict f (x’ ) based on f (x); if the application does not produce the expected output, then a defect must exist, and either f (x) or f (x’ ) (or both) is wrong. Previously we have presented an approach called “Automated Metamorphic System Testing” [37], in which metamorphic testing is conducted automatically as the program executes. In the approach, metamorphic properties of the entire application are specified, and then checked after execution is complete. Here, we improve upon that work by presenting a technique in which the metamorphic properties of individual functions are used, allowing for the specification of more complex properties and enabling finer-grained runtime checking. Our goal is to demonstrate that such an approach will be more effective than one based on specifying metamorphic properties at the system level, and is also feasible for use in the deployment environment. This technique, called Metamorphic Runtime Checking, is a system testing approach in which the metamorphic properties of individual functions are automatically checked during the program’s execution. The tester is able to easily specify the functions’ properties so that metamorphic testing can be conducted in a running application, allowing the tests to execute using real input data and in the context of real system states, without affecting those states. We also describe an implementation framework called Columbus, and present the results of empirical studies that demonstrate that checking Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. To copy otherwise, to republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. FSE 2009 Amsterdam, The Netherlands Copyright 200X ACM X-XXXXX-XX-X/XX/XX ...$5.00. the metamorphic properties of individual functions increases the effectiveness of the approach in detecting defects, with minimal performance impact.