A metamorphic testing approach for supporting program repair without the need for a test oracle

Abstract

Test suite based automated program repair (APR) relies on a test oracle to determine the execution result of individual test cases. The applicability of APR techniques, therefore, is limited by the fact that test oracles may not exist. Metamorphic Testing (MT) is a testing approach that, rather than checking the correctness of individual test outputs, checks testing results through verification of relations among multiple test cases and their outputs: MT can therefore be applied without test oracles. This paper presents an integration of MT with APR that enables application of APR without the need for a test oracle. Two important issues for this integration which have been thoroughly investigated and addressed are: (1) feasibility — which is addressed by proposing a framework to support the integration, and then presenting MT-GenProg, a tool incorporating MT with the popular APR technique GenProg; and (2) effectiveness — which is confirmed through an empirical study of GenProg and MT-GenProg on 1,143 program versions from the IntroClass benchmark suite, demonstrating MT-GenProg’s comparable performance to GenProg, in terms of repair effectiveness. We conclude that the proposed integration is both practically feasible and effective, and thus successfully extends APR techniques to a broader application domain.