Measurable Metrics for Qualitative Guidelines of Metamorphic Relation

Abstract

In software testing, oracles do not always exist or are too hard to generate, which is called test oracle problem. Metamorphic testing (MT) provide an effective way to alleviate the problem based on metamorphic relations (MR). As a MR is satisfied by a set of cases, it does not mean that the cases are success. Therefore, more MRs are needed to improve the effectiveness of MT. For different programs, great deals of valuable MRs have been generated. But not all of them are effective in fault detection. Then, the next issue is how to select the better MRs which could detect the failed cases as soon as possible. To resolve this problem, several qualitative guidelines were proposed for MR priority recently. For the MRs described with different forms, these guidelines could be used to select the effective MRs and improve the performance of MT observably. In our previous research, we constructed a formal model for MR, with which a MR could be decomposed into three sub relations. In this paper, based on the sub relations of MR, we provide three measurable metrics for MR selection: in-degree of MR, algebra complexity of MR, and distance between inputs of MR. To evaluate the effectiveness of these metrics, we also conducted experiments on four programs with different numbers of MRs. The empirical results show that the metrics are correlated with the effectiveness of MRs and the previous MR selection guidelines.