Harnessing multiple source test cases in metamorphic testing: a case study in bioinformatics

Abstract

Metamorphic testing (MT) has been applied to software verification, validation and quality assessment. In most previous studies, research has focused on deriving metamorphic relations (MRs) such that the input of one or more follow-up test cases is generated from one source test case. We note that some programs under test (PUT) naturally take multiple inputs and process them simultaneously to generate multiple outputs. This type of programs are common in the field of big data analysis and bioinformatics. This means, in the source execution of the program, we can obtain multiple outputs from multiple source test cases. Here we consider a type of MR in which multiple follow-up test cases are generated from multiple source test cases simultaneously. We hypothesise that harnessing the outputs from multiple source test cases enables us to obtain additional information about the PUT, and therefore allows us to construct more effective MRs. In this paper, we designed a new MR to test a popular RNA sequence alignment program. Since the MR was designed based on a desirable property of the PUT (which uses a complex heuristic algorithm) rather than a necessary property, violation of this MR indicates the program outputs deviate from user expectation, hence this MR can be used for assessing the quality of the outputs. Furthermore, we note that outputs of the follow-up test cases allow us to putatively assign alignment information to some sequences that were not aligned in the source test cases, hence potentially identifying and correcting these 'false negative' outputs. We believe this case study provides important insight into designing MRs based on multiple source test cases, and how testing results can be used to improve the performance of some heuristic-based programs.