An Interleaving Guided Metamorphic Testing Approach for Concurrent Programs

Abstract

Concurrent programs are normally composed of multiple concurrent threads sharing memory space. These threads are often interleaved, which may lead to some non-determinism in execution results, even for the same program input. This poses huge challenges to the testing of concurrent programs, especially on the test result verification, that is, the prevalent existence of the oracle problem. In this paper, we investigate the application of metamorphic testing, a mainstream technique to address the oracle problem, into the testing of concurrent programs. Based on the unique features of interleaved executions in concurrent programming, we propose an extended notion of metamorphic relations, the core part of metamorphic testing, which are particularly designed for the testing of concurrent programs. A comprehensive testing approach, namely ConMT, is thus developed and a tool is built to automate its implementation on concurrent programs written in Java. Empirical studies have been conducted to evaluate the performance of ConMT, and the experimental results show that in addition to addressing the oracle problem, ConMT outperforms the baseline traditional testing techniques with respect to higher degree of automation, better bug-detection capability, and shorter testing time. It is clear that ConMT can significantly improve the cost-effectiveness for the testing of concurrent programs and thus advances the state of the art in the field. The study also brings novelty into metamorphic testing, hence promoting the fundamental research of software testing.