How Do Assertions Impact Coverage-Based Test-Suite Reduction?

Abstract

Code coverage is the dominant criterion in test-suite reduction. Typically, most test-suite reduction techniques repeatedly remove tests covering code that has been covered by other tests from the test suite. However, test-suite reduction based on code coverage alone may incur fault-detection capability loss, because a test detects faults if and only if its execution covers buggy code and its test oracle catches the buggy state. In other words, test oracles may also affect test-suite reduction, However, to our knowledge, their impacts have never been studied before. In this paper, we conduct the first empirical study on such impacts by using 10 real-world GitHub Java projects, and find that assertions (i.e., a typical type of test oracles) are significantly correlated with coverage-based test-suite reduction. Based on our preliminary study results, we also proposed an assertion-aware test-suite reduction technique which outperforms traditional test-suite reduction in terms of cost-effectiveness.