Coverage-Based Reduction of Test Execution Time: Lessons from a Very Large Industrial Project

Abstract

There exist several coverage-based approaches to reduce time and resource costs of test execution. While these methods are well-investigated and evaluated for smaller to medium-size projects, we faced several challenges in applying them in the context of a very large industrial software project, namely SAP HANA. These issues include: varying effectiveness of algorithms for test case selection/prioritization, large amounts of shared (non-specific) coverage between different tests, high redundancy of coverage data, and randomness of test results (i.e. flaky tests), as well as of the coverage data (e.g. due to concurrency issues). We address these issues by several approaches. First, our study shows that compared to standard algorithms, so-called overlap-aware solvers can achieve up to 50% higher code coverage in a fixed time budget, significantly increasing the effectiveness of test case prioritization and selection. We also detected in our project high redundancy of line coverage data (up to 97%), providing opportunities for data size reduction. Finally, we show that removal of coverage shared by tests can significantly increase test specificity. Our analysis and approaches can help to narrow the gap between research and practice in context of coverage-based testing approaches, especially in case of very large software projects.