Colosseum: Regression Test Prioritization by Delta Displacement in Test Coverage

Abstract

The problem of test-case prioritization has been pursued for over three decades now and continues to be one of the active topics in software testing research. In this paper, we focus on a code-coverage based regression test-prioritization solution ( <inline-formula><tex-math notation="LaTeX">${{\sf Colosseum}}$</tex-math></inline-formula> ) that takes into account the position of changed (delta) code elements (basic-blocks) along the loop-free straight-line execution path of the regression test-cases. We propose a heuristic that logically associates each of these paths with three parameters: (i) the offset (displacement <i>a</i> ) of the first delta from the starting basic-block, (ii) the offset (displacement <i>c</i> ) of the last delta from the terminating basic block, and (iii) the average scattering (displacement <i>b</i> ) within all the intermediate basic-blocks. We hypothesize that a regression test-case path with a shorter overall displacement has a good chance of propagating the affects of the code-changes to the observable outputs in the program. <inline-formula><tex-math notation="LaTeX">${{\sf Colosseum}}$</tex-math></inline-formula> prioritizes test-cases with smaller overall displacements and executes them early in the regression test-execution cycle. The underlying intuition is that the probability of a test-case revealing a regression fault depends on the probability of the corresponding change propagation. The change in this context can potentially lead to an error. Extending this logic, delta displacement provides an approximation to failed error propagation. Evaluation on 20 open-source C projects from the Software-artifact Infrastructure Repository and GitHub (totaling: 694,512 SLOC, 280 versions, and 69,305 test-cases) against four state-of-the-art prioritizations reveals that: <inline-formula><tex-math notation="LaTeX">${{\sf Colosseum}}$</tex-math></inline-formula> outperforms the competitors with an overall 84.61% success in terms of 13 prioritization effectiveness metrics, majority of which prefer to execute top- <inline-formula><tex-math notation="LaTeX">$k\%$</tex-math></inline-formula> prioritized test-cases.