Which builds are really safe to skip? Maximizing failure observation for build selection in continuous integration

Abstract

Continuous integration (CI) is a widely used practice in modern software engineering. Unfortunately, it is also an expensive practice. Google and Mozilla estimate their expenses for their CI systems in millions of dollars. To reduce the cost of CI, researchers developed multiple approaches to reduce its computational workload requirements. However, these approaches sometimes make mispredictions and skip failing builds which are not desirable to be skipped. Thus, in this paper, we aim to save computational cost in CI, while also maximizing the observation of failing builds, i.e., to skip builds more safely. First, we perform empirical studies to understand which builds are safe to skip, starting from CI-Skip rules (Abdalkareem et al., 2019) that characterize builds that developers decide to skip. We observe that CI-Skip rules are not so safe as expected. We then develop a collection of CI-Run rules that can complement these rules. Based on our findings, we propose PreciseBuildSkip, a novel approach that maximizes build failure observation and reduces the cost of CI through the strategy of build selection. We evaluate our approach and results show that our approach saved more cost (5.5%) than the safest existing technique but reduced the falsely skipped failing builds from 4.1% to 0% (median value).