Semantic‐aware two‐phase test case prioritization for continuous integration

Abstract

SummaryContinuous integration (CI) is a widely applied development practice to allow frequent integration of software changes, detecting early faults. However, extremely frequent builds consume amounts of time and resources in such a scenario. It is quite challenging for existing test case prioritization (TCP) to address this issue due to the time‐consuming information collection (e.g. test coverage) or inaccurately modelling code semantics to result in the unsatisfied prioritization. In this paper, we propose a semantic‐aware two‐phase TCP framework, named SatTCP, which combines the coarse‐grained filtering and fine‐grained prioritization to perform the precise TCP with low time costs for CI. It consists of three parts: (1) code representation, parsing the programme changes and test cases to obtain the code change and test case representations; (2) coarse‐grained filtering, conducting the preliminary ranking and filtering of test cases based on information retrieval; and (3) fine‐grained prioritization, training a pretrained Siamese language model based on the filtered test set to further sort the test cases via semantic similarity. We evaluate SatTCP on a large‐scale, real‐world dataset with cross‐project validation from fault detection efficiency and time costs and compare it with five baselines. The results show that SatTCP outperforms all baselines by 6.3%–45.6% for mean average percentage of fault detected per cost (APFDc), representing an obvious upward trend as the project scale increases. Meanwhile, SatTCP can reduce the real CI testing by 71.4%, outperforming the best baseline by 17.2% for time costs on average. Furthermore, we discuss the impact of different configurations, flaky tests and hybrid techniques on the performance of SatTCP, respectively.