On the use of contextual information for machine learning based test case prioritization in continuous integration development

Abstract

Context:In most software organizations, Continuous Integration (CI) is a common practice usually subject to some budgets. Consequently, prioritizing test cases to be executed in the CI cycle is fundamental. The idea is first to execute test cases with higher failure-proneness to provide rapid feedback and decrease costs. To perform this task approaches in the literature adopt failure history and Machine Learning (ML). However, in addition to the failure history, it is also important to consider information from the CI context of the organizations and the application domain. Objective:For this end, we introduce a contextual information approach for ML algorithms. Such an approach considers information from the testing activity that can be easily collected, such as test case execution time, size, and complexity. We implement the approach by introducing two contextual versions of the algorithms: Multi-Armed Bandit (MAB) and Random Forest (RF). Method:Six systems are used to compare both contextual algorithms and to evaluate their performance regarding their corresponding non-contextual versions, considering three different budgets. Results:Contextual algorithms perform better when indicators related to test time reduction are considered, as the contextual information they use is related to execution time. Regarding NAPFD and APFDc, the non-contextual algorithms have better general performance, but both contextual versions obtain competitive results. Conclusions:The contextual versions implemented can capture the desired context information in the prioritization without negatively impacting their performance regarding fault-detection.