Testing and Debugging in Continuous Integration with Budget Quotas on Test Executions

Abstract

In Continuous Integration, a software application is developed through a series of development sessions, each with limited time allocated to testing and debugging on each of its modules. Test Case Prioritization can help execute test cases with higher failure estimate earlier in each session. When the testing time is limited, executing such prioritized test cases may only produce partial and prioritized execution coverage data. To identify faulty code, existing Spectrum-Based Fault Localization techniques often use execution coverage data but without the assumption of execution coverage priority. Is it possible to decompose these two steps for optimization within individual steps? In this paper, we study to what extent the selection of test case prioritization techniques may reduce its influence on the effectiveness of spectrum-based fault localization, thereby showing the possibility to decompose the process of continuous integration for optimization in workflow steps. We present a controlled experiment using the Siemens suite as subjects, nine test case prioritization techniques and four spectrum-based fault localization techniques. The findings showed that the studied test cases prioritization and spectrum-based fault localization can be customized separately, and, interestingly, prioritization over a smaller test suite can enable spectrum-based fault localization to achieve higher accuracy by assigning faulty statements with higher ranks.