Context-Aware Regression Test Selection

Abstract

Most modern software systems are continuously evolving, with changes frequently taking place in the core components or the execution context. These changes can adversely introduce regression faults, causing previously working functions to fail. Regression testing is essential for maintaining the quality of evolving complex software, but it can be overly time-consuming when the size of the test suite is large, or the execution of the test cases takes a long time. There are extensive research studies on selective regression testing aiming at minimizing the size of the regression test suite while maximizing the detection of the regression faults. However, most of the existing techniques focus on the regression faults caused by the code changes, the impact of the context changes on the non-modified software has barely been explored. This paper presents a context-aware regression test selection (CARTS) approach that not only accounts for the modification of code but also changes in the execution context, including libraries, external APIs, and databases. After a change, CARTS uses the program invariants denoted in the pre- and postconditions of a function to determine if the function is affected by the change and selects all the test cases that executed the modified code as well as the non-modified functions whose preconditions are affected by the change. To evaluate the effectiveness of our approach, we conducted empirical studies on multi-release open-source software and case studies on real-world systems that have ongoing changes in code as well as in the execution context. The results of our controlled experiments show that with an average of 32.5% of the regression test cases, CARTS selected all the fault-revealing test cases. In the case studies, all the fault-revealing test cases were selected by using an average of 25.3% of the regression test suite. These results suggest that CARTS can be effective for selecting fault-revealing test cases for both code and execution context changes.