Abstract Test Case Prioritization Using Repeated Small-Strength Level-Combination Coverage

Abstract

Abstract test cases (ATCs) have been widely used in practice, including in combinatorial testing and in software product line testing. When constructing a set of ATCs, due to limited testing resources in practice (e.g., in regression testing), test case prioritization (TCP) has been proposed to improve the testing quality, aiming at ordering test cases to increase the speed with which faults are detected. One intuitive and extensively studied TCP technique for ATCs is $\lambda$ -wise Level-combination Coverage based Prioritization ( $\lambda$ LCP), a static, black-box prioritization technique that only uses the ATC information to guide the prioritization process. A challenge facing $\lambda$ LCP, however, is the necessity for the selection of the fixed prioritization strength $\lambda$ before testing—testers need to choose an appropriate $\lambda$ value before testing begins. Choosing higher $\lambda$ values may improve the testing effectiveness of $\lambda$ LCP (e.g., by finding faults faster), but may reduce the testing efficiency (by incurring additional prioritization costs). Conversely, choosing lower $\lambda$ values may improve the efficiency, but may also reduce the effectiveness. In this paper, we propose a new family of $\lambda$ LCP techniques, Repeated Small-strength Level-combination Coverage-based Prioritization (RSLCP), that repeatedly achieves the full combination coverage at lower strengths. RSLCP maintains $\lambda$ LCP's advantages of being static and black box, but avoids the challenge of prioritization strength selection. We have performed an empirical study involving five different versions of each of five C programs. Compared with $\lambda$ LCP, and Incremental-strength LCP (ILCP), our results show that RSLCP could provide a good tradeoff between testing effectiveness and efficiency. Our results also show that RSLCP is more effective and efficient than two popular techniques of Similarity-based Prioritization (SP). In addition, the results of empirical studies also show that RSLCP can remain robust over multiple system releases.