Test Suite Prioritization by Switching Cost

Abstract

Test suite generation and prioritization are two main research fields to improve testing efficiency. Combinatorial testing has been proven as an effective method to generate test suite for highly configurable software systems, while test suites are often prioritized by interaction coverage to detect faults as early as possible. However, for some cases, there exists reasonable cost of reconfiguring parameter settings when switching test cases in different orders. Surprisingly, only few studies paid attention to it. In this paper, by proposing greedy algorithms and graph-based algorithms, we aim to prioritize a given test suite to minimize its total switching cost. We also compare two different prioritization strategies by a series of experiments, and discuss the advantages of our prioritization strategy and the selection of prioritization techniques. The results show that prioritization by switching cost can improve testing efficiency and our prioritization strategy can produce a small test suite with a reasonably low switching cost. This prioritization can be used widely and help locate fault causing interactions. The results also suggest that when testing highly configurable software systems and no knowledge of fault detection can be used, prioritization by switching cost is a good choice to detect faults earlier.