Abstract
Constraints depict the dependency relationships between parameters in a software system under test. Because almost all systems are constrained in some way, techniques that adequately cater for constraints have become a crucial factor for adoption, deployment and exploitation of Combinatorial Testing (CT). Currently, despite a variety of different constraint handling techniques available, the relationship between these techniques and the generation algorithms that use them remains unknown, yielding an important gap and pressing concern in the literature of constrained combination testing. In this article, we present a comparative empirical study to investigate the impact of four common constraint handling techniques on the efficiency of six representative (greedy and search-based) test suite generation algorithms. The results reveal that the <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">Verify</i> technique implemented with the Minimal Forbidden Tuple (MFT) approach is the fastest, while the <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">Replace</i> technique is promising for producing the smallest constrained covering arrays, especially for algorithms that construct test cases one-at-a-time. The results also show that there is an interplay between efficiency of the constraint handler and the test suite generation algorithm into which it is developed.
Highlights
C OMBINATORIAL TESTING (CT), or combinatorial interaction testing (CIT), is a potentially powerful testing technique for revealing failures triggered by interactions of parameters that govern software system execution behaviour [1]
We provide a comparative analysis that investigates the impact of four common constraint handling techniques (Verify, Solver, Tolerate, and Replace) on six widely used greedy and search-based combinatorial test suite generation algorithms (AETG, DDA, IPO, PSO, SA, and TS)
Our experimental results reveal that the constraint handler is, a crucial factor that influences the performance of the test suite generation algorithms into which it is developed
Summary
C OMBINATORIAL TESTING (CT), or combinatorial interaction testing (CIT), is a potentially powerful testing technique for revealing failures triggered by interactions of parameters that govern software system execution behaviour [1]. Since recent studies [10], [11] have revealed a diverse set of different constraint handling techniques, a subsequent natural question is the extent to which these techniques make a difference in the performance of different test suite generation algorithms This is important because test suite generation studies of CT usually follow (arbitrary choices of) one of the previous practices to determine the constraint handler to be used (for example, many recent studies directly use the SAT solver approach without further examination [17], [18], [19], [20], [21]), which may potentially restrict the performance of the proposed algorithms.
Sign-in/Register to view highlights & summary Sign-in/Register to access full text options 
Free) 
Talk to us
Join us for a 30 min session where you can share your feedback and ask us any queries you have
Schedule a call
