Combining minimization and generation for combinatorial testing

Abstract

Combinatorial Test Design (CTD) is an effective test planning technique that reveals faults resulting from feature interactions in a system. The standard application of CTD requires manual modeling of the test space, including a precise definition of restrictions between the test space parameters, and produces a test suite that corresponds to new test cases to be implemented from scratch. Interaction-based Test-Suite Minimization (ITSM) is a complementary approach to standard CTD, which reduces a given test suite without impacting its coverage of feature interactions. ITSM requires much less modeling effort, and does not require a definition of restrictions or generation of new test data. On the other hand, it does not improve the coverage obtained by the given test suite. In this work, we introduce Minimization Generation CTD (MG-CTD). MG-CTD is a combination of CTD with ITSM for addressing situations in which CTD is impractical, and ITSM is insufficient. In MG-CTD, one can define a subset of the parameters that can be freely assigned, as in CTD. The other parameter combinations must be selected from an existing set, as in ITSM. MG-CTD is suitable when for some parts of the test space it is easy to specify restrictions and generate new test data, while for others it is not. MG-CTD can be viewed as an enhancement of ITSM, and always achieves better interaction coverage than ITSM. We discuss the trade-offs between CTD, ITSM and MG-CTD, and present an efficient implementation which is based on binary decision diagrams. We then present some of the measures that one should take when implementing such an approach, in order to achieve the best possible coverage in the final result. Finally, we demonstrate MG-CTD on three real-life case studies.