Flexible Combinatorial Interaction Testing

Abstract

We present Flexible Combinatorial Interaction Testing (F-CIT), which aims to improve the flexibility of combinatorial interaction testing (CIT) by eliminating the necessity of developing specialized constructors for CIT problems that cannot be efficiently and effectively addressed by the existing CIT constructors. F-CIT expresses the entities to be covered and the space of valid test cases, from which the samples are drawn to obtain full coverage, as constraints. Computing an F-CIT object (i.e., a set of test cases obtaining full coverage under a given coverage criterion) then turns into an interesting constraint solving problem, which we call <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">cov-CSP</i> . cov-CSP aims to divide the constraints, each representing an entity to be covered, into a minimum number of satisfiable clusters, such that a solution for a cluster represents a test case and the collection of all the test cases generated (one per cluster) constitutes an F-CIT object, covering each required entity at least once. To solve the cov-CSP problem, thus to compute F-CIT objects, we first present two constructors. One of these constructors attempts to cover as many entities as possible in a cluster before generating a test case, whereas the other constructor generates a test case first and then marks all the entities accommodated by this test case as covered. We then use these constructors to evaluate F-CIT in three studies, each of which addresses a different CIT problem. In the first study, we develop structure-based F-CIT objects to obtain decision coverage-adequate test suites. In the second study, we develop order-based F-CIT objects, which enhance a number of existing order-based coverage criteria by taking the reachability constraints imposed by graph-based models directly into account when computing interaction test suites. In the third study, we develop usage-based F-CIT objects to address the scenarios, in which standard covering arrays are not desirable due to their sizes, by choosing the entities to be covered based on their usage statistics collected from the field. We also carry out user studies to further evaluate F-CIT. The results of these studies suggest that F-CIT is more flexible than the existing CIT approaches.