Combinatorial Interaction Testing Research Articles

Pairwise test case generation with harmony search, one-parameter-at-at-time, seeding, and constraint mechanism integration

Pairwise testing is a method for identifying defects through combinatorial analysis. It involves testing all possible combinations of input parameters in pairs within a system, ensuring that each pair is tested at least once. The field of test case generation is highly active in the realm of combinatorial interaction testing. Research in this area is particularly encouraged, as it falls under the category of non-deterministic polynomial-time hardness. A big challenge in this field is the combinatorial explosion problem. It is about finding the best test suite that covers all possible combinations of interaction strength. In this paper, we present the task of discovering a pairwise test set as a search problem and introduce an innovative testing tool referred to as pairwise test case generation in harmony search algorithm with seeding and constraint mechanism (PHOSC). Experimental results show that PHOSC performs better compared to some existing pairwise strategies in terms of test suite size. Additionally, PHOSC provides a comprehensive framework and serves as a research platform for the generation of pairwise test sets employing the harmony search algorithm. It adopts an approach that focuses on one parameter at a time (OPAT) and incorporates seeding and constraint mechanisms at the same time, thereby enhancing the efficiency and effectiveness of the testing process.

Design, implementation, and validation of a benchmark generator for combinatorial interaction testing tools

Combinatorial testing is a widely adopted technique for efficiently detecting faults in software. The quality of combinatorial test generators plays a crucial role in achieving effective test coverage. Evaluating combinatorial test generators remains a challenging task that requires diverse and representative benchmarks. Having such benchmarks might help developers to test their tools, and improve their performance.For this reason, in this paper, we present BenCIGen, a highly configurable generator of benchmarks to be used by combinatorial test generators, empowering users to customize the type of benchmarks generated, including constraints and parameters, as well as their complexity. An initial version of such a tool has been used during the CT-Competition, held yearly during the International Workshop on Combinatorial Testing. This paper describes the requirements, the design, the implementation, and the validation of BenCIGen. Tests for the validation of BenCIGen are derived from its requirements by using a combinatorial interaction approach. Moreover, we demonstrate the tool’s ability to generate benchmarks that reflect the characteristics of real software systems.BenCIGen not only facilitates the evaluation of existing generators but also serves as a valuable resource for researchers and practitioners seeking to enhance the quality and effectiveness of combinatorial testing methodologies.

Software Product Line Maintenance Using Multi-Objective Optimization Techniques

Currently, software development is more associated with families of configurable software than the single implementation of a product. Due to the numerous possible combinations in a software product line, testing these families of software product lines (SPLs) is a difficult undertaking. Moreover, the presence of optional features makes the testing of SPLs impractical. Several features are presented in SPLs, but due to the environment’s time and financial constraints, these features are rendered unfeasible. Thus, testing subsets of configured products is one approach to solving this issue. To reduce the testing effort and obtain better results, alternative methods for testing SPLs are required, such as the combinatorial interaction testing (CIT) technique. Unfortunately, the CIT method produces unscalable solutions for large SPLs with excessive constraints. The CIT method costs more because of feature combinations. The optimization of the various conflicting testing objectives, such as reducing the cost and configuration number, should also be considered. In this article, we proposed a search-based software engineering solution using multi-objective evolutionary algorithms (MOEAs). In particular, the research was applied to different types of MOEA method: the Indicator-Based Evolutionary Algorithm (IBEA), Multi-objective Evolutionary Algorithm based on Decomposition (MOEA/D), Non-dominant Sorting Genetic Algorithm II (NSGAII), NSGAIII, and Strength Pareto Evolutionary Algorithm 2 (SPEA2). The results of the algorithms were examined in the context of distinct objectives and two quality indicators. The results revealed how the feature model attributes, implementation context, and number of objectives affected the performances of the algorithms.

Input Distribution Coverage: Measuring Feature Interaction Adequacy in Neural Network Testing

Testing deep neural networks (DNNs) has garnered great interest in the recent years due to their use in many applications. Black-box test adequacy measures are useful for guiding the testing process in covering the input domain. However, the absence of input specifications makes it challenging to apply black-box test adequacy measures in DNN testing. The Input Distribution Coverage (IDC) framework addresses this challenge by using a variational autoencoder to learn a low dimensional latent representation of the input distribution, and then using that latent space as a coverage domain for testing. IDC applies combinatorial interaction testing on a partitioning of the latent space to measure test adequacy. Empirical evaluation demonstrates that IDC is cost-effective, capable of detecting feature diversity in test inputs, and more sensitive than prior work to test inputs generated using different DNN test generation methods. The findings demonstrate that IDC overcomes several limitations of white-box DNN coverage approaches by discounting coverage from unrealistic inputs and enabling the calculation of test adequacy metrics that capture the feature diversity present in the input space of DNNs.

Constrained detecting arrays: Mathematical structures for fault identification in combinatorial interaction testing

Detecting arrays are mathematical structures aimed at fault identification in combinatorial interaction testing. However, they cannot be directly applied to systems that have constraints on the test parameters. These constraints are prevalent in real-world systems. This paper proposes constrained detecting arrays (CDAs), an extension of detecting arrays, which can be used for systems with constraints. The properties and capabilities of CDAs are examined with rigorous arguments. Moreover, two algorithms are proposed for constructing CDAs: one is aimed at generating minimum CDAs, and the other is a heuristic algorithm aimed at fast generation of CDAs. The algorithms were experimentally evaluated using a benchmark dataset. Experimental results show that the first algorithm can generate minimum CDAs if a sufficiently long generation time is allowed, and the second algorithm can generate minimum or near-minimum CDAs in a reasonable time. CDAs extend the range of application of detecting arrays to systems with constraints. The two proposed algorithms have different advantages with respect to array size and generation time. • Constrained detecting arrays (CDAs) are proposed for identification of faulty interactions in combinatorial testing. • CDAs extend detecting arrays to deal with systems that have constraints on test parameters. • Two algorithms for constructing CDAs are proposed with experimental results using benchmark problems.

Test scenario generation for feature-based context-oriented software systems

Feature-based context-oriented programming reconciles ideas from context-oriented programming, feature modelling and dynamic software product lines. It offers a programming language, architecture, tools and methodology to develop software systems consisting of contexts and features that can become active at run-time to offer the most appropriate behaviour depending on the actual context of use. Due to their high run-time adaptivity, dedicated tool support to test such systems is needed. Building upon a pairwise combinatorial interaction testing approach from the domain of software product lines, we implement an algorithm to generate automatically a small set of relevant test scenarios, ordered to minimise the number of context activations between tests. We also explore how the generated scenarios can be enhanced incrementally when the software evolves, and how useful the proposed testing approach is in practice.

CIT-daily: A combinatorial interaction testing-based daily build process

In this work, we introduce an approach, called CIT-daily, which integrates combinatorial interaction testing (CIT) with the daily build processes to systematically test the interactions between the factors/parameters affecting the system’s behaviors, on a daily basis. We also develop a number of CIT-daily strategies and empirically evaluate them on highly-configurable systems. The first strategy tests the same t-way covering array every day throughout the process, achieving a t-way coverage on a daily basis by covering each possible combination of option settings for every combination of t options. The other strategies, on the other hand, while guaranteeing a t-way coverage on a daily basis, aim to cover higher order interactions between the configuration options over time by varying the t-way covering arrays tested. In the experiments, we observed that the proposed approach significantly improved the effectiveness (i.e., fault revealing abilities) of the daily build processes; randomizing the coverage of higher order interactions between the configuration options while guaranteeing a base t-way coverage every day, further improved the effectiveness; and the more the higher order interactions covered during the process, the higher the fault revealing abilities tended to be.

Accelerating covering array generation by combinatorial join for industry scale software testing.

Combinatorial interaction testing, which is a technique to verify a system with numerous input parameters, employs a mathematical object called a covering array as a test input. This technique generates a limited number of test cases while guaranteeing a given combinatorial coverage. Although this area has been studied extensively, handling constraints among input parameters remains a major challenge, which may significantly increase the cost to generate covering arrays. In this work, we propose a mathematical operation, called “weaken-product based combinatorial join”, which constructs a new covering array from two existing covering arrays. The operation reuses existing covering arrays to save computational resource by increasing parallelism during generation without losing combinatorial coverage of the original arrays. Our proposed method significantly reduce the covering array generation time by 13–96% depending on use case scenarios.

Improved Mixed Neighborhood Tabu Search by Random Selection for Combinatorial Interaction Testing

Improved Mixed Neighborhood Tabu Search by Random Selection for Combinatorial Interaction Testing

Constrained locating arrays for combinatorial interaction testing

This paper introduces the notion of Constrained Locating Arrays (CLAs), mathematical objects which can be used for fault localization in software testing. CLAs extend ordinary locating arrays to make them applicable to testing of systems that have constraints on test parameters. Such constraints are common in real-world systems; thus CLA enhances the applicability of locating arrays to practical testing problems. The paper also proposes an algorithm for constructing CLAs. Experimental results show that the proposed algorithm scales to problems of practical sizes.

Flexible Combinatorial Interaction Testing

We present Flexible Combinatorial Interaction Testing (F-CIT), which aims to improve the flexibility of combinatorial interaction testing (CIT) by eliminating the necessity of developing specialized constructors for CIT problems that cannot be efficiently and effectively addressed by the existing CIT constructors. F-CIT expresses the entities to be covered and the space of valid test cases, from which the samples are drawn to obtain full coverage, as constraints. Computing an F-CIT object (i.e., a set of test cases obtaining full coverage under a given coverage criterion) then turns into an interesting constraint solving problem, which we call <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">cov-CSP</i> . cov-CSP aims to divide the constraints, each representing an entity to be covered, into a minimum number of satisfiable clusters, such that a solution for a cluster represents a test case and the collection of all the test cases generated (one per cluster) constitutes an F-CIT object, covering each required entity at least once. To solve the cov-CSP problem, thus to compute F-CIT objects, we first present two constructors. One of these constructors attempts to cover as many entities as possible in a cluster before generating a test case, whereas the other constructor generates a test case first and then marks all the entities accommodated by this test case as covered. We then use these constructors to evaluate F-CIT in three studies, each of which addresses a different CIT problem. In the first study, we develop structure-based F-CIT objects to obtain decision coverage-adequate test suites. In the second study, we develop order-based F-CIT objects, which enhance a number of existing order-based coverage criteria by taking the reachability constraints imposed by graph-based models directly into account when computing interaction test suites. In the third study, we develop usage-based F-CIT objects to address the scenarios, in which standard covering arrays are not desirable due to their sizes, by choosing the entities to be covered based on their usage statistics collected from the field. We also carry out user studies to further evaluate F-CIT. The results of these studies suggest that F-CIT is more flexible than the existing CIT approaches.

Using simulated annealing for locating array construction

Context: Combinatorial interaction testing is known to be an efficient testing strategy for computing and information systems. Locating arrays are mathematical objects that are useful for this testing strategy, as they can be used as a test suite that enables fault localization as well as fault detection. In this application, each row of an array is used as an individual test. Objective: This paper proposes an algorithm for constructing locating arrays with a small number of rows. Testing cost increases as the number of tests increases; thus the problem of finding locating arrays of small sizes is of practical importance. Method: The proposed algorithm uses simulation annealing, a meta-heuristic algorithm, to find locating array of a given size. The whole algorithm repeatedly executes the simulated annealing algorithm by dynamically varying the input array size. Results: Experimental results show 1) that the proposed algorithm is able to construct locating arrays for problem instances of large sizes and 2) that, for problem instances for which nontrivial locating arrays are known, the algorithm is often able to generate locating arrays that are smaller than or at least equal to the known arrays. Conclusion: Based on the results, it is concluded that the proposed algorithm can produce small locating arrays and scale to practical problems.

Finding Minimum Locating Arrays Using a CSP Solver

Combinatorial interaction testing is an efficient software testing strategy. If all interactions among test parameters or factors needed to be covered, the size of a required test suite would be prohibitively large. In contrast, this strategy only requires covering $t$-wise interactions where $t$ is typically very small. As a result, it becomes possible to significantly reduce test suite size. Locating arrays aim to enhance the ability of combinatorial interaction testing. In particular, $(\overline{1}, t)$-locating arrays can not only execute all $t$-way interactions but also identify, if any, which of the interactions causes a failure. In spite of this useful property, there is only limited research either on how to generate locating arrays or on their minimum sizes. In this paper, we propose an approach to generating minimum locating arrays. In the approach, the problem of finding a locating array consisting of $N$ tests is represented as a Constraint Satisfaction Problem (CSP) instance, which is in turn solved by a modern CSP solver. The results of using the proposed approach reveal many $(\overline{1}, t)$-locating arrays that are smallest known so far. In addition, some of these arrays are proved to be minimum.

WCA: A weighting local search for constrained combinatorial test optimization

Abstract Context Covering array generation (CAG) is the core task of Combinatorial interaction testing (CIT), which is widely used to discover interaction faults in real-world systems. Considering the universality, constrained covering array generation (CCAG) is more in line with the characteristics of applications, and has attracted a lot of researches in the past few years. Objective In CIT, a covering array (CA) with smaller size means lower cost of testing, particularly for the systems where the execution of a test suite is time consuming. As constraints between parameters are ubiquitous in real systems, this work is dedicated to more efficient algorithms for CCAG. Specifically, we aim to develop a heuristic search algorithm for CCAG, which allows generating CAs with smaller size in a limited time when compared with existing algorithms. Method We propose a weighting local search algorithm named WCA, which makes use of weights associated with the tuples and dynamically adjusts them during the search, helping the algorithm to avoid search stagnation. As far as we know, this is the first weighting local search for solving CCAG. Results We apply WCA to a wide range of benchmarks, including real-world ones and synthetic ones. The results show that WCA achieves a significant improvement over three state-of-the-art competitors in 2-way and 3-way CCAG, in terms of both effectiveness and efficiency. The importance of weighting is also reflected by the experimental comparison between WCA and its alternative algorithm without the weighting mechanism. Conclusion WCA is an effective heuristic algorithm for CCAG to obtain smaller CAs efficiently, and the weighting mechanism plays a crucial role.

Implementation of Sine Cosine Algorithm (SCA) for Combinatorial Testing

Before being released to the market, software should be screened to ensure that the quality assurance measurement goals have been attained. To achieve this, one of the types of testing sorts is combinatorial testing (CT) aimed at discovering the faults that occur by interacting with the software. A minimization strategy for test cases is indeed important for optimizing test cases and reducing time. As NP hard (where NP is a non-deterministic polynomial) is the problem of generating the minimum test suite of combinatorial interaction testing (CIT). this paper discusses the implementation, and validation of an efficient strategy for t-way testing. The main contribution of the sine cosine algorithm SCA is to show that the strategy was sufficiently competitive as compared to other strategies in terms of the generated test suite size. Unlike most paper. The main contribution of SCA is to show the generation of test data for a high coverage strength (t < 12).

Combinatorial Interaction Testing For Improving Conformance Faults

In this paper we are describing about how to develop an interface for compiling of any java program and will show how many errors in the program. We will also perform some metrics like number of lines time taken to write the program, operands used in the program. And also find volume effort required to implement or understand the program. For this we are using Combinatorial Interaction testing (CIT), is a productive and viable strategy for identifying disappointments that are brought about by the collaborations of different framework input parameters. In this paper, we talk about CIT, call attention to a portion of challenges for applying it practically, and feature some ongoing advances that have improved CIT's immaterialness to present day frameworks. We additionally give a guide to future research and bearings one that we expectation will prompt new CIT look into and to more excellent testing of mechanical frameworks.

Big data driven genetic improvement for maintenance of legacy software systems

Big data driven genetic improvement for maintenance of legacy software systems

Code‐aware combinatorial interaction testing

Combinatorial interaction testing (CIT) is a useful testing technique to address the interaction of input parameters in software systems. CIT has been used as a systematic technique to sample the enormous test possibilities. Most of the research activities focused on the generation of CIT test suites as a computationally complex problem. Less effort has been paid for the application of CIT. To apply CIT, practitioners must identify the input parameters for the Software-under-test (SUT), feed these parameters to the CIT test generation tool, and then run those tests on the application with some pass and fail criteria for verification. Using this approach, CIT is used as a black-box testing technique without knowing the effect of the internal code. Although useful, practically, not all the parameters having the same impact on the SUT. This paper introduces a different approach to use the CIT as a gray-box testing technique by considering the internal code structure of the SUT to know the impact of each input parameter and thus use this impact in the test generation stage. The case studies results showed that this approach would help to detect new faults as compared to the equal impact parameter approach.

Sampling strategies for product lines with unbounded parametric real-time constraints

Combinatorial interaction testing (CIT) has been successfully applied to product-line testing for selecting from a usually very large configuration space a relatively small sample of test configurations sufficiently covering critical combinations of configuration options. As most recent CIT techniques like pairwise sampling are limited to finite configuration spaces (i.e., vectors of yes/no options), they are not directly applicable to configuration parameters with a priori unbounded value domains (e.g., for adjusting unlimited resources or non-functional properties). Applying existing sampling strategies to infinite configuration spaces therefore requires further heuristics for selecting a finite subset of parameter-value combinations to be covered. Nevertheless, applying purely black-box heuristics may produce inherently ineffective test suites in which particularly critical parameter-value combinations are missed. In order to tackle this problem, we present a novel methodology for effectively sampling product lines with infinite configuration spaces by means of freely configurable real-time behaviors. To this end, we employ solution-space information obtained from our new modeling formalism, configurable parametric timed automata, to generate samples for covering critical best-case/worst-case execution-time behaviors. We also present a tool implementation which we applied to a collection of subject systems to demonstrate the applicability of our approach.

Arrays for combinatorial interaction testing: a review on constructive approaches

Interaction faults in component-based complex systems have been investigated from various aspects in the past few decades. The key issue is that, in a complex system, not only a single variable may cause the failure, but also the interactions among different variables. Combinatorial interaction testing has been established as one of the most fundamental and effective tools for identifying and locating the faulty interactions. This paper presents a review of the brief history, essential problems, and significant results on arrays for combinatorial interaction testing, including covering arrays, locating arrays, detecting arrays, and their variations. Specific approaches on explicit constructions for covering arrays involving information-theoretic methods are reviewed. Finally, we propose two improved bounds for studying the optimality of locating arrays.