Combinatorial Interaction Testing Research Articles

Sampling strategies for product lines with unbounded parametric real-time constraints

Combinatorial interaction testing (CIT) has been successfully applied to product-line testing for selecting from a usually very large configuration space a relatively small sample of test configurations sufficiently covering critical combinations of configuration options. As most recent CIT techniques like pairwise sampling are limited to finite configuration spaces (i.e., vectors of yes/no options), they are not directly applicable to configuration parameters with a priori unbounded value domains (e.g., for adjusting unlimited resources or non-functional properties). Applying existing sampling strategies to infinite configuration spaces therefore requires further heuristics for selecting a finite subset of parameter-value combinations to be covered. Nevertheless, applying purely black-box heuristics may produce inherently ineffective test suites in which particularly critical parameter-value combinations are missed. In order to tackle this problem, we present a novel methodology for effectively sampling product lines with infinite configuration spaces by means of freely configurable real-time behaviors. To this end, we employ solution-space information obtained from our new modeling formalism, configurable parametric timed automata, to generate samples for covering critical best-case/worst-case execution-time behaviors. We also present a tool implementation which we applied to a collection of subject systems to demonstrate the applicability of our approach.

Arrays for combinatorial interaction testing: a review on constructive approaches

Interaction faults in component-based complex systems have been investigated from various aspects in the past few decades. The key issue is that, in a complex system, not only a single variable may cause the failure, but also the interactions among different variables. Combinatorial interaction testing has been established as one of the most fundamental and effective tools for identifying and locating the faulty interactions. This paper presents a review of the brief history, essential problems, and significant results on arrays for combinatorial interaction testing, including covering arrays, locating arrays, detecting arrays, and their variations. Specific approaches on explicit constructions for covering arrays involving information-theoretic methods are reviewed. Finally, we propose two improved bounds for studying the optimality of locating arrays.

Prioritising abstract test cases: an empirical study

Test-case prioritisation (TCP) attempts to schedule the order of test-case execution such that faults can be detected as quickly as possible. TCP has been widely applied in many testing scenarios such as regression testing and fault localisation. Abstract test cases (ATCs) are derived from models of the system under test and have been applied to many testing environments such as model-based testing and combinatorial interaction testing. Although various empirical and analytical comparisons for some ATC prioritisation (ATCP) techniques have been conducted, to the best of the authors’ knowledge, no comparative study focusing on the most current techniques has yet been reported. In this study, they investigated 18 ATCP techniques, categorised into four classes. They conducted a comprehensive empirical study to compare 16 of the 18 ATCP techniques in terms of their testing effectiveness and efficiency. They found that different ATCP techniques could be cost-effective in different testing scenarios, allowing us to present recommendations and guidelines for which techniques to use under what conditions.

Optimization Driven Constraints Handling in Combinatorial Interaction Testing

The combinatorial strategy is useful in the reduction of the number of input parameters into a compact set of a system based on the combinations of the parameters. This strategy can be used in testing the behaviour that takes place when the events are allowed to be executed in an appropriate order. Basically, in the software systems, for the highly configurable system, the input configurations are based on the constraints, and the construction of this idea undergoes various kinds of difficulties. The proposed Jaya-Bat optimization algorithm is developed with the combinatorial interaction test cases in an effective manner in the presence of the constraints. The proposed Jaya-Bat based optimization algorithm is the integration of the Jaya optimization algorithm (JOA) and the Bat optimization algorithm (BA). The experimentation is carried out in terms of average size and the average time to prove the effectiveness of the proposed algorithm. From the results, it is clear that the proposed algorithm is capable of selecting the test cases optimally with better performance.

Constraints handling in combinatorial interaction testing using multi-objective crow search and fruitfly optimization

Combinatorial testing strategies are the recent interest of the researchers because of their wide variety of applications. The combinatorial testing strategy posses a great deal of minimizing the count of the input parameters of a system such that a small set of parameters is obtained depending on their interaction. Practically, the input models of the software system are subjected to the constraints mainly in highly configurable systems. There exist a number of issues while integrating the constraint in the testing strategy that is overcome using the proposed method. The proposed method aims at developing the combinatorial interaction test suites in the presence of constraints. The proposed strategy is multi-objective crow search and fruitfly optimization that is developed by the integration of the crow search algorithm and the chaotic fruitfly optimization algorithm. The proposed algorithm offers an optimal selection of the test suites at the better convergence. The experimentation based on the constraints and the analysis are carried out in terms of average size and average time with their values as 10 and 30 s, respectively.

A greedy algorithm to construct covering arrays using a graph representation

Combinatorial interaction testing is an effective method to assess the functionality of hardware and software components, and usually, covering arrays (CAs) and mixed covering arrays (MCAs) are used as test-suites. The main contributions of this work are the development of a graph representation for the problem of constructing CAs and MCAs, and the creation of a competitive greedy algorithm to solve the construction problem of CAs and MCAs in the graph domain. The representation that we introduce in this paper is denoted as Coverage in Nodes (CN) and our algorithm is called Graph Based Greedy Algorithm (GBGA).We have compared the performance of GBGA with other state-of-the-art greedy algorithm regarding the construction of CAs and MCAs. It is remarkable to note that we were able to define 13 new upper bounds for MCAs.

Enhancing Similarity Distances Using Mandatory and Optional for Early Fault Detection

Software Product Line (SPL) describes procedures, techniques, and tools in software engineering by using a common method of production for producing a group of software systems that identical from a shared set of software assets. In SPL, the similarity-based prioritization can resemble combinatorial interaction testing in scalable and efficient way by choosing and prioritize configurations that most dissimilar. However, the similarity distances in SPL still not so much cover the basic detail of feature models which are the notations. Plus, the configurations always have been prioritized based on domain knowledge but not much attention has been paid to feature model notations. In this paper, we proposed the usage of mandatory and optional notations for similarity distances. The objective is to improve the average percentage of faults detected (APFD). We investigate four different distances and make modifications on the distances to increase APFD value. These modifications are the inclusion of mandatory and optional notations with the similarity distances. The results are the APFD values for all the similarity distances including the original and modified similarity distances. Overall, the results shown that by subtracting the optional notation value can increase the APFD by 3.71% from the original similarity distance.

An Empirical Comparison of Fixed-Strength and Mixed-Strength for Interaction Coverage Based Prioritization

Test case prioritization (TCP) plays an important role in identifying, characterizing, diagnosing, and correcting faults quickly. The TCP has been widely used to order test cases of different types, including model inputs (also called abstract test cases ). Model inputs are constructed by modeling the program according to its input parameters, values, and constraints, and has been used in different testing methods, such as combinatorial interaction testing and software product line testing. The Interaction coverage-based TCP (ICTCP) uses interaction coverage information derived from the model input to order inputs. Previous studies have focused generally on the fixed-strength ICTCP, which adopts a fixed strength ( i.e. , the level of parameter interactions) to support the ICTCP process. It is generally accepted that using more strengths for ICTCP, i.e. , mixed-strength ICTCP, may give better ordering than fixed-strength. To confirm whether mixed-strength is better than fixed-strength, in this paper, we report on an extensive empirical study using five real-world programs (written in C), each of which has six versions. The results of the empirical studies show that mixed-strength has better rates of interaction coverage overall than fixed-strength, but they have very similar rates of fault detection. Our results also show that fixed-strength should be used instead of the mixed-strength at the later stage of software testing. Finally, we offer some practical guidelines for testers when using interaction coverage information to prioritize model inputs, under different testing scenarios and resources.

Automation of Combinatorial Interaction Test (CIT) Case Generation and Execution for Requirements based Testing (RBT) of Complex Avionics Systems

In the field of avionics, most of the software systems are either safety critical or mission critical. These systems are developed with high quality standards strictly following the relevant guidelines and procedures. Due to the high criticality of the systems, it is mandatory that the verification and validation of these systems are done with utmost importance and only then any system is cleared for flight trials. The verification and validation activities need to be very exhaustive and hence take a considerable amount of time in the software development lifecycle. This paper describes about the innovative approach towards automation of Combinatorial Interaction Test case generation and execution for Requirements Based Testing of complex avionics systems for achieving test adequacy in a highly time efficient and cost efficient manner.

An algorithm for combinatorial interaction testing: definitions and rigorous evaluations

BackgroundCombinatorial Interaction Testing (CIT) approaches have drawn attention of the software testing community to generate sets of smaller, efficient, and effective test cases where they have been successful in detecting faults due to the interaction of several input parameters. Recent empirical studies show that greedy algorithms are still competitive for CIT. It is thus interesting to investigate new approaches to address CIT test case generation via greedy solutions and to perform rigorous evaluations within the greedy context.MethodsWe present a new greedy algorithm for unconstrained CIT, T-TupleReallocation (TTR), to generate CIT test suites specifically via the Mixed-value Covering Array (MCA) technique. The main reasoning behind TTR is to generate an MCA M by creating and reallocating t-tuples into this matrix M, considering a variable called goal (ζ). We performed two controlled experiments addressing cost-efficiency and only cost. Considering both experiments, we did 3200 executions related to 8 solutions. In the first controlled experiment, we compared versions 1.1 and 1.2 of TTR in order to check whether there is significant difference between both versions of our algorithm. In such experiment, we jointly considered cost (size of test suites) and efficiency (time to generate the test suites) in a multi-objective perspective. In the second controlled experiment we confronted TTR 1.2 with five other greedy algorithms/tools for unconstrained CIT: IPOG-F, jenny, IPO-TConfig, PICT, and ACTS. We performed two different evaluations within this second experiment where in the first one we addressed cost-efficiency (multi-objective) and in the second only cost (single objective).ResultsResults of the first controlled experiment indicate that TTR 1.2 is more adequate than TTR 1.1 especially for higher strengths (5, 6). In the second controlled experiment, TTR 1.2 also presents better performance for higher strengths (5, 6) where only in one case it is not superior (in the comparison with IPOG-F). We can explain this better performance of TTR 1.2 due to the fact that it no longer generates, at the beginning, the matrix of t-tuples but rather the algorithm works on a t-tuple by t-tuple creation and reallocation into M.ConclusionConsidering the metrics we defined in this work and based on both controlled experiments, TTR 1.2 is a better option if we need to consider higher strengths (5, 6). For lower strengths, other solutions, like IPOG-F, may be better alternatives.

Evaluation of Software Product Line Test Case Prioritization Technique

Software product line (SPL) engineering paradigm is commonly used to handle commonalities and variabilities of business applications to satisfy the specific needs or goal of a particular market. However, due to time and space complexities, testing all products is not feasible and SPL testing proven to be difficult due to combinatorial explosion of the number of products to be considered. Combinatorial interaction testing (CIT) is suggested to reduce the size of test suites to overcome budget limitations and deadlines. CIT is conducted to fulfill certain quality attributes. This method can be further improvised through the prioritization of list configuration generated from CIT to gain better results in terms of efficiency and scalability, However, to the best of our knowledge, not much research have been done to evaluate existing Test Case Prioritization (TCP) techniques in SPL. This paper provides a survey of existing works on test case prioritization technique. This study provide classification and compare the best technique, trends, gaps and proposed frameworks based on the literature. The evaluation and discussion is using Normative Information Model-based Systems Analysis and Design (NIMSAD) on aspects that include context, content and validation. The discussion highlights the lack of technique for scalability issue in SPL with most of the work is on academia setting but not on industrial practices.

Controller-Centric Combinatorial Wrap-Around Interaction Testing to Evaluate a Stateful PCE-Based Transport Network Architecture

The objective of this paper is to develop a controller-centric combinatorial wrap-around interaction testing methodology for a stateful path computation element (PCE)-based transport network architecture. By exploiting the internal contexts of the controller-centric network in conjunction with combinatorial, wrap-around, and interaction testing methodologies, the proposed testing methodology helps test engineers build a highly configurable testing environment, select high-quality test cases to obtain the best possible combination of interactions, and prune fault cases or useless cases from all possible test cases throughout the entire development process as gray-box testing. The experimental results verify that the combined testing methodology effectively evaluates the controller-centric network for all testing processes in a completely controlled environment.

An Implementation of Enhanced Combinatorial Interaction Testing Software (ECITS)

An Implementation of Enhanced Combinatorial Interaction Testing Software (ECITS)

Generating Pairwise Combinatorial Interaction Test Suites Using Single Objective Dragonfly Optimisation Algorithm

Combinatorial interaction testing has been addressed as an important and effective software testing technique recently. The technique can significantly reduce the number of test cases. It provides an alternative method to exhaustive testing and design of experiments (DOE) by allowing a minimised set of tests to represent the actual set of test cases. The reduction of the test cases is based on the combination of input parameters for the system-under-test. This combination could be considered as input-configuration of different software families. Pairwise combinatorial test suite takes the interaction of two input parameters into consideration instead of many parameter interactions. Evidences showed that this test suite can detect most of the faults in the software-under-test as compared to higher interactions. This paper shows the adaptation and assessment of Dragon Fly (DF), a novel swarm intelligent optimisation algorithm, for pairwise combinatorial test generation. The design of the algorithm is addressed in the paper. The algorithm is evaluated extensively through different experiments and benchmarks. The evaluation shows the efficiency of the proposed technique for test suite generation and the usefulness of DF optimisation algorithm for future investigations.

Handling constraints in combinatorial interaction testing in the presence of multi objective particle swarm and multithreading

ContextCombinatorial testing strategies have lately received a lot of attention as a result of their diverse applications. In its simple form, a combinatorial strategy can reduce several input parameters (configurations) of a system into a small set based on their interaction (or combination). In practice, the input configurations of software systems are subjected to constraints, especially in case of highly configurable systems. To implement this feature within a strategy, many difficulties arise for construction. While there are many combinatorial interaction testing strategies nowadays, few of them support constraints. ObjectiveThis paper presents a new strategy, to construct combinatorial interaction test suites in the presence of constraints. MethodThe design and algorithms are provided in detail. To overcome the multi-judgement criteria for an optimal solution, the multi-objective particle swarm optimisation and multithreading are used. The strategy and its associated algorithms are evaluated extensively using different benchmarks and comparisons. ResultsOur results are promising as the evaluation results showed the efficiency and performance of each algorithm in the strategy. The benchmarking results also showed that the strategy can generate constrained test suites efficiently as compared to state-of-the-art strategies. ConclusionThe proposed strategy can form a new way for constructing of constrained combinatorial interaction test suites. The strategy can form a new and effective base for future implementations.

Constrained Interaction Testing: A Systematic Literature Study

Interaction testing can be used to effectively detect faults that are otherwise difficult to find by other testing techniques. However, in practice, the input configurations of software systems are subjected to constraints, especially in the case of highly configurable systems. Handling constraints effectively and efficiently in combinatorial interaction testing is a challenging problem. Nevertheless, researchers have attacked this challenge through different techniques, and much progress has been achieved in the past decade. Thus, it is useful to reflect on the current achievements and shortcomings and to identify potential areas of improvements. This paper presents the first comprehensive and systematic literature study to structure and categorize the research contributions for constrained interaction testing. Following the guidelines of conducting a literature study, the relevant data are extracted from a set of 103 research papers belonging to constrained interaction testing. The topics addressed in constrained interaction testing research are classified into four categories of constraint test generation, application, generation and application, and model validation studies. The papers within each of these categories are extensively reviewed. Apart from answering several other research questions, this paper also discusses the applications of constrained interaction testing in several domains, such as software product lines, fault detection and characterization, test selection, security, and graphical user interface testing. This paper ends with a discussion of limitations, challenges, and future work in the area.

Using simulated annealing for computing cost-aware covering arrays

The configuration spaces of software systems are often too large to test exhaustively. Combinatorial interaction testing approaches, such as covering arrays, systematically sample the configuration space and test only the selected configurations. In an attempt to reduce the cost of testing, standard t-way covering arrays aim to cover all t-way combinations of option settings in a minimum number of configurations. By doing so, they simply assume that every configuration costs the same. When the cost varies from one configuration to another, however, minimizing the number of configurations is not necessarily the same as minimizing the cost. To overcome this issue, we have recently introduced cost-aware covering arrays. In a nutshell, a t-way cost-aware covering array is a standard t-way covering array that “minimizes” a given cost function modeling the actual cost of testing. In this work we develop a simulated annealing-based approach to compute cost-aware covering arrays, which takes as input a configuration space model enhanced with a cost function and computes a cost-aware covering array by using two alternating neighboring state generation strategies together with a fitness function expressed as a weighted sum of two objectives: covering all required t-way option setting combinations and minimizing the cost function. To the best of our knowledge, the proposed approach is the first approach that computes cost-aware covering arrays for general, non-additive linear cost functions with multiplicative interaction effects. We evaluate the approach both by conducting controlled experiments, in which we systematically vary the input models to study the sensitivity of the approach to various factors and by conducting experiments using real cost functions for real software systems. We also compare cost-aware covering arrays to standard covering arrays constructed by well-known algorithms and study how fast the construction costs are compensated by the cost reductions provided. Our empirical results suggest that the proposed approach is more effective and efficient than the existing approaches.

Combinatorial Interaction Testing of Software Product Lines: A Mapping Study

Software Product Line (SPL) is a software engineering paradigm that is inspired by the concept of reusability of common features, formulated for different software product. Complete testing on entire SPL is known to be unfeasible, due to the very large number of possible products to be produced, configured using a subset or all possible features in the SPL. This paper reports a systematic mapping study (SMS) of relevant primary studies as the evidence on the application of Combinatorial Interaction Testing (CIT) for SPL. In CIT, one has to construct a covering array, which is a set of configurations having valid feature combinations and every combination of t features appears at least once in the array. This is also known as t-wise testing. By following the systematic mapping study guidelines, we have selected and filtered 44 primary studies for review. The most prominent CIT techniques in aiding SPL testing are those based on greedy algorithms followed by meta-heuristics algorithms. The motivation of SPL testing is to anticipate the feature interaction problem, in which the majority of the works were reported to leverage test configuration selection approach, while some employed test configuration prioritization approach. Numerous works have been reported, but only few works managed to demonstrate their scalability, as most primary studies only deal with low strength (t is less than 4) of t-wise testing.

Construction of t-way covering arrays using genetic algorithm

In a component based system, interaction among components may give rise to interaction faults. Faults can be caused by the interactions of two or more than two components. Since, it is not feasible to test all possible interactions combinatorial interaction testing is being employed to generate covering arrays (CAs) that covers all t-way interactions between components. In this paper, we generalize our previous work, that uses a greedy based genetic algorithm, to generate CA from 2-way to t-way testing (2 ≤ t ≤ 4). We implement the proposed strategy by extending an open source tool PWiseGen. The main drawback of PWiseGen is the need to input the size of CA in advance. To overcome this restriction of PWiseGen, we propose a variation of binary search algorithm that generates optimal CA without knowing its size in advance. Experiments are conducted to evaluate the effectiveness of the proposed approach and it has been observed that the proposed approach performs well as compared to existing state-of-the-art algorithms.

ABC-CAG: Covering Array Generator for Pair-wise Testing Using Artificial Bee Colony Algorithm

Testing is an indispensable part of the software development life cycle. It is performed to improve the performance, quality and reliability of the software. Various types of testing such as functional testing and structural testing are performed on software to uncover the faults caused by an incorrect code, interaction of input parameters, etc. One of the major factors in deciding the quality of testing is the design of relevant test cases which is crucial for the success of testing. In this paper we concentrate on generating test cases to uncover faults caused by the interaction of input parameters. It is advisable to perform thorough testing but the number of test cases grows exponentially with the increase in the number of input parameters, which makes exhaustive testing of interaction of input parameters imprudent. An alternative to exhaustive testing is combinatorial interaction testing (CIT) which requires that every t-way interaction of input parameters be covered by at least one test case. Here, we present a novel strategy ABC-CAG (Artificial Bee Colony-Covering Array Generator) based on the Artificial Bee Colony (ABC) algorithm to generate covering an array and a mixed covering array for pair-wise testing. The proposed ABC-CAG strategy is implemented in a tool and experiments are conducted on various benchmark problems to evaluate the ecacy of the proposed approach. Experimental results show that ABC-CAG generates better/comparable results as compared to the existing state-of-the-art algorithms.