Combinatorial Testing Strategies Research Articles

Test Scenarios Construction Based on Combinatorial Testing Strategy for Automated Vehicles

Scenario-based testing is an important verification and certification measure to evaluate the safety of automated vehicles. In view of the existing test scenario composition methods, which may miss some critical scenario problems that have low occurrence probability, we fully combined the ego-vehicle with the possible relative positions and movement directions of surrounding traffic participants based on a complex scenario group. We applied scenario-screening rules to obtain the functional test scenarios with different traffic environments and driving task complexities, which ensured the coverage of the test scenarios and reduced the number of test scenarios. The problem arose that the amount of test cases was too large after the discretized combination of test scenario parameters, so we adopted a three-way combinatorial testing strategy to greatly reduce the number of test cases. Taking the complicated lane changing scenario of the ego-vehicle as an example, the simulation method was adopted, and the critical test cases were obtained by screening through safety indicators. Finally, the K-medoids clustering method was used to further reduce the number of critical test cases, and a pairwise combinatorial test strategy was used to combine dynamic scenario and static scenario elements to obtain critical test cases for closed-road testing.

Review of Combinatorial Testing Strategy

Review of Combinatorial Testing Strategy

Constraints handling in combinatorial interaction testing using multi-objective crow search and fruitfly optimization

Combinatorial testing strategies are the recent interest of the researchers because of their wide variety of applications. The combinatorial testing strategy posses a great deal of minimizing the count of the input parameters of a system such that a small set of parameters is obtained depending on their interaction. Practically, the input models of the software system are subjected to the constraints mainly in highly configurable systems. There exist a number of issues while integrating the constraint in the testing strategy that is overcome using the proposed method. The proposed method aims at developing the combinatorial interaction test suites in the presence of constraints. The proposed strategy is multi-objective crow search and fruitfly optimization that is developed by the integration of the crow search algorithm and the chaotic fruitfly optimization algorithm. The proposed algorithm offers an optimal selection of the test suites at the better convergence. The experimentation based on the constraints and the analysis are carried out in terms of average size and average time with their values as 10 and 30 s, respectively.

ABCVS: An Artificial Bee Colony for Generating Variable T-Way Test Sets

To achieve acceptable quality and performance of any software product, it is crucial to assess various software components in the application. There exist various software-testing techniques such as combinatorial testing and covering array. However, problems such as t-way combinatorial explosion is still challenging in any combinatorial testing strategy, as it takes into consideration the entire combinations of input variables. Therefore, to overcome this problem, several optimizations and metaheuristic strategies have been suggested. One of the most effective optimization algorithms based techniques is the Artificial Bee Colony (ABC) algorithm. This paper presents t-way generation strategy for both a uniform and variable strength test suite by applying the ABC strategy (ABCVS) to reduce the size of the test suite and to subsequently enhance the test suite generation interaction. To assess both the effectiveness and performance of the presented ABCVS, several experiments were conducted applying various sets of benchmarks. The results revealed that the proposed ABCVS outweigh the existing based strategies and demonstrated wider interaction between components as opposed to AI-search based and computational based strategies. The results also revealed higher prospect of ABCVS in the aspect of its effectiveness and performance as observed in the majority of case studies.

One-Parameter-at-a-Time Combinatorial Testing Strategy Based on Harmony Search Algorithm OPAT-HS

Software testing is required to verify and validate systems. Combinatorial testing in one of the significant testing techniques. Design and select test cases for combinatorial testing considered as combinatorial problem. Even though there are some existing optimization algorithm based combinatorial testing strategies, that minimize the number of test cases, but most of these strategies based on one-test-at-a-time (OTAT) approach and none of them has adopted the one-parameter-at-a-time (OPAT) approach. Therefore, this paper will propose a new OPAT strategy based on Harmony Search Algorithm (HS) called OPAT-HS. OPAT-HS was originally designed only to support Covering Array (CA) and Mixed Covering Array (MCA) for uniform interaction strength. The result obtained in the experiments appears that OPAT-HS is always best at configurations with MCA notations.

A tuned version of genetic algorithm for efficient test suite generation in interactive t-way testing strategy

ContextTo improve the quality and correctness of a software product it is necessary to test different aspects of the software system. Among different approaches for software testing, combinatorial testing along with covering array is a proper testing method. The most challenging problem in combinatorial testing strategies like t-way, is the combinatorial explosion which considers all combinations of input parameters. Many evolutionary and meta-heuristic strategies have been proposed to address and mitigate this problem. ObjectiveGenetic Algorithm (GA) is an evolutionary search-based technique that has been used in t-way interaction testing by different approaches. Although useful, all of these approaches can produce test suite with small interaction strengths (i.e. t ≤ 6). Additionally, most of them suffer from expensive computations. Even though there are other strategies which use different meta-heuristic algorithms to solve these problems, in this paper, we propose an efficient uniform and variable t-way minimal test suite generation approach to address these problems using GA, called Genetic Strategy (GS). MethodBy changing the bit structure and accessing test cases quickly, GS improves performance of the fitness function. These adjustments and reduction of the complexities of GA in the proposed GS decreases the test suite size and increases the speed of test suite generation up tot=20. ResultsTo evaluate the efficiency and performance of the proposed GS, various experiments are performed on different set of benchmarks. Experimental results show that not only GS supports higher interaction strengths in comparison with the existing GA-based strategies, but also its supported interaction strength is higher than most of other AI-based and computational-based strategies. ConclusionFurthermore, experimental results show that GS can compete against the existing (both AI-based and computational-based) strategies in terms of efficiency and performance in most of the case studies.

A Review on Recent T-way Combinatorial Testing Strategy

T-way combinatorial testing aims to generate a smaller test suite size. The purpose of t-way combinatorial testing is to overcome exhaustive testing. Although many existing strategies have been developed for t-way combinatorial testing, study in this area is encouraging as it falls under NP-hard optimization problem. This paper focuses on the analysis of existing algorithms or tools for the past seven years. Taxonomy of combinatorial testing is proposed to ease the analysis. 20 algorithms or tools were analysed based on strategy approach, search technique, supported interaction and year published. 2015 was the most active year in which researchers developed t-way algorithms or tools. OTAT strategy and metaheuristic search technique are the most encouraging research areas for t-way combinatorial testing. There is a slight difference in the type of interaction support. However, uniform strength is the most utilized form of interaction from 2010 to the first quarter of 2017.

Variable strength combinatorial testing of concurrent programs

Reachability testing is an important approach to testing concurrent programs. It generates and exercises synchronization sequences automatically and on-the-fly without saving any test history. Existing reachability testing can be classified into exhaustive and t-way testing. Exhaustive testing is impractical in many cases while t-way testing may decrease the capability of fault detection in some cases. In this paper, we present a variable strength reachability testing strategy, which adopts the dynamic framework of reachability testing and uses a variable strength combinatorial strategy. Different parameter groups are provided with different covering strength. Variable strength testing covers no t-way combinations but the necessary combinations of parameters having mutual interactions in a concurrent program. It is more reasonable than t-way testing because uniform interactions between parameters do not often exist in concurrent systems. We propose a merging algorithmthat implements the variable strength combinatorial testing strategy and conduct our experiment on several concurrent programs. The experimental results indicate that our variable strength reachability testing reaches a good tradeoff between the effectiveness and efficiency. It can keep the same capability of fault detection as exhaustive reachability testing while substantially reducing the number of synchronization sequences and decreasing the execution time in most cases.

Cost-effective strategies for the regression testing of database applications: Case study and lessons learned

Testing and, more specifically, the regression testing of database applications is highly challenging and costly. One can rely on production data or generate synthetic data, for example based on combinatorial techniques or operational profiles. Both approaches have drawbacks and advantages. Automating testing with production data is impractical and combinatorial test suites might not be representative of system operations.In this paper, based on a large scale case study in a representative development environment, we explore the cost and effectiveness of various approaches and their combination for the regression testing of database applications, based on production data and synthetic data generated through classification tree models of the input domain.The results confirm that combinatorial test suite specifications bear little relation to test suite specifications derived from the system operational profile. Nevertheless, combinatorial testing strategies are effective, both in terms of the number of regression faults discovered but also, more surprisingly, in terms of the importance of these faults. However, our study also shows that relying solely on synthesized test data derived from test models could lead to important faults slipping to production. Thus, we recommend that testing on production data and combinatorial testing be combined to achieve optimal results.

Editorial: Why should I review papers?

Editorial: Why should I review papers?

A combinatorial testing strategy for concurrent programs

AbstractOne approach to testing concurrent programs is called reachability testing, which derives test sequences automatically and on‐the‐fly, without constructing a static model. Existing reachability testing algorithms are exhaustive in that they are intended to exercise all possible synchronization sequences of a concurrent program with a given input. In this paper, we present a new testing strategy, called t‐way reachability testing, that adopts the dynamic framework of reachability testing but selectively exercises a subset of synchronization sequences. The selection of the synchronization sequences is based on a combinatorial testing strategy called t‐way testing. We present an algorithm that implements t‐way reachability testing, and report the results of several case studies that were conducted to evaluate its effectiveness. The results indicate that t‐way reachability testing can substantially reduce the number of synchronization sequences exercised during reachability testing while still effectively detecting faults. Copyright © 2007 John Wiley & Sons, Ltd.