T-way Test Research Articles

Comparative Analysis of IPOG Strategy Variants for Enhanced Combinatorial T-way Testing Efficiency

Comparative Analysis of IPOG Strategy Variants for Enhanced Combinatorial T-way Testing Efficiency

CS-FuzGA-PTS: Maximizing Fault Detection through Optimizing T-way Test Suite Prioritization based on Boundary Value Analysis

CS-FuzGA-PTS: Maximizing Fault Detection through Optimizing T-way Test Suite Prioritization based on Boundary Value Analysis

ENHANCED VERSION OF SEEDING AND CONSTRAINT SUPPORT IN IPOG STRATEGY FOR VARIABLE STRENGTH INTERACTION T-WAY TESTING

Interaction strength testing (known as t-way testing) is one of the effective software testing techniques used to find problems with how system components interact with one another. The emergence of variable strength interaction occurs because of the covering array's identified limitation, which prevents it from handling varying parameter testing strengths. Several works were formed in t-way testing to generate a small test suite size using either One-Parameter-at-a-Time (OPAT) or One-Test-at-a-Time (OTAT) approach and base on interaction strength that include uniform, variable and/or input-output relation interaction, since t-way testing is highly complicated (NP-hard). However, no single OPAT t-way strategy claims to support seeding and constraint in variable strength interaction. Motivated by these challenges, the paper presents an enhanced version of IPOG strategy for t-way testing with seeding, constraint, and variable strength interaction supports, named SCIPOG-VS. To assess its efficiency, experiments were performed and compared to other strategies supporting variable strength interaction. Experimental results demonstrated that SCIPOG-VS is competitive in terms of test suite size, and execution time against other existing strategies significantly. We statistically tested our findings with a non-parametric Wilcoxon Signed Rank test, despite this, the outcome shows a significant difference in the size of the test suite between the various strategies.

Innovations in t-way test creation based on a hybrid hill climbing-greedy algorithm

<p>In combinatorial testing development, the fabrication of covering arrays is the key challenge by the multiple aspects that influence it. A wide range of combinatorial problems can be solved using metaheuristic and greedy techniques. Combining the greedy technique utilizing a metaheuristic search technique like hill climbing (HC), can produce feasible results for combinatorial tests. Methods based on metaheuristics are used to deal with tuples that may be left after redundancy using greedy strategies; then the result utilization is assured to be near-optimal using a metaheuristic algorithm. As a result, the use of both greedy and HC algorithms in a single test generation system is a good candidate if constructed correctly. This study presents a hybrid greedy hill climbing algorithm (HGHC) that ensures both effectiveness and near-optimal results for generating a small number of test data. To make certain that the suggested HGHC outperforms the most used techniques in terms of test size. It is compared to others in order to determine its effectiveness. In contrast to recent practices utilized for the production of covering arrays (CAs) and mixed covering arrays (MCAs), this hybrid strategy is superior since allowing it to provide the utmost outcome while reducing the size and limit the loss of unique pairings in the CA/MCA generation.</p>

Combinatorial testing and machine learning for automated test generation

In this issue, we are pleased to present two papers that showcase innovative techniques in software testing in two different directions (memory-aware combinatorial test generation and survey about the use of machine learning for automated test generation). The first paper, ‘An investigation of distributed computing for combinatorial testing’ by Edmond La Chance and Sylvain Hallé, proposes the use of distributed computing to reduce the time and memory required for t-way test generation. The authors present a distributed graph colouring method and a distributed hypergraph vertex covering method for generating high-quality test suites. They also demonstrate how to build a distributed IPOG algorithm using these methods. (Recommended by Arnaud Gotlieb) The second paper, ‘The Integration of Machine Learning into Automated Test Generation: A Systematic Mapping Study’ by Afonso Fontes and Gregory Gay, investigates the integration of machine learning (ML) into automated test generation. Through a systematic study of 124 papers, the authors characterize the emerging research in this area and identify the testing practices, researcher goals, ML techniques applied, evaluation metrics, and challenges in integrating ML into testing. The results show that ML is used to generate inputs for different types of testing, improve the performance of existing generation methods, and generate test verdicts and oracles. The authors also identify common ML techniques used in this area, such as supervised and reinforcement learning, and the evaluation metrics used to assess the effectiveness of these techniques. (Recommended by Phil McMinn) The first paper contributes to the literature on combinatorial testing by demonstrating the effectiveness of distributed computing for test generation. The second paper highlights the potential of ML in automated test generation and provides insights into the challenges that researchers face when integrating ML into testing. We hope that these papers will inspire further research in these directions of software testing.

SCIPOG: Seeding and constraint support in IPOG strategy for combinatorial t-way testing to generate optimum test cases

Combinatorial t-way technique is efficient in generating test data and addressing the problem of combinatorial explosion. When constructing a test case, numerous literatures classified t-way strategies into two basic approaches as either One-test-at-a-time approach (OTAT) and One-parameter-at-a-time approach (OPAT). At least three major challenge groups can be encountered when creating test cases. The first one is provision of parameters seeding support that will improve the software quality. The second involves automatically obtaining data regarding parameter constraints and identifying interactions between systemcomponents. The last one is the execution speed and the test suite size. However, in all the existing OPAT t-way strategies, given that the system is loaded with this information, testing present-day software systems is made difficult or impossible. This study presents an effective combinatorial t-way test case generation strategy named Seeding and Constraint support in In-Parameter-Order Generalized (SCIPOG), to develop an improved paired testing approach. However, the study examines the present state-of-the-art and compares several OPAT strategies found in the literature. Moreover, experiments are discussed as part of this process to demonstrate the correctness of the implementation. When statistically analyzing the findings, two non-parametric tests—the Wilcoxon Rank and Friedman tests–were run. SCIPOG, however, produced competitive results. Finally, SCIPOG showed the efficiency of the two proposed methods, which are seeding and constraint support in IPOG strategy.

Whale Optimization Algorithm Strategies for Higher Interaction Strength T-Way Testing

Much of our daily tasks have been computerized by machines and sensors communicating with each other in real-time. There is a reasonable risk that something could go wrong because there are a lot of sensors producing a lot of data. Combinatorial testing (CT) can be used in this case to reduce risks and ensure conformance to specifications. Numerous existing meta-heuristic-based solutions aim to assist the test suite generation for combinatorial testing, also known as t-way testing (where t indicates the interaction strength), viewed as an optimization problem. Much previous research, while helpful, only investigated a small number of interaction strengths up to t = 6. For lightweight applications, research has demonstrated good fault-finding ability. However, the number of interaction strengths considered must be higher in the case of interactions that generate large amounts of data. Due to resource restrictions and the combinatorial explosion challenge, little work has been done to produce high-order interaction strength. In this context, the Whale Optimization Algorithm (WOA) is proposed to generate high-order interaction strength. To ensure that WOA conquers premature convergence and avoids local optima for large search spaces (owing to high-order interaction), three variants of WOA have been developed, namely Structurally Modified Whale Optimization Algorithm (SWOA), Tolerance Whale Optimization Algorithm (TWOA), and Tolerance Structurally Modified Whale Optimization Algorithm (TSWOA). Our experiments show that the third strategy gives the best performance and is comparable to existing state-of-the-arts based strategies.

Construction of Prioritized T-Way Test Suite Using Bi-Objective Dragonfly Algorithm

Software testing is important for ensuring the reliability of software systems. In software testing, effective test case generation is essential as an alternative to exhaustive testing. For improving the software testing technology, the t-way testing technique combined with metaheuristic algorithm has been great to analyze a large number of combinations for getting optimal solutions. However, most of the existing t-way strategies consider test case weights while generating test suites. Priority of test cases hasn’t been fully considered in previous works, but in practice, it’s frequently necessary to distinguish between high-priority and low-priority test cases. Therefore, the significance of test case prioritization is quite high. For this reason, this paper has proposed a t-way strategy that implements an adaptive Dragonfly Algorithm (DA) to construct prioritized t-way test suites. Both test case weight and test case priority have equal significance during test suite generation in this strategy. We have designed and implemented a Bi-objective Dragonfly Algorithm (BDA) for prioritized t-way test suite generation, and the two objectives are test case weight and test case priority. The test results demonstrate that BDA performs competitively against existing t-way strategies in terms of test suite size, and in addition, BDA generates prioritized test suites.

Review of Nature Inspired Metaheuristic Algorithm Selection for Combinatorial t-Way Testing

The metaheuristic algorithm is a very important area of research that continuously improves in solving optimization problems. Nature-inspired is one of the metaheuristic algorithm classifications that has grown in popularity among researchers over the last few decades. Nature-inspired metaheuristic algorithms contribute significantly to tackling many standing complex problems (such as the combinatorial t-way testing problem) and achieving optimal results. One challenge in this area is the combinatorial explosion problem, which is always intended to find the most optimal final test suite that will cover all combinations of a given interaction strength. As such, test case generation has been selected as the most active research area in combinatorial t-way testing as Non-deterministic Polynomial-Time Hardness (NP-hard). However, not all metaheuristics are effectively adopted in combinatorial t-way testing, some proved to be effective and thus have been popular tools selected for optimization, whilst others were not. This research paper outlines a hundred and ten (110) outstanding nature-inspired metaheuristic algorithms for the last decades (2001 and 2021), such as the Coronavirus Optimization Algorithm, Ebola Optimization Algorithm, Harmony Search, Tiki-Taka Algorithm, and so on. The purpose of this review is to revisit and carry out an up-to-date review of these distinguished algorithms with their respective current states of use. This is to inspire future research in the field of combinatorial t-way testing for better optimization. Thus, we found that all metaheuristics have a simple structure that can be adopted in different areas to become more efficient in optimization. Finally, we suggested some future paths of investigation for researchers who are interested in the combinatorial t-way testing field to employ more of these algorithms by tuning their parameters settings to achieve an optimal solution.

HABCSm: A Hamming Based t-way Strategy based on Hybrid Artificial Bee Colony for Variable Strength Test Sets Generation

Search-based software engineering that involves the deployment of meta-heuristics in applicable software processes has been gaining wide attention. Recently, researchers have been advocating the adoption of meta-heuristic algorithms for t-way testing strategies (where t points the interaction strength among parameters). Although helpful, no single meta-heuristic based t-way strategy can claim dominance over its counterparts. For this reason, the hybridization of meta-heuristic algorithms can help to ascertain the search capabilities of each by compensating for the limitations of one algorithm with the strength of others. Consequently, a new meta-heuristic based t-way strategy called Hybrid Artificial Bee Colony (HABCSm) strategy, based on merging the advantages of the Artificial Bee Colony (ABC) algorithm with the advantages of a Particle Swarm Optimization (PSO) algorithm is proposed in this paper. HABCSm is the first t-way strategy to adopt Hybrid Artificial Bee Colony (HABC) algorithm with Hamming distance as its core method for generating a final test set and the first to adopt the Hamming distance as the final selection criterion for enhancing the exploration of new solutions. The experimental results demonstrate that HABCSm provides superior competitive performance over its counterparts. Therefore, this finding contributes to the field of software testing by minimizing the number of test cases required for test execution.

Gravitational search algorithm based strategy for combinatorial t-way test suite generation

Due to different users' requirements, contemporary software has become feature-rich in terms of input functions (i.e., parameters) and selections (i.e., values). Exhaustive testing on sophisticated software systems is impractical as far as testing time and cost are concerned. Various test case design strategies have been proposed in the literature, such as equivalence class partitioning, boundary value analysis and decision tables. Unlike earlier works, combinatorial t-way testing supports the detection of faults caused by two or more input parameter interactions and thus efficiently minimizes the size of the test suite. Over the past few years, metaheuristic algorithms have appeared to be the most common choice for researchers since their effectiveness proves to offer optimal/near-optimal results. However, generating a t-way test suite is an NP-hard problem, and no single t-way strategy can guarantee to show superiority to others for all types of system configurations. Hence, this paper presents a new t-way strategy based on the Gravitational Search Algorithm (GSA), known as the Gravitational Search Test Generator (GSTG). The primary contribution of this paper is that GSA has adapted for the first time to t-way test data generation. The benchmarking results showcase that GSTG obtains competitive results in most system configurations compared to other existing strategies and addresses higher combination coverage (i.e., t ≤ 10).

Adopting Ant Colony Optimization Algorithm for Pairwise T-Way Test Suite Generation Strategy

Combinatorial testing or t-way testing (t represents strength) is useful to detect faults due to interactions. Pairwise testing is one type of t-way testing. The technique is effective in reducing the number of test cases without decreasing the level of coverage. Besides, its purpose is to overcome the problem of exhaustive testing that produces a great number of test cases and is impossible to be implemented due to time and cost constraints. Pairwise T-way Test Suite Generation Strategy based on Ant Colony Optimization (pTTSGA) is introduced to generate a near-optimum test suite size. Experiments have been conducted to evaluate the ability of this strategy for pairwise testing. The results are compared to benchmark results. Overall, pTTSGA produces a comparable test suite size.

A Tuned Version of Ant Colony Optimization Algorithm (TACO) for Uniform Strength T-way Test Suite Generator: An Execution’s Time Comparison

Software testing is one of important phase in software development. The capabilities of t-way testing to cater bugs due to interactions while reducing the test suite size compare to exhaustive testing has been proven in past decades. However, the execution’s time of the t-way strategy also should be given attention as it could increase the productivity of the testing phase. Thus, this paper proposed a tune version of ant colony optimization algorithm (TACO). TACO is metaheuristic strategy where it adopts ant colony optimization in generating test suites. As further improvement, TACO also integrated with fuzzy logic to dynamically select amount of ant in the algorithm. TACO able to supports uniform strength t-way testing. Experiment result shows that TACO produce a remarkable result of test suite size and execution’s time compared to other strategy for uniform strength t-way testing.

Hybrid Migrating Birds Optimization Strategy for t-way Test Suite Generation

Hybrid meta-heuristics algorithms have gained popularity in recent years to solve t-way test suite generation problems due to better exploration and exploitation capabilities of the hybridization. This paper presents the implementation of meta-heuristic search algorithms that are Migrating Birds Optimization (MBO) algorithm and Genetic Algorithm (GA) hybrid to a t-way test data generation strategy. The proposed strategy is called Elitist Hybrid MBO-GA Strategy (EMBO-GA). Based on the published benchmarking results, the result of these strategies are competitive with most existing strategies in terms of the generated test size in many of the parameter configurations. In the case where this strategy is not the most optimal, the resulting test size is sufficiently competitive.

Utilization of Gravitational Search Algorithm for Combinatorial T-Way Testing

Different techniques of software testing are adopted to deal with bugs found in the highly complicated multifunctional software. However, those techniques have difficulty detecting bugs effectively because most of the bugs are triggered by interaction failures between the input parameters and values in the system. Thus, combinatorial t-way testing strategies have come into existence to produce quality minimized test cases, as well as those test cases can cover all the necessary interactions of parameters once at the least. Besides, as t-way testing is considered as an NP-hard problem, new strategies are always welcomed in this research area in pursuit of the optimum test suite. The main point of this paper is to propose the concept of a type of artificial intelligence (AI) algorithm called gravitational search algorithm (GSA) for t-way interaction testing. GSA is a stochastic optimization algorithm inspired by Newton’s law of gravity and motion and has been widely applied to figure out optimal solutions to real-world issues.

T-way Test Suite Generation Strategy based on Ant Colony Algorithm to Support T-way Variable Strength

T-way test suite generation strategy based on Ant Colony algorithm (TTSGA) has been developed to support t-way variable strength testing which tackles exhaustive testing issues. It employs the ant colony optimization algorithm to generate near-optimal number of test suite size. Even though the test suite size is smaller than exhaustive testing, the strategy covers every possible combination of interacting parameters. The strategy has been evaluated by using benchmarked experiments. Results obtained were compared with other existing strategies that support variable strength. It was found that TTSGA produces comparable results with other existing strategies especially for higher strength configurations. Two non-parametric tests, which are Wilcoxon Rank and Friedman test, have been conducted to analyze the results statistically between TTSGA and HSS as only both strategies have complete experiments results. Although the results shows that there is no significant difference of test suite size among them, TTSGA is in the first rank in the Friedman test.

SCAVS: Implement Sine Cosine Algorithm for generating Variable t-way test suite

Exhaustive testing occurs between interaction elements (i.e., factor and level). In reality, The software system is often impractical because of other constraints, such as cost and time restrictions. The t -way testing (where t indicates the interaction strength) is one of the powerful software, and practical test techniques apply to detect interaction errors between components (i.e., produce quality minimized test cases). The t-way testing is highly complicated (NP-hard). Therefore, many Meta-heuristic algorithms are used to solve combinatorial problems successfully. In general, one of the most effective optimizations algorithm-based techniques SCA. With respect mentioned above, this paper discusses the implementation Sine Cosine Algorithm for generating a variable strength t-way test suite called SCAVS. The results of the experiments showed that SCAVS outperformed and yielded better test sets than other existing other strategies.

Binary Black Hole-Based Optimization For T-Way Testing

Software testing is an important process in software development life cycle, which aims to guarantee the quality of software and reduce the number of errors and bugs. In such a process, software inputs and parameters are used to create a set of testing cases. Nevertheless, the number of testing cases increases enormously when considering all combinations of those inputs. Although t-way testing can reduce the test cases, generating the minimum, yet representative t-way testing set is challenging due to the large search space, which renders finding the best solution computationally prohibitive. The extant solutions suffer from the sensitivity to the random initialization and the subjectivity to the local minima, which adversely affects the reproducibility of these algorithms and obstructs finding the optimal solution. To this end, this paper proposes a novel meta-heuristic searching algorithm called Binary Black Hole (BBH) optimization that formulates the t-way testing as a binary optimization problem. Experimental results show the superiority of BBH over the famous Binary Particle Swarm Optimization (BPSO) algorithm. The achieved improvement shows the capability of BBH in generating smaller covering arrays with the same t-strength compared to those generated by BPSO.

Ant colony algorithm to generate t-way test suite with constraints

T-way testing is one of the testing techniques offered to generate test suites. It focuses on interactions of input parameters based on strength. Besides that, t-way is used to overcome exhaustive testing problem. Another important aspect in t-way is constraints. It forbids certain interactions of input parameters. Therefore, the final test suite contains only valid interactions. T-way testing is NP-hard problem. No single strategy can always generates the best test suite size at all time for all configurations. Thus, Const-TTSGA strategy has been developed to generate test suites. The strategy supports constraints variable strength. Const-TTSGA is a metaheuristic strategy which applies ant colony algorithm to generate the best test cases. Two types of experiments have been conducted; constraints uniform and variable strength which consists of few other configurations. Results obtained are compared to benchmarked results. Const-TTSGA outperformed other strategy for costraints uniform strength experiments except for one configuration. However, the strategy outperformed other strategies for constraints variable strength experiments.

A Hybrid Artificial Bee Colony Strategy for t-way Test Set Generation with Constraints Support

t-way interaction testing is a systematic approach for exhaustive test set generation. It is a vital test planning method in software testing, which generates test sets based on interaction between parameters to cover every possible test sets combinations. t-way strategy clarifies the interaction strength between the number of parameters. However, there are some test sets combinations that should be excluded when generating the final test set as a result of invalid outputs, impossible or unwanted test sets combinations (e.g. system requirements set). These types of set combinations are known as constraint’s combinations or forbidden combinations. From existing studies, several t-way strategies have been proposed to address the test set combination problem, however, generating the optimal test set is still open research being an NP-hard problem. Therefore, this study proposed a novel hybrid artificial bee colony (HABC) t-way test set generation strategy with constraints support. The proposed approach is based on a hybrid artificial bee colony (ABC) algorithm with a particle swarm optimization (PSO) algorithm. PSO was integrated as the exploratory agent for the ABC hence the hybrid nature. The information sharing ability of PSO via the Weight Factor is used to enhance the performance of ABC. The output of the hybrid ABC is a set of promising optimal test set combinations. The results of the experiments showed that HABC outperformed and yielded better test sets than existing methods (HSS, LAHC, SA_SAT, PICT, TestCover, mATEG_SAT).