Covering Arrays Research Articles

New Upper Bounds for Covering Arrays of Order Seven

New Upper Bounds for Covering Arrays of Order Seven

Weight Vector Definition for MOEA/D-Based Algorithms Using Augmented Covering Arrays for Many-Objective Optimization

Many-objective optimization problems are today ever more common. The decomposition-based approach stands out among the evolutionary algorithms used for their solution, with MOEA/D and its variations playing significant roles. MOEA/D variations seek to improve weight vector definition, improve the dynamic adjustment of weight vectors during the evolution process, improve the evolutionary operators, use alternative decomposition methods, and hybridize with other metaheuristics, among others. Although an essential topic for the success of MOEA/D depends on how well the weight vectors are defined when decomposing the problem, not as much research has been performed on this topic as on the others. This paper proposes using a new mathematical object called augmented covering arrays (ACAs) that enable a better sampling of interactions of M objectives using the least number of weight vectors based on an interaction level (strength), defined a priori by the user. The proposed method obtains better results, measured in inverted generational distance, using small to medium populations (up to 850 solutions) of 30 to 100 objectives over DTLZ and WFG problems against the traditional weight vector definition used by MOEA/D-DE and results obtained by NSGA-III. Other MOEA/D variations can include the proposed approach and thus improve their results.

Effectively computing high strength mixed covering arrays with constraints

Covering arrays have become a key piece in Combinatorial Testing. In particular, we focus on the efficient construction of Covering Arrays with Constraints of high strength. SAT solving technology has been proven to be well suited when solving Covering Arrays with Constraints. However, the size of the SAT reformulations rapidly grows up with higher strengths. To this end, we present a new incomplete algorithm that mitigates substantially memory blow-ups and its parallel version that allows reducing run-time consumption. Thanks to these new developments we provide a tool for Combinatorial Testing in practical environments. The experimental results confirm the goodness of the approach, opening avenues for new practical applications.

An Effective Approach to High Strength Covering Array Generation in Combinatorial Testing

An Effective Approach to High Strength Covering Array Generation in Combinatorial Testing

Innovations in t-way test creation based on a hybrid hill climbing-greedy algorithm

<p>In combinatorial testing development, the fabrication of covering arrays is the key challenge by the multiple aspects that influence it. A wide range of combinatorial problems can be solved using metaheuristic and greedy techniques. Combining the greedy technique utilizing a metaheuristic search technique like hill climbing (HC), can produce feasible results for combinatorial tests. Methods based on metaheuristics are used to deal with tuples that may be left after redundancy using greedy strategies; then the result utilization is assured to be near-optimal using a metaheuristic algorithm. As a result, the use of both greedy and HC algorithms in a single test generation system is a good candidate if constructed correctly. This study presents a hybrid greedy hill climbing algorithm (HGHC) that ensures both effectiveness and near-optimal results for generating a small number of test data. To make certain that the suggested HGHC outperforms the most used techniques in terms of test size. It is compared to others in order to determine its effectiveness. In contrast to recent practices utilized for the production of covering arrays (CAs) and mixed covering arrays (MCAs), this hybrid strategy is superior since allowing it to provide the utmost outcome while reducing the size and limit the loss of unique pairings in the CA/MCA generation.</p>

Deep Learning Optimisation of Static Malware Detection with Grid Search and Covering Arrays

This paper investigates the impact of several hyperparameters on static malware detection using deep learning, including the number of epochs, batch size, number of layers and neurons, optimisation method, dropout rate, type of activation function, and learning rate. We employed the cAgen tool and grid search optimisation from the scikit-learn Python library to identify the best hyperparameters for our Keras deep learning model. Our experiments reveal that cAgen is more efficient than grid search in finding the optimal parameters, and we find that the selection of hyperparameter values has a significant impact on the model’s accuracy. Specifically, our approach leads to significant improvements in the neural network model’s accuracy for static malware detection on the Ember dataset (from 81.2% to 95.7%) and the Kaggle dataset (from 94% to 98.6%). These results demonstrate the effectiveness of our proposed approach, and have important implications for the field of static malware detection.

Covering Arrays ML HPO for Static Malware Detection

Malware classification is a well-known problem in computer security. Hyper-parameter optimisation (HPO) using covering arrays (CAs) is a novel approach that can enhance machine learning classifier accuracy. The tuning of machine learning (ML) classifiers to increase classification accuracy is needed nowadays, especially with newly evolving malware. Four machine learning techniques were tuned using cAgen, a tool for generating covering arrays. The results show that cAgen is an efficient approach to achieve the optimal parameter choices for ML techniques. Moreover, the covering array shows a significant promise, especially cAgen with regard to the ML hyper-parameter optimisation community, malware detectors community and overall security testing. This research will aid in adding better classifiers for static PE malware detection.

Balanced covering arrays: A classification of covering arrays and packing arrays via exact methods

AbstractIn this paper we investigate the intersections of classes of covering arrays (CAs) and packing arrays (PAs). The arrays appearing in these intersections obey to upper and lower bounds regarding the appearance of tuples in sub‐matrices—we call these arrays balanced covering arrays. We formulate and formalize first observations for which upper and lower bounds on the appearance of tuples it is of interest to consider these intersections of CAs and PAs. Outside of these bounds the intersections will be either empty, for the case of too restrictive constraints, or equal to the maximum element in the emerging lattices, for the case of too weak constraints. We present a column extension algorithm for classification of nonequivalent balanced CAs that uses a SAT solver or a pseudo‐Boolean (PB) solver to compute the columns suitable for array extension together with a lex‐leader ordering to identify unique representatives for each equivalence class of balanced CAs. These computations bring to light a dissection of classes of CAs that is partially nested due to the nature of the considered intersections. These dissections can be trivial, containing only a single type of balanced CAs, or can also appear as highly structured containing multiple nested types of balanced CAs. Our results indicate that balanced CAs are an interesting class of designs that is rich of structure.

QC-LDPC Codes From Difference Matrices and Difference Covering Arrays

QC-LDPC Codes From Difference Matrices and Difference Covering Arrays

Metaheuristic algorithm for the construction of mixed covering arrays of different strength levels to be used in the design of black box software test cases

Introduction— Currently, software quality is a fundamental feature to ensure a space in the global market that day by day is more competitive and demanding. Software testing allows software developers to find and fix bugs and, thereby, raise the quality of their products. The cost of testing is now estimated at 50% of the total cost of software development, so it is necessary to find less expensive ways to ensure elevated levels of fault detection. In this scenario, black box tests play a key role; within these tests, combinatorial approaches are one of the best options. In combinatorial tests, test users must have a tool that provides them with the least number of test cases with the greatest coverage (failure detection) according to the parameters of the method (procedure, function, or other) that they want to test, and this is the main objective of this work. Objective— This paper presents an algorithm that supports the generation of test cases in black box tests based on the creation of Mixed Covering Arrays (MCA). These fixes allow you to generate the smallest number of test cases required to test a unit of code with the highest required coverage. The proposed algorithm builds an initial solution based on a greedy algorithm and then improves this solution through an iterative process oriented by simulated annealing (metaheuristic algorithm). Methodology— The research was conducted following the Iterative Research Pattern proposed by Pratt. First, the main problems reported in the state of the art for the construction of CSF were identified, then a review of the proposed solutions to these problems was performed. Then a first algorithm was created and then iteratively modified this algorithm, including, and removing components according to the experimental results of its operation. When the desired version was obtained, a process of refinement of parameters was performed and compared with the best results presented in the literature, results obtained by different algorithms. Results— The proposed algorithm obtains MCAs that are competitive (on average 3 additional test cases) against the best reported in the state of the art in an abbreviated execution time, an aspect that is of special interest for software testers. Conclusions— It was confirmed that the greedy and metaheuristic approach based on simulated annealing is a suitable alternative for the construction of a CSF. Neighboring solution construction algorithms are key to finding the required MCA in a shorter execution time.

Incomplete MaxSAT approaches for combinatorial testing

We present a Satisfiability (SAT)-based approach for building Mixed Covering Arrays with Constraints of minimum length, referred to as the Covering Array Number problem. This problem is central in Combinatorial Testing for the detection of system failures. In particular, we show how to apply Maximum Satisfiability (MaxSAT) technology by describing efficient encodings for different classes of complete and incomplete MaxSAT solvers to compute optimal and suboptimal solutions, respectively. Similarly, we show how to solve through MaxSAT technology a closely related problem, the Tuple Number problem, which we extend to incorporate constraints. For this problem, we additionally provide a new MaxSAT-based incomplete algorithm. The extensive experimental evaluation we carry out on the available Mixed Covering Arrays with Constraints benchmarks and the comparison with state-of-the-art tools confirm the good performance of our approaches.

IPO-MAXSAT: Combining the In-Parameter-Order Strategy for Covering Array Generation with MaxSAT Solving (Extended Abstract)

Covering arrays (CAs) are discrete objects appearing in combinatorial design theory that find practical applications, most prominently in software testing. The generation of optimized CAs is a difficult combinatorial optimization problem being subject to ongoing research. Previous studies have shown that many different algorithmic approaches are best suited for different instances of CAs. In this extended abstract we describe the IPO-MAXSAT algorithm, which adopts the prominent IPO strategy for CA generation and uses MaxSAT solving to optimize the occurring sub-problems.

New upper bounds for sequence Covering Arrays using a 3-stage approach

Nowadays, it is mandatory to test the functionality of any software component. In this sense, Covering Arrays (CAs) have been used as test-suites to exercise combinations of parameters’ values. However, in some software components, the functionality depends not only on the combination of parameters’ values but also on how the parameters’ values are fed to a software component. Therefore, the need to use sequence coverage arrays (SCAs) to test the order in which the input parameters are defined in a software component naturally arises. The combined use of CAs and SCAs allows the creation of test-suites that exercise the functionality of software components, taking into account the combination of parameters’ values and the order in which these combinations are used. In this paper, three applications of SCAs will be presented. The first application is related to testing the functionality of a water filtering process and only one SCA is required. The second application combines an SCA with a CA to test a user interface of a software component. The third application refers to the construction of wetlands for water bioremediation using a mixed covering array (MCA) and a sequence covering array. The coverage of an SCA is determined by its strength (i.e., the size of the subsequences of variables covered at least once). We present a 3-stage approach to construct SCAs for strengths t∈{3,4,5,6,7,8}. The stages of our approach are a greedy stage that builds an initial SCA one row at-a-time; a post-optimization stage that involves an algorithm for the detection of redundant elements and an algorithm that aims to reduce the cardinality of an SCA, and a metaheuristic stage that is implemented using a simulated annealing algorithm. Relevant computational results are 502 new upper bounds (178 cases for t=3, 177 cases for t=4, 88 cases for t=5, 43 cases for t=6, 12 cases for t=7, and 4 cases for t=8). Also we have compared the performance of our approach with 12 algorithms using a known benchmark consisting of 101 cases, our approach wins in 69 cases (18 cases for t=3, 24 cases for t=4, 24 cases for t=5, and 3 cases for t=6).

Effective Covering Array Generation Using an Improved Particle Swarm Optimization

In the test case generation process of combinatorial testing, particle swarm optimization (PSO) is widely concerned for its simple implementation and fast convergence rate; however, it often falls into local optimum due to premature convergence. To attack this problem, a novel adaptive value measurement strategy is adopted by weighing the relationship between current test cases and historical test cases. The test case with the minimum average hamming distance is selected as the optimal test case, and the inertial weight linear differential decrease strategy is developed to ensure better inertial weight in different search stages, further to improve the capability of generating smaller covering arrays. In addition, we integrate the simulated annealing strategy into the improved PSO to improve the ability of particles jumping out of the local optimum, and an innovative approach for generating a better covering array is proposed. Experiments on 16 classical random strength covering arrays suggest that our approach outperforms six other techniques in terms of effectiveness.

An Adaptive Penalty based Parallel Tabu Search for Constrained Covering Array Generation

The generation of the optimal constrained covering arrays is a key challenge in the research field of combinatorial testing, where a variety of Constrained Covering Array Generation (CCAG) algorithms have been developed. However, existing algorithms typically reuse constraint solver or forbidden tuple-based techniques to handle constraints, which might restrict their potentials on finding smaller arrays.

In-Parameter-Order strategies for covering perfect hash families

Combinatorial testing makes it possible to test large systems effectively while maintaining certain coverage guarantees. At the same time, the construction of optimized covering arrays (CAs) with a large number of columns is a challenging task. Heuristic and Metaheuristic approaches often become inefficient when applied to large instances, as the computation of the quality for new moves or solutions during the search becomes too slow. Recently, the generation of covering perfect hash families (CPHFs) has led to vast improvements to the state of the art for many different instances of covering arrays. CPHFs can be considered a compact form of a specific family of covering arrays. Their compact representation makes it possible to apply heuristic methods for instances with a much larger number of columns. In this work, we adapt the ideas of the well-known In-Parameter-Order (IPO) strategy for covering array generation to efficiently construct CPHFs, and therefore implicitly covering arrays. We design a way to realize the concept of vertical extension steps in the context of CPHFs and discuss how a horizontal extension can be implemented in an efficient manner. Further, we develop a horizontal extension strategy for CPHFs with subspace restrictions that identifies candidate columns greedily based on conditional expectation. Then using a local optimization strategy, a candidate may be adjoined to the solution or may replace one of the existing columns. An extensive set of computational results yields many significant improvements on the sizes of the smallest known covering arrays.

Correction to: Factorials Experiments, Covering Arrays, and Combinatorial Testing

Correction to: Factorials Experiments, Covering Arrays, and Combinatorial Testing

Factorials Experiments, Covering Arrays, and Combinatorial Testing

In the twenty-first century, our life will increasingly depend on software-based products and complex interconnected systems. Thus, the quality and security of software-based systems is a world-wide concern. Combinatorial testing is a versatile methodology for finding errors (bugs) and vulnerabilities in software-based systems. This paper offers a review of combinatorial testing. Combinatorial testing (CT) methods evolved from investigations which looked like factorial experiments (FE) with pass/fail outcomes. We will discuss the similarities and differences between FE and CT. Use of CT for detecting errors (bugs) in software-based systems has gained significant interest from the international software testing community. Many successful results have been reported from the use of CT to detect software errors in aerospace, automotive, defense, cybersecurity, electronic medical systems, and financial service industries. Now, combinatorial testing methods are being increasingly used to investigate vulnerabilities in software-based systems. Combinatorial testing could be useful in detecting errors and security vulnerabilities in Internet of Things, Autonomous Systems, and Artificially Intelligent Software.

Mixed covering arrays on graphs of small treewidth

Covering arrays are combinatorial objects that have been successfully applied in design of test suites for testing systems such as software, hardware, and networks where failures can be caused by the interaction between their parameters. Let [Formula: see text] and [Formula: see text] be positive integers with [Formula: see text]. Two vectors [Formula: see text] and [Formula: see text] are qualitatively independent if for any ordered pair [Formula: see text], there exists an index [Formula: see text] such that [Formula: see text]. Let [Formula: see text] be a graph with [Formula: see text] vertices [Formula: see text] with respective vertex weights [Formula: see text]. A mixed covering array on[Formula: see text] , denoted by [Formula: see text], is a [Formula: see text] array such that row [Formula: see text] corresponds to vertex [Formula: see text], entries in row [Formula: see text] are from [Formula: see text]; and if [Formula: see text] is an edge in [Formula: see text], then the rows [Formula: see text] are qualitatively independent. The parameter [Formula: see text] is the size of the array. Given a weighted graph [Formula: see text], a mixed covering array on [Formula: see text] with minimum size is optimal. In this paper, we introduce some basic graph operations to provide constructions for optimal mixed covering arrays on the family of graphs with treewidth at most three.

A primer on the application of neural networks to covering array generation

In the past, combinatorial structures have been used only to tune parameters of neural networks. In this work, we employ neural networks in the form of Boltzmann machines and Hopfield networks for the construction of a specific class of combinatorial designs, namely covering arrays (CAs). In past works, these neural networks were successfully used to solve set cover instances. For the construction of CAs, we consider the corresponding set cover instances and use neural networks to solve such instances. We adapt existing algorithms for solving general set cover instances, which are based on Boltzmann machines and Hopfield networks and apply them for CA construction. Furthermore, for the algorithm based on Boltzmann machines, we consider newly designed versions, where we deploy structural changes of the underlying Boltzmann machine, adding a feedback loop. Additionally, one variant of this algorithm employs learning techniques based on neural networks to adjust the various connections encountered in the graph representing the considered set cover instances. Culminating in a comprehensive experimental evaluation, our work presents the first study of applications of neural networks in the field of covering array generation and related discrete structures and may act as a guideline for future investigations.