Covering Arrays Research Articles

On a Combinatorial Framework for Fault Characterization

Covering arrays have been widely used to detect the presence of faults in large software and hardware systems. Indeed, finding failures that result from faulty interactions requires that all interactions that may cause faults be covered by a test case. However, finding the actual faults requires more, because the failures resulting from two potential sets of faults must not be the same. The combinatorial requirements on test suites to enable a tester to locate the faults are developed, and set in the context of similar combinatorial search questions. Test suites known as locating and detecting arrays to locate faults both in principle and in practice generalize covering arrays, thereby addressing combinatorial fault characterization. In common with covering arrays, these locating and detecting arrays scale logarithmically in size with the number of factors, but unlike covering arrays they support complete characterization of the interactions that underlie faults.

Research Notes: Binary Test-Suites Using Covering Arrays

Software testing is an essential activity to ensure the quality of software systems. Combinatorial testing is a method that facilitates the software testing process; it is based on an empirical evidence where almost all faults in a software component are due to the interaction of very few parameters. The test generation problem for combinatorial testing can be represented as the construction of a matrix that has certain properties; typically this matrix is a covering array. Covering arrays have a small number of tests, in comparison with an exhaustive approach, and provide a level of interaction coverage among the parameters involved. This paper presents a repository that contains binary covering arrays involving many levels of interaction. Also, it discusses the importance of covering array repositories in the construction of better covering arrays. In most of the cases, the size of the covering arrays included in the repository reported here are the best upper bounds known, moreover, the files containing the matrices of these covering arrays are available to be downloaded. The final purpose of our Binary Covering Arrays Repository (BCAR) is to provide software testing practitioners the best-known binary test-suites.

Binary Covering Arrays on Tournaments

We introduce graph-dependent covering arrays which generalize covering arrays on graphs, introduced by Meagher and Stevens (2005), and graph-dependent partition systems, studied by Gargano, Körner, and Vaccaro (1994). A covering array $\hbox{CA}(n; 2, G, H)$ (of strength 2) on column graph $G$ and alphabet graph $H$ is an $n\times |V(G)|$ array with symbols $V(H)$ such that for every arc $ij \in E(G)$ and for every arc $ab\in E(H)$, there exists a row $\vec{r} = (r_{1},\dots, r_{|V(G)|})$ such that $(r_{i}, r_{j}) = (a,b)$. We prove bounds on $n$ when $G$ is a tournament graph and $E(H)$ consists of the edge $(0,1)$, which corresponds to a directed version of Sperner's 1928 theorem. For two infinite families of column graphs, transitive and so-called circular tournaments, we give constructions of covering arrays which are optimal infinitely often.

A greedy-metaheuristic 3-stage approach to construct covering arrays

Covering arrays are combinatorial designs used as test-suites in software and hardware testing. Because of their practical applications, the construction of covering arrays with a smaller number of rows is desirable. In this work we develop a greedy-metaheuristic 3-stage approach to construct covering arrays that improve some of the best-known ones. In the first stage, a covering perfect hash family is created using a metaheuristic approach; this initial array may not be complete, and so the derived covering array may have missing tuples. In the second stage, the covering perfect hash family is converted to a covering array and, in case there are missing tuples, a greedy approach completes the covering array through the addition of some rows. The third stage is an iterative postoptimization stage that combines two greedy algorithms and a metaheuristic algorithm; the greedy algorithms detect and reduce redundancy in the covering array, and the metaheuristic algorithm covers the tuples that may become uncovered after the reduction of redundancy. The effectiveness of our greedy-metaheuristic 3-stage approach is assessed through the construction of covering arrays of order four and strengths 3–6; the main results are the improvement of 9473 covering arrays of strength three, 9303 of strength four, 2150 of strength five, and 291 of strength six. To see how to apply covering arrays to real testing scenarios, the final part of this work presents the use of covering arrays of order four for setting up a composting process.

Extended Covering Arrays for Sequence Coverage

Although combinatorial testing has been widely studied and used, there are still some situations and requirements that combinatorial testing does not apply to well, such as a system under test whose test cases need to be performed contiguously. For thorough testing, the testing requirements are not only to cover all the interactions among factors but also to cover all the value sequences of every factor. Generally, systems under test always involve constraints and dependencies in or among test cases. The constraints among test cases have not been effectively specified. First, we introduce extended covering arrays that can achieve both t-way combinatorial coverage and t-wise sequence coverage, and propose a clocked computation tree logic-based formal specification method for specifying constraints. Then, Particle Swarm Optimization (PSO) based Extended covering array Generator (PEG) is elaborated. To evaluate the constructed test suites, a method for verifying the constraints’ validity is presented, and kernel functions for measuring the coverage are also introduced. Finally, the performance of the proposed PEG is evaluated using several sets of benchmark experiments for some common constraints, and the feasibility and usefulness of PEG is validated.

Covering arrays of strength three from extended permutation vectors

A covering array $$\text{ CA }(N;t,k,v)$$ is an $$N\times k$$ array such that in every $$N\times t$$ subarray each possible t-tuple over a v-set appears as a row of the subarray at least once. The integers t and v are respectively the strength and the order of the covering array. Let v be a prime power and let $${\mathbb {F}}_v$$ denote the finite field with v elements. In this work the original concept of permutation vectors generated by a $$(t-1)$$ -tuple over $${\mathbb {F}}_v$$ is extended to vectors generated by a t-tuple over $${\mathbb {F}}_v$$ . We call these last vectors extended permutation vectors (EPVs). For every prime power v, a covering perfect hash family $$\text{ CPHF }(2;v^2-v+3,v^3,3)$$ is constructed from EPVs given by subintervals of a linear feedback shift register sequence over $${\mathbb {F}}_v$$ . When $$v\in \{7,9,11,13,16,17,19,23,25\}$$ the covering array $$\text{ CA }(2v^3-v;3,v^2-v+3,v)$$ generated by $$\text{ CPHF }(2;v^2-v+3,v^3,3)$$ has less rows than the best-known covering array with strength three, $$v^2-v+3$$ columns, and order v. CPHFs formed by EPVs are also constructed using simulated annealing; in this case the results improve the size of eighteen covering arrays of strength three.

New optimal covering arrays using an orderly algorithm

A covering array [Formula: see text] is an [Formula: see text] array such that every [Formula: see text] subarray covers at least once each [Formula: see text]-tuple from [Formula: see text] symbols. For given [Formula: see text], [Formula: see text], and [Formula: see text], the minimum number of rows for which exists a CA is denoted by [Formula: see text] (CAN stands for Covering Array Number) and the corresponding CA is optimal. Optimal covering arrays have been determined algebraically for a small subset of cases; but another alternative to find CANs is the use of computational search. The present work introduces a new orderly algorithm to construct non-isomorphic covering arrays; this algorithm is an improvement of a previously reported algorithm for the same purpose. The construction of non-isomorphic covering arrays is used to prove the nonexistence of certain covering arrays whose nonexistence implies the optimality of other covering arrays. From the computational results obtained, the following CANs were established: [Formula: see text] for [Formula: see text], [Formula: see text], and [Formula: see text]. In addition, the new result [Formula: see text], and the already known existence of [Formula: see text], imply [Formula: see text].

차 커버링 배열을 이용한 복제 기반 분산 저장 기술

차 커버링 배열을 이용한 복제 기반 분산 저장 기술

Optimal shortening of uniform covering arrays.

Software test suites based on the concept of interaction testing are very useful for testing software components in an economical way. Test suites of this kind may be created using mathematical objects called covering arrays. A covering array, denoted by CA(N; t, k, v), is an N × k array over with the property that every N × t sub-array covers all t-tuples of at least once. Covering arrays can be used to test systems in which failures occur as a result of interactions among components or subsystems. They are often used in areas such as hardware Trojan detection, software testing, and network design. Because system testing is expensive, it is critical to reduce the amount of testing required. This paper addresses the Optimal Shortening of Covering ARrays (OSCAR) problem, an optimization problem whose objective is to construct, from an existing covering array matrix of uniform level, an array with dimensions of (N − δ) × (k − Δ) such that the number of missing t-tuples is minimized. Two applications of the OSCAR problem are (a) to produce smaller covering arrays from larger ones and (b) to obtain quasi-covering arrays (covering arrays in which the number of missing t-tuples is small) to be used as input to a meta-heuristic algorithm that produces covering arrays. In addition, it is proven that the OSCAR problem is NP-complete, and twelve different algorithms are proposed to solve it. An experiment was performed on 62 problem instances, and the results demonstrate the effectiveness of solving the OSCAR problem to facilitate the construction of new covering arrays.

Improved Strength Four Covering Arrays with Three Symbols

A covering array t-CA(n, k, g), of size n, strength t, degree k, and order g, is a $$k\times n$$ array on g symbols such that every $$t\times n$$ sub-array contains every $$t\times 1$$ column on g symbols at least once. Covering arrays have been studied for their applications to software testing, hardware testing, drug screening, and in areas where interactions of multiple parameters are to be tested. In this paper, we present an algebraic construction that improves many of the best known upper bounds on n for covering arrays 4-CA(n, k, 3). The t-coverage of a testing array $$\mathscr {A}$$ is the number of distinct t-tuples contained in the column vectors of $$\mathscr {A}$$ divided by the total number of t-tuples. If the testing array is a covering array of strength t, its t-coverage is 100%. The covering arrays with budget constraints problem is the problem of constructing a testing array of size at most n having largest possible coverage, given values of t, k, g and n. This paper also presents several testing arrays with high 4-coverage.

An algorithm for combinatorial interaction testing: definitions and rigorous evaluations

BackgroundCombinatorial Interaction Testing (CIT) approaches have drawn attention of the software testing community to generate sets of smaller, efficient, and effective test cases where they have been successful in detecting faults due to the interaction of several input parameters. Recent empirical studies show that greedy algorithms are still competitive for CIT. It is thus interesting to investigate new approaches to address CIT test case generation via greedy solutions and to perform rigorous evaluations within the greedy context.MethodsWe present a new greedy algorithm for unconstrained CIT, T-TupleReallocation (TTR), to generate CIT test suites specifically via the Mixed-value Covering Array (MCA) technique. The main reasoning behind TTR is to generate an MCA M by creating and reallocating t-tuples into this matrix M, considering a variable called goal (ζ). We performed two controlled experiments addressing cost-efficiency and only cost. Considering both experiments, we did 3200 executions related to 8 solutions. In the first controlled experiment, we compared versions 1.1 and 1.2 of TTR in order to check whether there is significant difference between both versions of our algorithm. In such experiment, we jointly considered cost (size of test suites) and efficiency (time to generate the test suites) in a multi-objective perspective. In the second controlled experiment we confronted TTR 1.2 with five other greedy algorithms/tools for unconstrained CIT: IPOG-F, jenny, IPO-TConfig, PICT, and ACTS. We performed two different evaluations within this second experiment where in the first one we addressed cost-efficiency (multi-objective) and in the second only cost (single objective).ResultsResults of the first controlled experiment indicate that TTR 1.2 is more adequate than TTR 1.1 especially for higher strengths (5, 6). In the second controlled experiment, TTR 1.2 also presents better performance for higher strengths (5, 6) where only in one case it is not superior (in the comparison with IPOG-F). We can explain this better performance of TTR 1.2 due to the fact that it no longer generates, at the beginning, the matrix of t-tuples but rather the algorithm works on a t-tuple by t-tuple creation and reallocation into M.ConclusionConsidering the metrics we defined in this work and based on both controlled experiments, TTR 1.2 is a better option if we need to consider higher strengths (5, 6). For lower strengths, other solutions, like IPOG-F, may be better alternatives.

Mixed covering arrays on 3-uniform hypergraphs

Covering arrays are combinatorial objects that have been successfully applied in the design of test suites for testing systems such as software, circuits and networks, where failures can be caused by the interaction between their parameters. Let n and k be positive integers with k≥3. Three vectors x∈Zg1n, y∈Zg2n, z∈Zg3n are 3-qualitatively independent if for any triple (a,b,c)∈Zg1×Zg2×Zg3, there exists an index j∈{1,2,…,n} such that (x(j),y(j),z(j))=(a,b,c). Let H be a 3-uniform hypergraph with k vertices v1,v2,…,vk with respective vertex weights g1,g2,…,gk. A mixed covering array on H, denoted by CA(n,H,∏i=1kgi), is a k×n array such that row i corresponds to vertex vi, entries in row i are from Zgi; and if {vx,vy,vz} is a hyperedge in H, then the rows x,y,z are 3-qualitatively independent. The parameter n is called the size of the array. Given a weighted 3-uniform hypergraph H, a mixed covering array on H with minimum size is called optimal. In this article, we introduce four basic hypergraph operations to construct optimal mixed covering arrays on hypergraphs. Using these operations, we provide constructions for optimal mixed covering arrays on the family of 2-tree hypergraphs, α-acyclic 3-uniform hypergraphs, conformal 3-uniform hypertrees having a binary tree as host tree, and 3-uniform loose cycle hypergraphs.

A Hybrid Artificial Bee Colony and Harmony Search Algorithm to Generate Covering Arrays for Pair-wise Testing

A Hybrid Artificial Bee Colony and Harmony Search Algorithm to Generate Covering Arrays for Pair-wise Testing

Partial Covering Arrays: Algorithms and Asymptotics

A covering array CA(N;t, k, v) is an N × k array with entries in {1,2,…, v}, for which every N × t subarray contains each t-tuple of {1,2,…, v} t among its rows. Covering arrays find application in interaction testing, including software and hardware testing, advanced materials development, and biological systems. A central question is to determine or bound CAN(t, k, v), the minimum number N of rows of a CA(N;t, k, v). The well known bound CAN(t, k, v) = O((t − 1)v t log k) is not too far from being asymptotically optimal. Sensible relaxations of the covering requirement arise when (1) the set {1,2,…, v} t need only be contained among the rows of at least $(1-\epsilon )\binom {k}{t}$ of the N × t subarrays and (2) the rows of every N × t subarray need only contain a (large) subset of {1,2,…, v} t . In this paper, using probabilistic methods, significant improvements on the covering array upper bound are established for both relaxations, and for the conjunction of the two. In each case, a randomized algorithm constructs such arrays in expected polynomial time.

Metaheuristic post-optimization of the NIST repository of covering arrays

Construction of Covering Arrays (CA) with minimum possible number of rows is challenging. Often the available CA have redundant combinatorial interaction that could be removed to reduce the number of rows. This paper addresses the problem of removing redundancy of CA using a metaheuristic post-optimization (MPO) approach. Our approach consists of three main components: a redundancy detector (RD); a row reducer (RR); and a missing-combinations reducer (MCR). The MCR is a metaheuristic component implemented using a simulated annealing algorithm. MPO was instantiated with 21,964 CA taken from the National Institute of Standards and Technology (NIST) repository. It is a remarkable result that this instantiation of MPO has delivered 349 new upper bounds for these CA.

Selecting the Best Wild Card Entries in a Covering Array

For some covering arrays, there are some wild card positions in the array where there is flexibility about what factor levels can be chosen with no impact on the basic properties of covering array, because all of the required pairs have been covered. The choice of how to fill these wild card positions can influence other properties, such as the degrees of orthogonality or three‐way coverage of the array. In this paper, some criteria are proposed for identifying the best choices for the wild card positions to create covering arrays with highly desirable properties. Accompanying graphical summaries are also described to highlight differing performance for several examples. Copyright © 2017 John Wiley & Sons, Ltd.

Asymptotic Size of Covering Arrays: An Application of Entropy Compression

AbstractA covering array is an array A such that each cell of A takes a value from a v‐set V, which is called the alphabet. Moreover, the set is contained in the set of rows of every subarray of A. The parameter N is called the size of an array and denotes the smallest N for which a exists. It is well known that [10]. In this paper, we derive two upper bounds on using an algorithmic approach to the Lovász local lemma also known as entropy compression.

Upper Bounds on the Size of Covering Arrays

Covering arrays find important application in software and hardware interaction testing. For practical applications it is useful to determine or bound the minimum number of rows, $\mathsf{CAN}(t,k,v)$, in a covering array for given values of the parameters $t,k$, and $v$. Asymptotic upper bounds for $\mathsf{CAN}(t,k,v)$ have been established using the Stein--Lovasz--Johnson strategy and the Lovasz local lemma. A series of improvements on these bounds is developed in this paper. First an estimate for the discrete Stein--Lovasz--Johnson bound is derived. Then using alteration, the Stein--Lovasz--Johnson bound is improved upon, leading to a two-stage construction algorithm. Bounds from the Lovasz local lemma are improved upon in a different manner, by examining group actions on the set of symbols. Two asymptotic upper bounds on $\mathsf{CAN}(t,k,v)$ are established that are tighter than the known bounds. A two-stage bound is derived that employs the Lovasz local lemma and the conditional Lovasz local lemma ...

Tuning the Parameters of a Convolutional Artificial Neural Network by Using Covering Arrays

Tuning the Parameters of a Convolutional Artificial Neural Network by Using Covering Arrays

Covering arrays from m-sequences and character sums

A covering array of strength t on v symbols is an array with the property that, for every t-set of column vectors, every one of the $$v^t$$ possible t-tuples of symbols appears as a row at least once in the sub-array defined by these column vectors. Arrays constructed using m-sequences over a finite field possess many combinatorial properties and have been used to construct various combinatorial objects; see the recent survey Moura et al. (Des Codes Cryptogr 78(1):197–219, 2016). In this paper we construct covering arrays whose elements are the remainder of the division by some integer of the discrete logarithm applied to selected m-sequence elements. Inspired by the work of Colbourn (Des Codes Cryptogr 55(2–3):201–219, 2010), we prove our results by connecting the covering array property to a character sum, and we evaluate this sum by taking advantage of the balanced way in which the m-sequence elements are distributed. Our results include new infinite families of covering arrays of arbitrary strength.