Failure-detection Effectiveness Research Articles

VPP-ART: An Efficient Implementation of Fixed-Size-Candidate-Set Adaptive Random Testing Using Vantage Point Partitioning

<italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">Adaptive random testing</i> (ART) is an enhancement of <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">random testing</i> (RT), and aims to improve the RT failure-detection effectiveness by distributing test cases more evenly in the input domain. Many ART algorithms have been proposed, with <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">fixed-size-candidate-set</i> ART (FSCS-ART) being one of the most effective and popular. FSCS-ART ensures high failure-detection effectiveness by selecting as the next test case the candidate farthest from previously executed test cases. Although FSCS-ART has good failure-detection effectiveness, it also faces some challenges, including heavy computational overheads. In this article, we propose an enhanced version of FSCS-ART, <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">vantage point partitioning ART</i> (VPP-ART). VPP-ART addresses the FSCS-ART computational overhead problem using VPP, while maintaining the failure-detection effectiveness. VPP-ART partitions the input domain space using a <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">modified vantage point tree</i> (VP-tree) and finds the approximate nearest executed test cases of a candidate test case in the partitioned subdomains—thereby significantly reducing the time overheads compared with the searches required for FSCS-ART. To enable the FSCS-ART dynamic insertion process, we modify the traditional VP-tree to support dynamic data. The simulation results show that VPP-ART has a much lower time overhead compared to FSCS-ART, but also delivers similar (or better) failure-detection effectiveness, especially in the higher dimensional input domains. According to statistical analyses, VPP-ART can improve on the FSCS-ART failure-detection effectiveness by approximately 50–58%. VPP-ART also compares favorably with the <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">KD-tree-enhanced fixed-size-candidate-set ART</i> (KDFC-ART) algorithms (a series of enhanced ART algorithms based on the KD-tree). Our experiments also show that VPP-ART is more cost-effective than FSCS-ART and KDFC-ART.

A nearest-neighbor divide-and-conquer approach for adaptive random testing

• We propose a new ART approach to enhance testing efficacy and efficiency of FSCS- ART. • We report on simulations and empirical studies to investigate the proposed method. • Compared with FSCS-ART, our proposed method is more cost-effective. • Compared with other ART algorithms, our method has similar or even better results. • Our proposed method can alleviate the boundary effect problem of FSCS-ART. Adaptive Random Testing (ART) aims at enhancing the failure detection capability of Random Testing (RT) by evenly distributing test cases over the input domain. Many ART algorithms have been proposed to achieve an even spread of test cases, according to different notations. A well-known ART algorithm, namely Fixed-Size-Candidate-Set ART (FSCS-ART), chooses an element as the next test case such that it is farthest away from previously executed test cases. Previous studies have demonstrated that FSCS-ART could achieve good improvements in test effectiveness over RT, but suffers from some drawbacks such as high computational overhead and boundary effect problem. In this paper, we propose an alternative approach to enhance FSCS-ART, namely Nearest-Neighbor Divide-and-Conquer based ART (NNDC-ART), which combines the concepts of nearest-neighbor and divide-and-conquer . It attempts to reduce the computational overhead of test case generation, and also to alleviate the boundary effect problem; while maintaining the advantages of FSCS-ART. The simulation results show that compared with FSCS-ART, NNDC-ART requires much less computational overhead, and also achieves much better test case distribution; while maintaining better or comparable failure-detection effectiveness. Our empirical studies further show that the proposed approach is much more cost-effective than FSCS-ART. In addition, we further compare NNDC-ART against two enhancements of FSCS-ART based on the partitioning strategy, i.e., ART with Divide-and-Conquer (ART-DC) and ART by Bisection and Localization (ART-B-Loc), and find that our proposed approach can achieve similar or even better testing performances in some scenarios.

Candidate test set reduction for adaptive random testing: An overheads reduction technique

• We propose a cost-effective ART approach, FSCS-CTSR, to improve the efficiency of FSCS. • We examine the effectiveness and efficiency of FSCS-CTSR through simulations and experimental studies. • Compared with FSCS, FSCS-CTSR achieves similar effectiveness to FSCS, but significantly reduces the computational costs. Adaptive Random Testing (ART) is a family of testing techniques that were proposed as an enhancement of random testing (RT). ART achieves better failure-detection capability than RT by more evenly distributing test cases throughout the input domain. However, this process of selecting more diverse test cases incurs a heavy computational cost. In this paper, we propose a new ART method that improves on the efficiency of Fixed-Size-Candidate-Set ART (FSCS) by applying a test set reduction strategy. The proposed method, FSCS by Candidate Test Set Reduction (FSCS-CTSR), reduces the number of randomly generated candidate test cases, but supplements them with earlier, unused candidates that have lower similarity to the executed test cases. Simulations and experimental studies were conducted to examine the effectiveness and efficiency of the method, with the experimental results showing a comparable failure-detection effectiveness to FSCS, but with lower computational costs.

SWFC-ART: A cost-effective approach for Fixed-Size-Candidate-Set Adaptive Random Testing through small world graphs

Adaptive random testing (ART) improves the failure-detection effectiveness of random testing by leveraging properties of the clustering of failure-causing inputs of most faulty programs: ART uses a sampling mechanism that evenly spreads test cases within a software’s input domain. The widely-used Fixed-Sized-Candidate-Set ART (FSCS-ART) sampling strategy faces a quadratic time cost, which worsens as the dimensionality of the software input domain increases. In this paper, we propose an approach based on small world graphs that can enhance the computational efficiency of FSCS-ART: SWFC-ART. To efficiently perform nearest neighbor queries for candidate test cases, SWFC-ART incrementally constructs a hierarchical navigable small world graph for previously executed, non-failure-causing test cases. Moreover, SWFC-ART has shown consistency in programs with high dimensional input domains. Our simulation and empirical studies show that SWFC-ART reduces the computational overhead of FSCS-ART from quadratic to log-linear order while maintaining the failure-detection effectiveness of FSCS-ART, and remaining consistent in high dimensional input domains. We recommend using SWFC-ART in practical software testing scenarios, where real-life programs often have high dimensional input domains and low failure rates.

Adaptive random testing based on flexible partitioning

Adaptive random testing (ART) achieves better failure-detection effectiveness than random testing due to its even spreading of test cases. ART by random partitioning (RP-ART) is a lightweight method, but its advantage over random testing is relatively low. Although iterative partition testing (IPT) method has good performance for detecting failures in a block pattern, it loses randomness during the test case generation. To overcome the shortcomings of the above two algorithms, a new algorithm named ART by flexible partitioning (FP-ART) is proposed. In the FP-ART, a set of random candidates is used to select an appropriate test case by considering their boundary distance. Accordingly, the corresponding sub-domain is also partitioned by the new test case. Based on this kind of flexible partitioning, the randomness of test case selection can be guaranteed and the spatial distribution of test cases is even more diverse. According to the results in simulation and empirical experiments, FP-ART demonstrates better failure-detection effectiveness than RP-ART and is more suitable to detect the failures in strip patterns than the IPT method. Meanwhile, its failure-detection ability is much stronger than that of fixed-size-candidate-set ART in the cases of a relatively high failure rate.

Effective test data generation using probabilistic networks

This paper presents a novel test data generation method called Bayes-TDG. It is based on principles of Bayesian networks (BNs) and provides the possibility of making inference from probabilistic data in the model to increase the prime path coverage ratio (PPCR) for a given program under test (PUT). In this regard, a new program structure-based probabilistic network, TDG-NET, is proposed that is capable of modelling the conditional dependencies among the program basic blocks (BBs) in one hand and conditional dependencies of the transitions between its BBs and input parameters on the other hand. To achieve failure-detection effectiveness, we propose a path selection strategy that works based on the predicted outcome of generated test cases. So, we mitigate the need for a human oracle, and the generated test suite could be directly used in fault localisation. Several experiments are conducted to evaluate the performance of Bayes-TDG. The results reveal that the method is promising and the generated test suite could be quite effective.

KDFC-ART: a KD-tree approach to enhancing Fixed-size-Candidate-set Adaptive Random Testing

Adaptive random testing (ART) was developed as an enhanced version of random testing to increase the effectiveness of detecting failures in programs by spreading the test cases evenly over the input space. However, heavy computation may be incurred. In this paper, three enhanced algorithms for fixed-size-candidate-set ART (FSCS-ART) are proposed based on the $k$ -dimensional tree (KD-tree) structure. The first algorithm Naive-KDFC constructs a KD-tree by splitting the input space with respect to every dimension successively in a round-robin fashion. The second algorithm SemiBal-KDFC improves the balance of the KD-tree by prioritizing the splitting according to the spread in each dimension. In order to control the number of traversed nodes in backtracking, the third algorithm LimBal-KDFC introduces an upper bound for the nodes involved. Simulation and empirical studies have been conducted to investigate the efficiency and effectiveness of the three algorithms. The experimental results show that these algorithms significantly reduce the computation time of the original FSCS-ART for low dimensions and for the case of high dimensions with low failure rates. The efficiency of SemiBal-KDFC is better than that of Naive-KDFC when the dimension is no more than 8, but LimBal-KDFC is the most efficient of all three. Although the limited backtracking leads only to an approximate nearest neighbor in LimBal-KDFC, its failure-detection effectiveness is, in fact, better than FSCS-ART in high-dimensional input spaces and has no significant deterioration in low-dimensional spaces.

An Enhanced Adaptive Random Testing by Dividing Dimensions Independently

Random testing (RT) is widely applied in the area of software testing due to its advantages such as simplicity, unbiasedness, and easy implementation. Adaptive random testing (ART) enhances RT. It improves the effectiveness of RT by distributing test cases as evenly as possible. Fixed Size Candidate Set (FSCS) is one of the most well-known ART algorithms. Its high failure-detection effectiveness only shows at low failure rates in low-dimensional spaces. In order to solve this problem, the boundary effect of the test case distribution is analyzed, and the FSCS algorithm of a limited candidate set (LCS-FSCS) is proposed. By utilizing the information gathered from success test cases (no failure-causing test inputs), a tabu generation domain of candidate test case is produced. This tabu generation domain is eliminated from the current candidate test case generation domain. Finally, the number of test cases at the boundary is reduced by constraining the candidate test case generation domain. The boundary effect is effectively relieved, and the distribution of test cases is more even. The results of the simulation experiment show that the failure-detection effectiveness of LCS-FSCS is significantly improved in high-dimensional spaces. Meanwhile, the failure-detection effectiveness is also improved for high failure rates and the gap of failure-detection effectiveness between different failure rates is narrowed. The results of an experiment conducted on some real-life programs show that LCS-FSCS is less effective than FSCS only when the failure distribution is concentrated on the boundary. In general, the effectiveness of LCS-FSCS is higher than that of FSCS.

A Modified Similarity Metric for Unit Testing of Object-Oriented Software Based on Adaptive Random Testing

Finding an effective method for testing object-oriented software (OOS) has proven elusive in the software community due to the rapid development of object-oriented programming (OOP) technology. Although significant progress has been made by previous studies, challenges still exist in relation to the object distance measurement of OOS using Adaptive Random Testing (ART). This is partly due to the unique features of OOS such as encapsulation, inheritance and polymorphism. In a previous work, we proposed a new similarity metric called the Object and Method Invocation Sequence Similarity (OMISS) metric to facilitate multi-class level testing using ART. In this paper, we broaden the set of models in the metric (OMISS) by considering the method parameter and adding the weight in the metric to develop a new distance metric to improve unit testing of OOS. We used the new distance metric to calculate the distance between the set of objects and the distance between the method sequences of the test cases. Additionally, we integrate the new metric in unit testing with ART and applied it to six open source subject programs. The experimental result shows that the proposed method with method parameter considered in this study is better than previous methods without the method parameter in the case of the single method. Our finding further shows that the proposed unit testing approach is a promising direction for assisting software engineers who seek to improve the failure-detection effectiveness of OOS testing.

A cost‐effective adaptive random testing approach by dynamic restriction

A key objective of software testing is to find program errors that cause failure in software, at less cost. One basic testing technique is random testing (RT), but many researchers have criticised its failure-detection effectiveness. Several researchers have proposed that an enhancement of the failure-detection effectiveness of RT is achieved if test cases are evenly spread within the input domain. Adaptive RT (ART) describes a family of algorithms that employ various strategies to evenly and randomly spread test cases. Fixed sized candidate set ART (FSCS-ART) is an ART algorithm that has gained many research studies far and wide; however, the high distance computations make its algorithm computationally expensive. The authors propose a new ART method that restricts distance computations to only test cases inside an exclusion zone. The experimental results show that the new ART method not only improves RT but also provides failure-detection effectiveness similar to FSCS-ART, while significantly minimising computation overhead.

Adaptive Random Testing in Detecting Layout Faults of Web Applications

As part of a software testing process, output verification poses a challenge when the output is not numeric or textual, such as graphical. The industry practice of using human oracles (testers) to observe and verify the correctness of the actual results is both expensive and error-prone. In particular, this practice is usually unsustainable when developing web applications — the most popular software of our era. This is because web applications change frequently due to the fast-evolving requirements amid popular demand. To improve the cost effectiveness of browser output verification, in this study we design failure-based testing techniques and evaluate the effectiveness and efficiency thereof in the context of web testing. With a novel application of the concept of adaptive random sequence (ARS), our approach leverages peculiar characteristics of failure patterns found in browser layout rendering. An empirical study shows that the use of failure patterns and inclination to guide the testing flow leads to more cost-effective results than other classic methods. This study extends the application of ARSs from the input space of programs to their output space, and also shows that adaptive random testing (ART) can outperform random testing (RT) in both failure detection effectiveness (in terms of F-measure) and failure detection efficiency (in terms of execution time).

Bayes‐TDG: effective test data generation using Bayesian belief network: toward failure‐detection effectiveness and maximum coverage

This study presents a novel test data generation method called Bayes-TDG. It is based on principles of Bayesian networks and provides the possibility of making inference from probabilistic data in the model to increase the prime path-coverage ratio for a given programme under test (PUT). In this regard, a new programme structure-based probabilistic network, TDG-NET, is proposed that is capable of modelling the conditional dependencies among the programme basic blocks (BBs) on one hand and conditional dependencies of the transitions between its BBs and input parameters on the other hand. To achieve failure-detection effectiveness, the authors propose a path selection strategy that works based on the predicted outcome of generated test cases. So, they mitigate the need for a human oracle, and the generated test suite could be directly used in fault localisation. Several experiments are conducted to evaluate the performance of Bayes-TDG. The results reveal that the method is promising and the generated test suite could be quite effective.

Out of sight, out of mind: a distance-aware forgetting strategy for adaptive random testing

Adaptive random testing (ART) achieves better failure-detection effectiveness than random testing by increasing the diversity of test cases. However, the intention of ensuring even spread of test cases inevitably causes an overhead problem. Although two basic forgetting strategies (i.e. random forgetting and consecutive retention) were proposed to reduce the computation cost of ART, they only considered the temporal distribution of test cases. In the paper, we presented a distance-aware forgetting strategy for the fixed size candidate set version of ART (DF-FSCS), in which the spatial distribution of test cases is taken into consideration. For a given candidate, the test cases out of its “sight” are ignored to reduce the distance computation cost. At the same time, the dynamic adjustment for partitioning and the second-round forgetting are adopted to ensure the linear complexity of DF-FSCS algorithm. Both simulation analysis and empirical study are employed to investigate the efficiency and effectiveness of DF-FSCS. The experimental results show that DF-FSCS significantly outperforms the classical ART algorithm FSCS-ART in efficiency, and has comparable failure-detection effectiveness. Com-pared with two basic forgetting methods, DF-FSCS is better in both efficiency and effectiveness. In contrast with a typical linear-time ART algorithm RBCVT-Fast, our algorithm requires less computational overhead and exhibits the similar failure-detection capability. In addition, DF-FSCS has more reliable performance than RBCVT-Fast in detecting failures for the programs with high-dimensional input domain.

Randomized Quasi-Random Testing

Random testing is a fundamental testing technique that can be used to generate test cases for both hardware and software systems. Quasi-random testing was proposed as an enhancement to the cost-effectiveness of random testing: In addition to having similar computation overheads to random testing, it makes use of quasi-random sequences to generate low-discrepancy and low-dispersion test cases that help deliver high failure-detection effectiveness. Currently, few algorithms exist to generate quasi-random sequences, and these are mostly deterministic, rather than random. A previous study of quasi-random testing has examined two methods for randomizing quasi-random sequences to improve their applicability in testing. However, these randomization methods still have shortcomings—one method does not introduce much randomness to the test cases, while the other does not support incremental test case generation. In this paper, we present an innovative approach to incrementally randomizing quasi-random sequences. The test cases generated by this new approach show a high degree of randomness and evenness in distribution. We also conduct simulations and empirical studies to demonstrate the applicability and effectiveness of our approach in software testing.

A Cost-Effective Random Testing Method for Programs with Non-Numeric Inputs

Random testing (RT) has been widely used in the testing of various software and hardware systems. Adaptive random testing (ART) is a family of random testing techniques that aim to enhance the failure-detection effectiveness of RT by spreading random test cases evenly throughout the input domain. ART has been empirically shown to be effective on software with numeric inputs. However, there are two aspects of ART that need to be addressed to render its adoption more widespread—applicability to programs with non-numeric inputs, and the high computation overhead of many ART algorithms. We present a linear-order ART algorithm for software with non-numeric inputs. The key requirement for using ART with non-numeric inputs is an appropriate “distance” measure. We use the concepts of categories and choices from category-partition testing to formulate such a measure. We investigate the failure-detection effectiveness of our technique by performing an empirical study on 14 object programs, using two standard metrics—F-measure and P-measure. Our ART algorithm statistically significantly outperforms RT on 10 of the 14 programs studied, and exhibits performance similar to RT on three of the four remaining programs. The selection overhead of our ART algorithm is close to that of RT.

Enhancing mirror adaptive random testing through dynamic partitioning

ContextAdaptive random testing (ART), originally proposed as an enhancement of random testing, is often criticized for the high computation overhead of many ART algorithms. Mirror ART (MART) is a novel approach that can be generally applied to improve the efficiency of various ART algorithms based on the combination of “divide-and-conquer” and “heuristic” strategies. ObjectiveThe computation overhead of the existing MART methods is actually on the same order of magnitude as that of the original ART algorithms. In this paper, we aim to further decrease the order of computation overhead for MART. MethodWe conjecture that the mirroring scheme in MART should be dynamic instead of static to deliver a higher efficiency. We thus propose a new approach, namely dynamic mirror ART (DMART), which incrementally partitions the input domain and adopts new mirror functions. ResultsOur simulations demonstrate that the new DMART approach delivers comparable failure-detection effectiveness as the original MART and ART algorithms while having much lower computation overhead. The experimental studies further show that the new approach also delivers a better and more reliable performance on programs with failure-unrelated parameters. ConclusionIn general, DMART is much more cost-effective than MART. Since its mirroring scheme is independent of concrete ART algorithms, DMART can be generally applied to improve the cost-effectiveness of various ART algorithms.

Enhancing adaptive random testing for programs with high dimensional input domains or failure-unrelated parameters

Adaptive random testing (ART), an enhancement of random testing (RT), aims to both randomly select and evenly spread test cases. Recently, it has been observed that the effectiveness of some ART algorithms may deteriorate as the number of program input parameters (dimensionality) increases. In this article, we analyse various problems of one ART algorithm, namely fixed-sized-candidate-set ART (FSCS-ART), in the high dimensional input domain setting, and study how FSCS-ART can be further enhanced to address these problems. We propose to add a filtering process of inputs into FSCS-ART to achieve a more even-spread of test cases and better failure detection effectiveness in high dimensional space. Our study shows that this solution, termed as FSCS-ART-FE, can improve FSCS-ART not only in the case of high dimensional space, but also in the case of having failure-unrelated parameters. Both cases are common in real life programs. Therefore, we recommend using FSCS-ART-FE instead of FSCS-ART whenever possible. Other ART algorithms may face similar problems as FSCS-ART; hence our study also brings insight into the improvement of other ART algorithms in high dimensional space.

Quasi-Random Testing

Our paper proposes an implementable procedure for using the method of quasi-random sequences in software debug testing. In random testing, the sequence of tests (if considered as points in an -dimensional unit hypercube) will give rise to regions where there are clusters of points, as well as underpopulated regions. Quasi-random sequences, also known as low-discrepancy or low-dispersion sequences, are sequences of points in such a hypercube that are spread more evenly throughout. Based on the observation that program faults tend to lead to contiguous failure regions within a program's input domain, and that an even spread of random tests enhances the failure detection effectiveness for certain failure patterns, we examine the use of quasi-random sequences as a replacement for random sequences in automated testing. Because there are only a small number of quasi-random sequence generation algorithms, and each of them can only generate a small number of distinct sequences, the applicability of quasi-random sequences in testing real programs is severely restricted. To alleviate this problem, we examine the use of two types of randomized quasi-random sequences, which are quasi-random sequences permuted in a nondeterministic fashion in such a way as to retain their low discrepancy properties. We show that testing using randomized quasi-random sequences is often significantly more effective than random testing.