Adaptive Random Testing Algorithms Research Articles

L[formula omitted]OP-ART: A linear-time adaptive random testing algorithm for object-oriented programs

Object-oriented (OO) programming offers useful and desirable characteristics in the resulting code. These OO characteristics, however, demand an effective testing method. Adaptive random testing (ART) serves as an effective testing technique where test cases are distributed uniformly according to a proper metric for difference (distance) between test cases. Recently proposed object and method invocation sequence similarity (OMISS) distance metric was incorporated into a new ART algorithm (OMISS-ART) for testing OO software. Unfortunately, OMISS-ART’s high effectiveness in finding faults comes at significantly higher computational overheads. To alleviate the problem, we present in this paper a linear-time ART algorithm called L′OP-ART (Linear-time Object-oriented Program testing by ART) based on a lightweight OMISS metric. L′OP-ART designs specific data structures that store information about executed test cases, thus, the differences between candidates and executed test cases can be obtained with reduced overheads. Empirical studies show that L′OP-ART detects failures more effectively than random testing (RT), and has comparable fault-detection effectiveness to other ART algorithms for OO testing (according to the F-measure, or the expected number of test case executions required to find a failure). Furthermore, the test case generation overhead of L′OP-ART is low, close to that of RT.

A response coordination design technology for the main loaded structures of the internal combustion engines based on NSEHO

AbstractResponse coordination design becomes an inevitable trend based on the multidisciplinary optimization design (MOD) in the development of the design for complex mechanical structures. Aiming at the problem of unclear mechanism of response coordination for main loaded structures, a response coordination design technology characterized by a response coordination evaluation system is proposed based on an analytic hierarchy process. In addition, a nondominated sorting elephant herding optimization (NSEHO) algorithm is proposed by introducing the Pareto optimization theory into elephant herding optimization and combining fixed‐sized candidate set adaptive random testing algorithm to overcome inherent limitations of the traditional algorithm in response coordination design. Taking the main bearing assembly structure as an example, the response coordination design is conducted based on the proposed technology and NSEHO algorithm. The response coordination design scheme of the main bearing assembly structure is experimentally verified. The results show that the Pareto frontier of NSEHO is smoother and is closer to the real Pareto frontier compared to NSGA‐II in condition of small population size and restricted iteration generations, revealing the advancement of the solving ability of NSEHO. The coordination coefficients of strength, stiffness, contact strength, and mass of the main bearing assembly structure are increased by 13.91%, 14.96%, 2.63%, and 0.07%, respectively, and the overall coordination coefficient are increased by 11.10%. The indicators of strength, stiffness, and contact strength of the response coordination scheme are better than those of the original scheme by experimental verification, demonstrating the effectiveness of the response coordination design technology. The proposed technology reveals the response coordination mechanism of the main loaded structures of internal combustion engines and provides an important guiding significance to MOD for complex mechanical structures.

A Novel Test Case Generation Approach for Adaptive Random Testing of Object-Oriented Software Using K-Means Clustering Technique

Random testing (RT) is considered important owing to the popularity of fuzzing techniques. Yet, its effectiveness has been questioned because it disregards the property that failures are usually conglomerated. Adaptive random testing (ART) has been developed to spread the test cases more evenly. A set of candidate test cases is first generated randomly. Then, the candidate farthest from the executed test cases is selected as the next test case. However, this process is time-consuming, especially for object-oriented programs. In order to improve the efficiency, a forgetting strategy may be added, taking into account only part of the executed test cases. Of course, the failure detection capability is compromised. In this paper, we propose a new approach that improves the efficiency of object-oriented program testing through K-means clustering. Frequency transform and improved wavelet transform are adopted to better determine the dissimilarity of test cases. Two measures, namely, the wavelet transform (WT) metric and the trisection frequency conversion (TFC) metric, are proposed. Based on the metrics, we develop two algorithms called WClustering-ART and TFClustering-ART. Experimental studies and statistical analysis have been conducted to evaluate the F-measure and the Fm-time of our approach. The results show that both algorithms demonstrate better effectiveness and efficiency than RT and object-oriented ART algorithms with forgetting.

A nearest-neighbor divide-and-conquer approach for adaptive random testing

• We propose a new ART approach to enhance testing efficacy and efficiency of FSCS- ART. • We report on simulations and empirical studies to investigate the proposed method. • Compared with FSCS-ART, our proposed method is more cost-effective. • Compared with other ART algorithms, our method has similar or even better results. • Our proposed method can alleviate the boundary effect problem of FSCS-ART. Adaptive Random Testing (ART) aims at enhancing the failure detection capability of Random Testing (RT) by evenly distributing test cases over the input domain. Many ART algorithms have been proposed to achieve an even spread of test cases, according to different notations. A well-known ART algorithm, namely Fixed-Size-Candidate-Set ART (FSCS-ART), chooses an element as the next test case such that it is farthest away from previously executed test cases. Previous studies have demonstrated that FSCS-ART could achieve good improvements in test effectiveness over RT, but suffers from some drawbacks such as high computational overhead and boundary effect problem. In this paper, we propose an alternative approach to enhance FSCS-ART, namely Nearest-Neighbor Divide-and-Conquer based ART (NNDC-ART), which combines the concepts of nearest-neighbor and divide-and-conquer . It attempts to reduce the computational overhead of test case generation, and also to alleviate the boundary effect problem; while maintaining the advantages of FSCS-ART. The simulation results show that compared with FSCS-ART, NNDC-ART requires much less computational overhead, and also achieves much better test case distribution; while maintaining better or comparable failure-detection effectiveness. Our empirical studies further show that the proposed approach is much more cost-effective than FSCS-ART. In addition, we further compare NNDC-ART against two enhancements of FSCS-ART based on the partitioning strategy, i.e., ART with Divide-and-Conquer (ART-DC) and ART by Bisection and Localization (ART-B-Loc), and find that our proposed approach can achieve similar or even better testing performances in some scenarios.

MT-ART: A Test Case Generation Method Based on Adaptive Random Testing and Metamorphic Relation

Most of metamorphic testing (MT) research works focused on the generation and application of metamorphic relations (MRs). There is no clear conclusion about the relationship between test case generation methods and performance of MT. In this article, we introduce a novel method based on adaptive random testing (ART) and MR for MT test case generation. It proposes a family of algorithms for MT test cases generation, named as MT based ART (MT-ART). Three distances are measured to generate the next MT test case. In order to verify the performance of this method, series of experiments on four programs with different numbers of inputs are introduced. The results show that MT-ART performs better than other ART algorithms not only in test effectiveness, but also in test efficiency and test coverage. Based on this article, the following conclusions can be drawn: first, considering the effectiveness of MRs and test cases in MT may lead to better results. In this way, most of the existing research can be improved by this method. This is the most important contribute of our research. Second, not only the source test cases, but also the follow-up test cases can improve the performance of MT. Therefore, they should be considered together during the process of the next test case generation. Third, the average distance performs better than the max distance and the minimum distance in metamorphic test case selection.

A cost-effective adaptive random testing algorithm for object-oriented software testing

The rapid development of object oriented programming (OOP) technology has made it one of the mainstream programming technologies that has been widely used in the design and development of object oriented software (OOS). The inheritance, encapsulation and polymorphism properties of object-oriented language can improve the reusability, scalability and interoperability of software while increasing the difficulty of testing OOS. Researchers have proposed a variety of testing methods to test OOS among which random testing (RT) has been widely used due to its simplicity and ease of use. An OMISS-ARTsum algorithm is proposed in this paper that uses improved OMISS random test FSCS-ART with max-sum standard, which is an implementation version of fixed-sized-candidate-set ART. The OMISS-ARTsum algorithm calculates the total distance between a candidate test case and the executed test case set before the next test case is selected from the set of candidate test cases. Unlike the traditional max-sum based FSCS-ART algorithm, OMISS-ARTsum does not calculate the distance between each executed test case and the candidate case and then sum up the total distance, but uses the method of summing up all the executed test cases and the candidate cases. The information of executing test cases is saved as a whole and the distance between the executed test case set and candidate cases is calculated at the same time. Experiment shows that compared to the OMISS-ART algorithm, the proposed OMISS-ARTsum algorithm can reduce the time overhead.

Regression Test Case Prioritization Based on Fixed Size Candidate Set ART Algorithm

Regression testing is a very time-consuming and expensive testing activity. Many test case prioritization techniques have been proposed to speed up regression testing. Previous studies show that no one technique is always best. Random strategy, as the simplest strategy, is not always so bad. Particularly, when a test suite has higher fault detection capability, the strategy can generate a better result. Nevertheless, due to the randomness, the strategy is not always as satisfactory as expected. In this context, we present a test case prioritization approach using fixed size candidate set adaptive random testing algorithm to reduce the effect of randomness and improve fault detection effectiveness. The distance between pair-wise test cases is assessed by exclusive OR. We designed and conducted empirical studies on eight C programs to validate the effectiveness of the proposed approach. The experimental results, confirmed by a statistical analysis, indicate that the approach we proposed is more effective than random and the total greedy prioritization techniques in terms of fault detection effectiveness. Although the presented approach has comparable fault detection effectiveness to ART-based and the additional greedy techniques, the time cost is much lower. Consequently, the proposed approach is much more cost-effective.

Hybrid adaptive random testing

Adaptive random testing (ART) subsumes a family of random testing techniques with an effective improvement. It is based on the observation that failure causing inputs tend to be clustered together. Hence the ART methods spread test cases more evenly within the input domain to improve the fault-detection capability of random testing. There have been several implementations of ART based on different intuitions and principles with their own advantages and disadvantages. In the different variants of ART methods, the majority of them use a variety of distance calculations, with corresponding computational overhead. The newly methods try to decrease computational overhead while maintaining the performance through partitioning the input domain. We outline a new partitioning-based ART algorithm with a hybrid search method and demonstrate experimentally that it can further improve the performance, with considerably lower overhead than other ART algorithms.

Exploiting the Largest Available Zone: A Proactive Approach to Adaptive Random Testing by Exclusion

Adaptive random testing (ART) has been proposed to enhance the effectiveness of random testing (RT) through more even spreading of the test cases. In particular, restricted random testing (RRT) is an ART algorithm based on the intuition of skipping all the candidate test cases that are within the neighborhoods (or zones) of previously executed test cases. RRT has higher effectiveness than RT in terms of failure detection but incurs a higher time cost. In this paper, we aim to further reduce the time costs for RRT and improve the effectiveness for RT and ART methods. We propose a proactive technique known as “RRT by largest available zone” (RRT-LAZ). Like RRT, RRT-LAZ first defines an exclusion zone around every executed test case in order to determine the available zones. Unlike the original RRT, RRT-LAZ then compares all the available zones to proactively pick the largest one, from which the next test case is randomly generated. Both simulation analyses and empirical studies have been employed to investigate the efficiency and effectiveness of RRT-LAZ in relation to RT and related ART algorithms. The results show that RRT-LAZ has significantly lower time costs than RRT. Furthermore, RRT-LAZ is more effective than RT and related ART methods for block failure patterns in low-dimensional input spaces. In general, since RRT-LAZ employs a proactive technique instead of a passive one in generating next cases, it is much more cost-effective than RRT. RRT-LAZ is also more cost-effective than RT and other ART methods that we have studied.

An Enhanced Adaptive Random Testing by Dividing Dimensions Independently

Random testing (RT) is widely applied in the area of software testing due to its advantages such as simplicity, unbiasedness, and easy implementation. Adaptive random testing (ART) enhances RT. It improves the effectiveness of RT by distributing test cases as evenly as possible. Fixed Size Candidate Set (FSCS) is one of the most well-known ART algorithms. Its high failure-detection effectiveness only shows at low failure rates in low-dimensional spaces. In order to solve this problem, the boundary effect of the test case distribution is analyzed, and the FSCS algorithm of a limited candidate set (LCS-FSCS) is proposed. By utilizing the information gathered from success test cases (no failure-causing test inputs), a tabu generation domain of candidate test case is produced. This tabu generation domain is eliminated from the current candidate test case generation domain. Finally, the number of test cases at the boundary is reduced by constraining the candidate test case generation domain. The boundary effect is effectively relieved, and the distribution of test cases is more even. The results of the simulation experiment show that the failure-detection effectiveness of LCS-FSCS is significantly improved in high-dimensional spaces. Meanwhile, the failure-detection effectiveness is also improved for high failure rates and the gap of failure-detection effectiveness between different failure rates is narrowed. The results of an experiment conducted on some real-life programs show that LCS-FSCS is less effective than FSCS only when the failure distribution is concentrated on the boundary. In general, the effectiveness of LCS-FSCS is higher than that of FSCS.

Semiautomated Metamorphic Testing Approach for Geographic Information Systems: An Empirical Study

A geographic information system (GIS) provides basic location-enabled services for many different applications related to navigation, education, and telecommunications. It is a foundation for analysis and visualization. Testing GIS is critical, but challenging due to the difficulty to assess the correctness of GIS outputs, which is called the test oracle problem of software testing. Metamorphic testing alleviates the problem by constructing metamorphic relations (MRs) among multiple inputs and outputs of the program under test. In this article, a semiautomated metamorphic testing (SAMT) method, based on the formal MR model and an improved adaptive random testing algorithm, was proposed to the GIS. To evaluate the performance of our approach, we conducted a case study on a superficial area calculation program, a typical component of GIS. Six kinds of MR construction methods were suggested for the GIS domain program testing. The experimental results show that SAMT can detect the mutations effectively that could solve the test oracle problem efficiently. More importantly, there is no need to manual participation in the testing process, except for the MR construction.

A cost‐effective adaptive random testing approach by dynamic restriction

A key objective of software testing is to find program errors that cause failure in software, at less cost. One basic testing technique is random testing (RT), but many researchers have criticised its failure-detection effectiveness. Several researchers have proposed that an enhancement of the failure-detection effectiveness of RT is achieved if test cases are evenly spread within the input domain. Adaptive RT (ART) describes a family of algorithms that employ various strategies to evenly and randomly spread test cases. Fixed sized candidate set ART (FSCS-ART) is an ART algorithm that has gained many research studies far and wide; however, the high distance computations make its algorithm computationally expensive. The authors propose a new ART method that restricts distance computations to only test cases inside an exclusion zone. The experimental results show that the new ART method not only improves RT but also provides failure-detection effectiveness similar to FSCS-ART, while significantly minimising computation overhead.

Out of sight, out of mind: a distance-aware forgetting strategy for adaptive random testing

Adaptive random testing (ART) achieves better failure-detection effectiveness than random testing by increasing the diversity of test cases. However, the intention of ensuring even spread of test cases inevitably causes an overhead problem. Although two basic forgetting strategies (i.e. random forgetting and consecutive retention) were proposed to reduce the computation cost of ART, they only considered the temporal distribution of test cases. In the paper, we presented a distance-aware forgetting strategy for the fixed size candidate set version of ART (DF-FSCS), in which the spatial distribution of test cases is taken into consideration. For a given candidate, the test cases out of its “sight” are ignored to reduce the distance computation cost. At the same time, the dynamic adjustment for partitioning and the second-round forgetting are adopted to ensure the linear complexity of DF-FSCS algorithm. Both simulation analysis and empirical study are employed to investigate the efficiency and effectiveness of DF-FSCS. The experimental results show that DF-FSCS significantly outperforms the classical ART algorithm FSCS-ART in efficiency, and has comparable failure-detection effectiveness. Com-pared with two basic forgetting methods, DF-FSCS is better in both efficiency and effectiveness. In contrast with a typical linear-time ART algorithm RBCVT-Fast, our algorithm requires less computational overhead and exhibits the similar failure-detection capability. In addition, DF-FSCS has more reliable performance than RBCVT-Fast in detecting failures for the programs with high-dimensional input domain.

A Cost-Effective Random Testing Method for Programs with Non-Numeric Inputs

Random testing (RT) has been widely used in the testing of various software and hardware systems. Adaptive random testing (ART) is a family of random testing techniques that aim to enhance the failure-detection effectiveness of RT by spreading random test cases evenly throughout the input domain. ART has been empirically shown to be effective on software with numeric inputs. However, there are two aspects of ART that need to be addressed to render its adoption more widespread—applicability to programs with non-numeric inputs, and the high computation overhead of many ART algorithms. We present a linear-order ART algorithm for software with non-numeric inputs. The key requirement for using ART with non-numeric inputs is an appropriate “distance” measure. We use the concepts of categories and choices from category-partition testing to formulate such a measure. We investigate the failure-detection effectiveness of our technique by performing an empirical study on 14 object programs, using two standard metrics—F-measure and P-measure. Our ART algorithm statistically significantly outperforms RT on 10 of the 14 programs studied, and exhibits performance similar to RT on three of the four remaining programs. The selection overhead of our ART algorithm is close to that of RT.

통합 시스템을 위한 출력 분포 기반 적응적 랜덤 테스팅

Adaptive Random Testing (ART) aims to enhance the performance of pure random testing by detecting failure region in a software. The ART algorithm generates effective test cases which requires less number of test cases than that of pure random testing. However, all ART algorithms currently proposed are designed for the tests of monolithic system or unit level. In case of integrated system tests, ART approaches do not achieve same performances as those of ARTs applied to the unit or monolithic system. In this paper, we propose an extended ART algorithm which can be applied to the integrated system testing environment without degradation of performance. The proposed approach investigates an input distribution of the unit under a test with limited number of seed input data and generates information to be used to resizing input domain partitions. The simulation results show that our approach in an integration environment could achieve similar level of performance as an ART is applied to a unit testing. Results also show resilient effectiveness for various failure rates.

Adaptive random testing based on distribution metrics

Random testing (RT) is a fundamental software testing technique. Adaptive random testing (ART), an enhancement of RT, generally uses fewer test cases than RT to detect the first failure. ART generates test cases in a random manner, together with additional test case selection criteria to enforce that the executed test cases are evenly spread over the input domain. Some studies have been conducted to measure how evenly an ART algorithm can spread its test cases with respect to some distribution metrics. These studies observed that there exists a correlation between the failure detection capability and the evenness of test case distribution. Inspired by this observation, we aim to study whether failure detection capability of ART can be enhanced by using distribution metrics as criteria for the test case selection process. Our simulations and empirical results show that the newly proposed algorithms not only improve the evenness of test case distribution, but also enhance the failure detection capability of ART.

Application of a Failure Driven Test Profile in Random Testing

Random testing techniques have been extensively used in reliability assessment, as well as in debug testing. When used to assess software reliability, random testing selects test cases based on an operational profile; while in the context of debug testing, random testing often uses a uniform distribution. However, generally neither an operational profile nor a uniform distribution is chosen from the perspective of maximizing the effectiveness of failure detection. Adaptive random testing has been proposed to enhance the failure detection capability of random testing by evenly spreading test cases over the whole input domain. In this paper, we propose a new test profile, which is different from both the uniform distribution, and operational profiles. The aim of the new test profile is to maximize the effectiveness of failure detection. We integrate this new test profile with some existing adaptive random testing algorithms, and develop a family of new random testing algorithms. These new algorithms not only distribute test cases more evenly, but also have better failure detection capabilities than the corresponding original adaptive random testing algorithms. As a consequence, they perform better than the pure random testing.

입력 도메인 확장을 이용한 반복 분할 기반의 적응적 랜덤 테스팅 기법

적응적 랜덤 테스팅 (Adaptive Random Testing, ART)은 입력 도메인 내에 테스트 케이스를 넓고 고르게 분산시키는 방법을 통해 입력 도메인 내에 존재하는 오류 패턴을 순수 랜덤 테스팅 (Random Testing, RT)보다 효율적으로 찾아내기 위한 테스트 케이스 선택 기법이다. 테스트 케이스 선택에 많은 연산량을 필요로 하는 초기 ART 기법인 거리 기반 ART (Distance-based ART, D-ART)와 제한 영역 기반 ART (Restricted Random Testing, RRT)의 개선을 위해 입력 도메인을 반복 분할하는 기법들이 제안되었고, 이 기법들은 낮은 연산량 및 성능 향상등의 효과를 가져왔다. 하지만, 입력 도메인 반복 분할 기반 기법에서도 기존 ART 기법에서 나타나는 테스트 케이스 분포 불균일 문제가 존재하고, 이는 기법의 확장성에 장애 요소로 작용한다. 따라서 본 논문에서는 반복 분할 기반 기법에서 나타나는 테스트 케이스 분포의 특성을 파악하고, 이를 적정 수준으로 제어하기 위한 입력 도메인 확장 정책을 제안하였으며, 실험을 통해 2차원 입력 도메인에서 3%, 3차원 입력 도메인에서 10% 수준의 성능 향상을 확인하였다. An Adaptive Random Testing(ART) is one of test case generation algorithms, which was designed to get better performance in terms of fault-detection capability than that of Random Testing(RT) algorithm by locating test cases in evenly spreaded area. Two ART algorithms, such as Distance-based ART(D-ART) and Restricted Random Testing(RRT), had been indicated that they have significant drawbacks in computations, i.e., consuming quadratic order of runtime. To reduce the amount of computations of D-ART and RRT, iterative partitioning of input domain strategy was proposed. They achieved, to some extent, the moderate computation cost with relatively high performance of fault detection. Those algorithms, however, have yet the patterns of non-uniform distribution in test cases, which obstructs the scalability. In this paper we analyze the distribution of test cases in an iterative partitioning strategy, and propose a new method of input domain enlargement which makes the test cases get much evenly distributed. The simulation results show that the proposed one has about 3 percent of improvement in terms of mean relative F-measure for 2-dimension input domain, and shows 10 percent improvement for 3-dimension space.

Distributing test cases more evenly in adaptive random testing

Adaptive random testing (ART) has recently been proposed to enhance the failure-detection capability of random testing. In ART, test cases are not only randomly generated, but also evenly spread over the input domain. Various ART algorithms have been developed to evenly spread test cases in different ways. Previous studies have shown that some ART algorithms prefer to select test cases from the edge part of the input domain rather than from the centre part, that is, inputs do not have equal chance to be selected as test cases. Since we do not know where the failure-causing inputs are prior to testing, it is not desirable for inputs to have different chances of being selected as test cases. Therefore, in this paper, we investigate how to enhance some ART algorithms by offsetting the edge preference, and propose a new family of ART algorithms. A series of simulations have been conducted and it is shown that these new algorithms not only select test cases more evenly, but also have better failure detection capabilities.

Enhancing adaptive random testing for programs with high dimensional input domains or failure-unrelated parameters

Adaptive random testing (ART), an enhancement of random testing (RT), aims to both randomly select and evenly spread test cases. Recently, it has been observed that the effectiveness of some ART algorithms may deteriorate as the number of program input parameters (dimensionality) increases. In this article, we analyse various problems of one ART algorithm, namely fixed-sized-candidate-set ART (FSCS-ART), in the high dimensional input domain setting, and study how FSCS-ART can be further enhanced to address these problems. We propose to add a filtering process of inputs into FSCS-ART to achieve a more even-spread of test cases and better failure detection effectiveness in high dimensional space. Our study shows that this solution, termed as FSCS-ART-FE, can improve FSCS-ART not only in the case of high dimensional space, but also in the case of having failure-unrelated parameters. Both cases are common in real life programs. Therefore, we recommend using FSCS-ART-FE instead of FSCS-ART whenever possible. Other ART algorithms may face similar problems as FSCS-ART; hence our study also brings insight into the improvement of other ART algorithms in high dimensional space.