Studies of different variations of Adaptive Random Testing

Abstract

The importance of testing software has been increasingly acknowledged in recent years. Different approaches have been investigated and reported upon. One method which has aroused considerable discussion is Random Testing (RT), a method of testing software which relies on a random selection of test cases (collections of inputs representing a single use of the software being tested). Among the advantages of RT are its ease of use, the minimal overheads in test case generation, and the statistical support available. Research has indicated that failure patterns (portions of an input domain which, when executed, cause the program to fail or reveal an error) can influence the effectiveness of some testing strategies. For certain types of failure patterns, it has been found that a widespread and even distribution of test cases in the input domain can be significantly more effective at detecting failure than ordinary RT. Testing methods based on RT, but which aim to achieve even and widespread distributions, have been called Adaptive Random Testing (ART) strategies. In this thesis, the background and motivations behind ART are explained. Because there are many ways to achieve a widespread and even distribution of test cases, there are many possible implementations of ART. Several major ART implementations are examined in detail, and various computational cost reduction strategies and enhancements are presented. The methods presented are analysed mathematically and empirically. The methods are applied to simulations and fault-seeded programs, and their performance in terms of the number of test cases required to find a failure (a metric introduced as the F-measure) is examined.