Abstract
Test suite reduction approaches aim at decreasing software regression testing costs by selecting a representative subset from large-size test suites. Most existing techniques are too expensive for handling modern massive systems and moreover depend on artifacts, such as code coverage metrics or specification models, that are not commonly available at large scale. We present a family of novel very efficient approaches for similarity-based test suite reduction that apply algorithms borrowed from the big data domain together with smart heuristics for finding an evenly spread subset of test cases. The approaches are very general since they only use as input the test cases themselves (test source code or command line input). We evaluate four approaches in a version that selects a fixed budget B of test cases, and also in an adequate version that does the reduction guaranteeing some fixed coverage. The results show that the approaches yield a fault detection loss comparable to state-of-the-art techniques, while providing huge gains in terms of efficiency. When applied to a suite of more than 500K real world test cases, the most efficient of the four approaches could select B test cases (for varying B values) in less than 10 seconds.
Highlights
In recent years testing has consistently been the most actively investigated topic of main software engineering conferences [6]
To address the above question, in the last three decades many techniques have been proposed, which can be roughly divided in two groups: those that aim at reordering the test cases in T so that those more likely to fail are executed first, and those that select a subset T ′ ⊆ T that should ideally include the failing test cases, if any; the latter group of techniques is referred to as test case selection or test suite reduction,1 depending on whether when choosing
M dn is the median fault detection loss, σ is the standard deviation, and δ is the group for the pairwise comparisons after the Kruskal-Wallis test
Summary
In recent years testing has consistently been the most actively investigated topic of main software engineering conferences [6]. The proposed techniques have been evaluated and compared against each other using metrics relative to their fault detection effectiveness (e.g., the Average Percentage of Fault Detection of the reordered test suite, or the loss in faults detected by the reduced test suite T ′); for test reduction and selection, metrics relative to cost savings, e.g., the size or the execution time of T ′ are compared against those of the full suite T. Another important factor that should be taken into account is the cost of the technique itself, both in terms of the computational effort and of the resources it requires.
Sign-in/Register to view highlights & summary Sign-in/Register to access full text options 
Free) 
Talk to us
Join us for a 30 min session where you can share your feedback and ask us any queries you have
Schedule a call
