An Experimental Assessment of Using Theoretical Defect Predictors to Guide Search-Based Software Testing

Anjana Perera,Marcel Bohme,Aldeida Aleti,Burak Turhan

doi:10.1109/tse.2022.3147008

Anjana Perera, Marcel Bohme + Show 2 more

Open Access

https://doi.org/10.1109/tse.2022.3147008

Copy DOI

Abstract

Automated test generators, such as search-based software testing (SBST) techniques are primarily guided by coverage information. As a result, they are very effective at achieving high code coverage. However, is high code coverage alone sufficient to detect bugs effectively? In this paper, we propose a new SBST technique, predictive many objective sorting algorithm (PreMOSA), which augments coverage information with defect prediction information to decide where to increase the test coverage in the class under test (CUT). Through an experimental evaluation using 420 labelled bugs on the Defects4J benchmark and using theoretical defect predictors, we demonstrate the improved effectiveness and efficiency of PreMOSA in detecting bugs when using any acceptable defect predictor, i.e., a defect predictor with recall and precision <inline-formula><tex-math notation="LaTeX">$\geq$</tex-math></inline-formula> 75%, compared to the state-of-the-art dynamic many objective sorting algorithm (DynaMOSA). PreMOSA detects up to 8.3% more labelled bugs on average than DynaMOSA when given a time budget of 2 minutes for test generation per CUT.

Highlights

Search-based software testing (SBST) techniques consider test cases with high code coverage as high quality test cases, and aim at maximising code coverage [1, 2, 3]
Our experimental evaluation demonstrates that predictive many objective sorting algorithm (PreMOSA) is significantly more effective than DynaMOSA with large effect sizes when using any acceptable defect predictor, i.e., recall and precision ≥ 0.75 [25]
The differences of the number of bugs detected by PreMOSA-100/PreMOSA-75 and DynaMOSA are statistically significant according to the Mann-Whitney U-Test with large effect sizes (A12 ≥ 0.98)

Summary

Introduction

Search-based software testing (SBST) techniques consider test cases with high code coverage as high quality test cases, and aim at maximising code coverage [1, 2, 3]. As a result, they are very effective at achieving high code coverage [4]. DynaMOSA [3], a stateof-the-art SBST technique, could only detect on average 22% of the bugs from the Defects4J dataset, when it is given a 30 seconds time budget per class and using branch coverage as criterion [8]. Branch distance [27, 28] is a widely used heuristic in fitness functions to guide the search to find inputs that evaluate the logic in branch predicates as desired (i.e., to true or false)

Objectives

Methods

Results

Discussion

Conclusion