An empirical validation of FindBugs issues related to defects

Abstract

Background: Effective use of bug finding tools promise to speed up the process of source code verification and to move a portion of discovered defects from testing to coding phase. However, many problems related to their usage, especially the large number of false positives, could easily hinder the potential benefits of such tools. Aims: Assess the percentage and type of issues of a popular bugfinding tool (FindBugs) that are actual defects. Method: We analyzed 301 Java Projects developed at a university with FindBugs, collecting the issues signalled on the source code. Afterwards, we checked the precision of issues with information on changes, we ranked and validated them using both manual inspection and validation with tests failures. Results: We observed that a limited set of issues have high precision and conversely we identified those issues characterized by low precision. We compared findings first with our previous experiment and then to related work: results are consistent with both of them. Conclusions: Since our and other empirical studies demonstrated that few issues are related to real defects with high precision, developers could enable only them (or prioritize), reducing the information overload of FindBugs and having the possibility to discover defects earlier. Furthermore, the technique presented in the paper can be adopted to other tools on a code base with tests to find issues with high precision that can be checked on code in production to find defects earlier.