On the feasibility of automated prediction of bug and non-bug issues

Steffen Herbold,Fabian Trautsch,Alexander Trautsch

doi:10.1007/s10664-020-09885-w

Steffen Herbold, Fabian Trautsch + Show 1 more

Open Access

https://doi.org/10.1007/s10664-020-09885-w

Copy DOI

Abstract

ContextIssue tracking systems are used to track and describe tasks in the development process, e.g., requested feature improvements or reported bugs. However, past research has shown that the reported issue types often do not match the description of the issue.ObjectiveWe want to understand the overall maturity of the state of the art of issue type prediction with the goal to predict if issues are bugs and evaluate if we can improve existing models by incorporating manually specified knowledge about issues.MethodWe train different models for the title and description of the issue to account for the difference in structure between these fields, e.g., the length. Moreover, we manually detect issues whose description contains a null pointer exception, as these are strong indicators that issues are bugs.ResultsOur approach performs best overall, but not significantly different from an approach from the literature based on the fastText classifier from Facebook AI Research. The small improvements in prediction performance are due to structural information about the issues we used. We found that using information about the content of issues in form of null pointer exceptions is not useful. We demonstrate the usefulness of issue type prediction through the example of labelling bugfixing commits.ConclusionsIssue type prediction can be a useful tool if the use case allows either for a certain amount of missed bug reports or the prediction of too many issues as bug is acceptable.

Highlights

The tracking of tasks and issues is a common part of modern software engineering, e.g., through dedicated systems like Jira and Bugzilla, or integrated into other other systems like GitHub Issues
– We found that rules that determine supposedly easy subsets of data based on null pointers do not help to improve the quality of issue type prediction models aimed at identifying bugs
We observe that while there is a strong variance in the F1 score using the CVALL data with values between 0.0 (Limsettho2014-Naıve Bayes (NB)) and 0.643 (Herbold2020-FTA), the results on the CVBUG data are more stable with values between 0.610 (Terdchanakul2017-Random Forest (RF)) and 0.809 (Herbold2020-RF)

Summary

Introduction

The tracking of tasks and issues is a common part of modern software engineering, e.g., through dedicated systems like Jira and Bugzilla, or integrated into other other systems like GitHub Issues. Developers and sometimes users of software file issues, e.g., to describe bugs, request improvements, organize work, or ask for feedback. This manifests in different types into which the issues are classified. Wrong types of issues can have different kinds of negative consequences, depending on the use of the issue tracking system. Mislabeled issues may have direct negative consequences on development processes as well as indirect consequences due to the downstream use of possibly noisy data. Studies by Herzig et al (2013) and Herbold et al (2020) have independently and on different data shown that on average about 40% issues are mislabelled, and most mislabels are issues wrongly classified as BUG

Methods

Results

Conclusion