Problems with Statistical Practice in Human-Centric Software Engineering Experiments

Abstract

Background Examples of questionable statistical practice, when published in high quality software engineering (SE) journals, may lead to novice researchers adopting incorrect statistical practices. Objective Our goal is to highlight issues contributing to poor statistical practice in human-centric SE experiments. Method We reviewed the statistical analysis practices used in the 13 papers that reported families of human-centric SE experiments and were published in high quality journals. Results Reviewed papers related to 45 experiments and involved a total of 1303 human participants. We searched for issues that were related to questionable statistical practice that were found in more than one paper. We observed three types of bad practice: incorrect use of terminology, incorrect analysis of repeated measures designs, and post-hoc power testing. We also found two analysis practices (i.e., multiple testing and pre-testing for normality) where statisticians disagree about good practice. Conclusions Identified issues pose a problem because readers may expect the statistical methods used in papers published in top quality, peer-reviewed journals to be correct. We explain why the practices are problematic and provide recommendations for improved practice.