Statistical significance misuse in public health research: an investigation of the current situation and possible solutions

Abstract

Background Despite the efforts of leading statistical authorities and experts worldwide, misuse of statistical significance remains a common, dangerous practice in public health research. There is an urgent need to quantify this phenomenon. Methods 200 studies were randomly selected within the PubMed database. An evaluation scale for the interpretation and presentation of statistical results (SRPS) was adopted. The maximum achievable score was 4 points. Abstracts (A) and full texts (FT) were compared to highlight any differences in presentation. The Wilcoxon signed-rank test was employed in this scope. Results All studies failed to adopt P-values as continuous measures of compatibility between the data and the target hypothesis as assessed by the chosen test. The vast majority did not provide information on the model specification. However, in most cases, all findings were reported in full within the manuscripts. The Wilcoxon signed-rank test showed a marked incompatibility of the null hypothesis of zero difference between A and FT scores with the data obtained in the survey: null P < .001 (as assessed by the model), r = 0.87 (standardized effect size). Additionally, the score difference (207.5 points for A vs. 441.5 points for FT) indicates a scenario consistent with a substantial disparity in the completeness of the outcomes reporting. Conclusion These findings align with the hypothesis of widespread and severe shortcomings in the use and interpretation of statistical significance within public health research during 2023. Therefore, it is essential for academic journals to compulsorily demand higher scientific quality standards. The suggestions provided in this study could be helpful for this purpose.