#293 : Trends of Statistical Significance Reporting in Abstracts of Reproductive Medicine Studies from 1990 to 2022

Abstract

Background and Aims: Despite efforts to minimize publication reporting bias in medical research, it remains unclear whether the magnitude and patterns of the bias have changed over time. We aim to assess the frequency in which the statistical information is presented in forms other than P-values and evaluate how statistical significance was reported in the abstracts of reproductive medicine studies. Method: We studied reproductive medicine studies published in 23 Q1 journals in reproductive medicine, and five top medical journals, including JAMA, Lancet, BMJ, NEJM, and PLOS Medicine. We text-mined abstracts from 1990 to 2022 to extract P-value, confidence interval (CI), and text description in statistical significance; the presence of effect size metrics and Bayes factors were searched. One thousand abstracts were randomly selected and manually checked. The extracted statistical significance information was then analysed for temporal trends and distribution. Results: We identified 24,550 reproductive medicine abstracts. The proportion of abstracts reporting isolated P-values was 13.9% in 1990 and 18.8% in 2021 (Figure 1); reporting isolated P-values was 3.0% (95% CI, 0.5%-11.3%) in meta-analyses, 20.8% (95% CI, 11.3%-34.5%) in randomized controlled trials, and 33.3% (95% CI, 23.8%-44.3%) in basic research in 2021. By contrast, the proportion of abstracts in reporting effect measures without P-value surged from 4.1% to 23.3% from 1990 to 2021. The reported P-values (n = 18,132) mainly clustered at 0.001 (20.0%) and 0.05 (16.2%). Of the 14,689 abstracts containing at least one statement describing statistical significance, 74.9% of abstracts made at least one statistically significant statement; this proportion had merely changed over time (Figure 2). Conclusion: Despite the increasing adoption of effect measures to convey statistical significance, reporting isolated P-values remains common. Publication reporting bias is pervasive and persisting in reproductive medicine abstracts; the inflated treatment could mislead all types of patient care and policy decisions.