Beyond statistical significance: nuanced interpretations of statistically nonsignificant results were rare in Cochrane reviews – a metaepidemiological study

Abstract

ObjectivesTo assess the language used by systematic review authors to emphasize that statistically nonsignificant results show meaningful differences. To determine whether the magnitude of these treatment effects was distinct from nonsignificant results that authors interpreted as not different. Study Design and SettingWe screened Cochrane reviews published between 2017 and 2022 for statistically nonsignificant effect estimates that authors presented as meaningful differences. We classified interpretations qualitatively and assessed them quantitatively by calculating the areas under the curve of the portions of confidence intervals exceeding the null or a minimal important difference, indicating one intervention's greater effect. ResultsIn 2,337 reviews, we detected 139 cases where authors emphasized meaningful differences in nonsignificant results. Authors commonly used qualifying words to express uncertainty (66.9%). Sometimes (26.6%), they made absolute claims about one intervention's greater benefit or harm without acknowledging statistical uncertainty. The areas under the curve analyses indicated that some authors may overstate the importance of nonsignificant differences, whereas others may overlook meaningful differences in nonsignificant effect estimates. ConclusionNuanced interpretations of statistically nonsignificant results were rare in Cochrane reviews. Our study highlights the need for a more nuanced approach by systematic review authors when interpreting statistically nonsignificant effect estimates.