Evaluating the Strength of Evidence in Favor of Rehabilitation Effects: A Bayesian Analysis.

Abstract

Relying solely on null hypothesis significance testing to investigate rehabilitation interventions may result in researchers erroneously concluding the presence of a treatment effect. We sought to quantify the strength of evidence in favour of rehabilitation treatment effects by calculating Bayes factors (BF10s) for significant findings. Additionally, we sought to examine associations between BF10s, P-values, and Cohen's d effect sizes. We searched the Cochrane Database of Systematic Reviews for meta-analyses with "rehabilitation" as a keyword that evaluated a rehabilitation intervention. We extracted means, standard deviations, and sample sizes for treatment and comparison groups from individual findings within 175 meta-analyses. Investigators independently classified the interventions according to the Rehabilitation Treatment Specification System. We calculated t-statistics, P-values, effect sizes, and BF10s for each finding. We isolated statistically significant findings (P≤0.05); applied evidential categories to BF10s, P-values, and effect sizes; and examined relationships descriptively. We analysed 1935 rehabilitation findings. Across intervention types, 25% of significant findings offered only anecdotal evidence in favour of a treatment effect; only 48% indicated strong evidence. This pattern persisted within intervention types and when conducting robustness analyses. Smaller P-values and larger effect sizes were associated with stronger evidence in favour of a treatment effect. However, a notable portion of findings with P-value 0.01 to 0.05 (63%) or a large effect size (18%) offered anecdotal evidence in favour of an effect. For a substantial portion of statistically significant rehabilitation findings, the data neither support nor refute the presence of a treatment effect. This was the case among a notable portion of large treatment effects and for most findings with P-value>0.01. Rehabilitation evidence would be improved by researchers adopting more conservative levels of significance, complementing the use of null hypothesis significance testing with Bayesian techniques and reporting effect sizes.