Novel Effect Size Interpretation Guidelines and an Evaluation of Statistical Power in Rehabilitation Research

Abstract

ObjectiveFirst, to establish empirically-based effect size interpretation guidelines for rehabilitation treatment effects. Second, to evaluate statistical power in rehabilitation research. Data SourcesThe Cochrane Database of Systematic Reviews was searched through June 2019. Study SelectionMeta-analyses included in the Cochrane Database of Systematic Reviews with “rehabilitation” as a keyword and clearly evaluated a rehabilitation intervention. Data ExtractionWe extracted Cohen’s d effect sizes and associated sample sizes for treatment and comparison groups. Two independent investigators classified the interventions into 4 categories using the Rehabilitation Treatment Specification System. The 25th, 50th, and 75th percentile values within the effect size distribution were used to establish interpretation guidelines for small, medium, and large effects, respectively. A priori power analyses established sample sizes needed to detect the empirically-based values for small, medium, and large effects. Post-hoc power analyses using median sample sizes revealed whether the “typical” rehabilitation study was sufficiently powered to detect the empirically-based values. Post hoc power analyses established the statistical power of each test based on the sample size and reported effect size. Data SynthesisWe analyzed 3381 effect sizes extracted from 99 meta-analyses. Interpretation guidelines for small effects ranged from 0.08 to 0.15; medium effects ranged from 0.19 to 0.36; and large effects ranged from 0.41 to 0.67. We present sample sizes needed to detect these values based on a priori power analyses. Post hoc power analyses revealed that a “typical” rehabilitation study lacks sufficient power to detect the empirically-based values. Post hoc power analyses using reported sample sizes and effects indicated the studies were underpowered, with median power ranging from 0.14 to 0.23. ConclusionsThis study presented novel and empirically-based interpretation guidelines for small, medium, and large rehabilitation treatment effects. The observed effect size distributions differed across intervention categories, indicating that researchers should use category-specific guidelines. Furthermore, many published rehabilitation studies are underpowered.