Large variation existed in standardized mean difference estimates using different calculation methods in clinical trials

Abstract

Background and ObjectivesThe standardized mean difference (SMD) can be calculated from different mean differences (MDs) and standard deviations (SDs). This study aims to investigate how clinical trials calculated, reported and interpreted the SMD, and to examine the variation between different SMDs. MethodsWe searched the PubMed for randomized controlled trials of general medicine and psychiatry that estimated SMDs. We explored how the SMD was computed and interpreted. We calculated SMDs based on different MDs and SDs, and the variation in these SMD estimates for each study. ResultsWe included 161 articles. Various MDs and SDs were used to calculate SMDs, yet 69.0% studies failed to provide sufficient details. Variations in SMD estimates using different MDs and SDs in one study could be substantial (median of the absolute differences was 0.3, interquartile range IQR 0.17 to 0.53). However, 68.3% studies interpreted the SMD based on the same reference, Cohen's rule of thumb. The largest variations were observed in studies with small sample sizes and large reported effects. ConclusionSMDs using different MDs and SDs could vary considerably, but the report was often insufficient and the interpretation was oversimplified. To avoid selective reporting bias and misinterpretation, prespecifying and reporting the method and interpreting the result from multiple perspectives are desirable.