Strategies in adjusting for multiple comparisons: A primer for pediatric surgeons

Abstract

Background/PurposeIn pediatric surgery research, the issue of multiple comparisons commonly arises when there are multiple patient or experimental groups being compared two at a time on an outcome of interest. Performing multiple statistical comparisons increases the likelihood of finding a false positive result when there truly are no statistically significant group differences (falsely rejecting the null hypothesis when it is true). In order to control for the risk of false positive results, there are several statistical approaches that surgeons should consider in collaboration with a biostatistician when performing a study that is prone to the issue of false discovery related to multiple comparisons. It is becoming increasingly more common for high impact journals to require authors to carefully consider multiplicity in their studies. Therefore, the objective of this primer is to provide surgeons with a useful guide and recommendations on how to go about taking multiple comparisons into account to keep false positive results at an acceptable level. MethodsWe provide background on the issue of multiple comparisons and risk of type I error and guidance on statistical approaches (i.e. multiple comparisons procedures) that can be implemented to control the type I false positive error rate based on the statistical analysis plan. These include, but are not limited to, the Bonferroni correction, the False Discovery Rate (FDR) approach, Tukey's procedure, Scheffé's procedure, Holm's procedure, and Dunnett's procedure. ResultsWe present the results of the various approaches following one-way analysis of the variance (ANOVA) using a hypothetical surgical research example of the comparison between three experimental groups of rats on skin defect coverage for experimental spina bifida: the TRASCET group, sham control, and saline control. The ultimate decision in accounting for multiple comparisons is situation-dependent and surgeons should work with their statistical colleagues to ensure the best approach for controlling the type I error rate and interpreting the evidence when making multiple inferences and comparisons. ConclusionsThe risk of rejecting the null hypothesis increases when multiple hypotheses are tested using the same data. Surgeons should be aware of the available approaches and considerations to take into account multiplicity in the statistical plan or protocol of their clinical and basic science research studies. This strategy will improve their study design and ensure the most appropriate analysis of their data. Not adjusting for multiple comparisons can lead to misleading presentation of evidence to the surgical research community because of exaggerating treatment differences or effects. Type of studyReview article. Level of evidenceN/A