An Analysis of the Use of Multiple Comparison Corrections in Ophthalmology Research

Abstract

The probability of type I error, or a false-positive result, increases as the number of statistical comparisons in a study increases. Statisticians have developed numerous corrections to account for the multiple comparison problem. This study discusses recent guidelines involving multiple comparison corrections, calculates the prevalence of corrections in ophthalmic research, and estimates the corresponding number of false-positive results reported at a recent international research meeting. The 6415 abstracts presented at ARVO 2010 were searched for statistical comparisons (P values) and for use of multiple comparison corrections. Studies that reported five or more P values while reporting no correction factor were used in a simulation study. The simulation study was conducted to estimate the number of false-positive results reported in these studies. Overall, 36% of abstracts reported P values and 1.2% of abstracts used some form of correction. Whereas 8% of abstracts reported at least five P values, only 5% of these used a multiple comparison correction. In these highly statistical studies, simulations resulted in 185 false-positive outcomes found in 30% of abstracts. The paucity of multiple comparison corrections in ophthalmic research results in inflated type I error and may produce unwarranted shifts in clinical or surgical care. Researchers must make a conscious effort to decide if and when to use a correction factor to ensure the validity of the data.