Multiple Comparisons

Abstract

The study by Tozzi, et al, provides a nice illustration of the problem of multiple comparisons. The authors compared results of children with two levels of exposure to thimerosal on 24 different neuropsychological tests: overall, and separately in boys and girls. Of the total of 72 different comparisons, two had P-values less than 0.05.The P-value provides an estimate of the probability of observing a difference at least as large as was observed in the study, if there were in fact no difference between the groups. A P-value of 0.05 means a difference was observed that would be expected to occur by chance only one time in 20. But intuitively, we know that unlikely events eventually happen if given enough chances. Since the authors tested 72 different outcomes for association with thimerosal, it seems unimpressive that two of them would be statistically significant with P<0.05.How should we deal with the multiple comparisons issue? One approach, named after the Italian mathematician Carlo Bonferroni (1892–1960), is to require a more stringent level of statistical significance (called α) if more comparisons are made. Rather than rejecting the null hypothesis at the traditional α = 0.05, we divide α by the number of comparisons. For example, for 72 different comparisons, we would set α at 0.05/72 = 0.0007, meaning we would require P-values <0.0007 before we would consider them “significant.” The Bonferroni approach tends to be overly conservative, however, so it is useful mainly when results remain significant even with the lower α.A better approach is to consider the prior probability of differences between groups for each of the comparisons made.1 Before the results of Tozzi, et al, were analyzed, how likely was it that the 75 μg difference in cumulative ethylmercury exposure in the two groups would lead to differences in neuropsychogical tests 10 years later? And how likely was it that the difference would be confined to Boston Naming and finger tapping with the dominant hand, and be present in girls only?With multiple prior studies showing no adverse effects of thimerosal and without prior studies or a biological basis to suggest particular vulnerability of girls in domains tested by these tests, it would appear that the prior probability of these results was exceedingly low, and hence the posterior probability (after the study) that the effect is real remains low. The overwhelming likelihood remains that the observed differences are due to chance.