Chapter 43 - Statistical considerations when making potential multiple comparisons

Abstract

In health services research, researchers attempt to minimize Type-I errors by exerting control over their P-value thresholds or alpha level. When a statistical test is conducted only once in a study, it is indeed possible for the researcher to ascertain control, so that the likelihood of a Type-I error is equal to or less than the significance (P-value) level. Rarely in health services research, however, do investigators simply make a single comparison. Usually, multiple comparisons are undertaken, which can dramatically increase the likelihood of making a Type-I error. Researchers have customarily attempted to control for the increased risk of Type-I errors associated with multiple comparisons by making adjustments to their alpha or significance threshold level. However, such adjustments are not risk free, and when applied arbitrarily, they may create worse problems than those were employed to attenuate. The objective of this chapter is to provide a balanced commentary on the advantages and disadvantages of making adjustments when undertaking multiple comparisons in health services research. Examples, using results from a small dataset, are presented along with the implications of the choices available to a researcher. Last, advice on when researchers should consider making adjustments in P-value thresholds or alpha levels, and when adjustments should be avoided, is provided.