ANOVA: A Paradigm for Low Power and Misleading Measures of Effect Size?

Abstract

Over 30 years ago, Tukey made it evident that slight departures from normality can substantially lower power when means are compared, and that a popular measure of effect size can be highly misleading. At the time there were no methods for dealing with the problem raised in Tukey’s paper, and some of the more obvious and seemingly intuitive solutions have since been found to be highly unsatisfactory. Today there are practical methods for not only dealing with the problem raised by Tukey, but also achieving more accurate confidence intervals and control over the probability of a Type I error. More generally, there are many robust and exploratory ways of comparing groups that can reveal important differences that are missed by conventional methods based on means, and even modern methods based solely on robust measures of location. This article reviews these new techniques.