Logical Contradictions in the One-Way ANOVA and Tukey–Kramer Multiple Comparisons Tests with More Than Two Groups of Observations

Abstract

We show that the one-way ANOVA and Tukey–Kramer (TK) tests agree on any sample with two groups. This result is based on a simple identity connecting the Fisher–Snedecor and studentized probabilistic distributions and is proven without any additional assumptions; in particular, the standard ANOVA assumptions (independence, normality, and homoscedasticity (INAH)) are not needed. In contrast, it is known that for a sample with k>2 groups of observations, even under the INAH assumptions, with the same significance level α, the above two tests may give opposite results: (i) ANOVA rejects its null hypothesis H0A:μ1=…=μk, while the TK one, H0TK(i,j):μi=μj, is not rejected for any pair i,j∈{1,…,k}; (ii) the TK test rejects H0TK(i,j) for a pair (i,j) (with i≠j), while ANOVA does not reject H0A. We construct two large infinite pseudo-random families of samples of both types satisfying INAH: in case (i) for any k≥3 and in case (ii) for some larger k. Furthermore, case (ii) ANOVA, being restricted to the pair of groups (i,j), may reject equality μi=μj with the same α. This is an obvious contradiction, since μ1=…=μk implies μi=μj for all i,j∈{1,…,k}. Such contradictions appear already in the symmetric case for k=3, or in other words, for three groups of d,d, and c observations with sample means +1,−1, and 0, respectively. We outline conditions necessary and sufficient for this phenomenon. Similar contradictory examples are constructed for the multivariable linear regression (MLR). However, for these constructions, it seems difficult to verify the Gauss–Markov assumptions, which are standardly required for MLR. Mathematics Subject Classification: 62 Statistics.