I’ve been Testing Some Statistical Hypotheses— Can you Guess What they Are?

Abstract

This is intended as a commentary on three articles that appeared in recent issues of American Educational Research Journal. While subject matter and research problems discussed in these articles are of little interest here, they have been chosen because they typify a rather serious analytical (or perhaps even conceptual) error that diminishes usefulness of much of published research in field of education. It is my hope that by drawing attention to this problem it can be avoided in future.1 Fennema and Sherman (4) investigated sex-related differences in several factors, including mathematics achieve ment; Sherman and Fennema (23) considered variables that are related to study of mathematics by high school boys and girls; and Wright (26) inquired into affective and cognitive consequences of open education elementary schools. The common thread of interest in all three articles is that in each case data were analyzed within con text of a factorial analysis of variance (ANOVA) design. In each instance author(s) pointed out those which were or were not statistically and commented on these effects when appropriate. The problem emerges, however, when authors seem to ignore fact that they are dealing with unbalanced fac torial designs and, instead, make interpretative statements appropriate only for balanced factorial designs. As an indication of scope of this problem, it may be appropriate to note that three articles cited above were first three articles using unbalanced factorial ANOVA designs found by this author in a search of literature. Many other examples were found, but were not cited because of space considerations. An author employing a balanced factorial design may state that the row main effect was statistically significant without fear of being misunderstood; same statement, made in connection with an unbalanced factorial design, provides almost no information as it stands and is virtually worthless unless additional information is provided. The truth of this statement becomes evident when various hypotheses listed in Table 1 are considered. All of hypotheses noted have been characterized as being for unbalanced factorial designs2 and all have been suggested in standard research literature and/or have been incorporated into various packaged computer programs designed to analyze unbalanced factorial designs (1, 3, 5, 6, 7, 8,9, 10,12, 13,14,15,16, 17,18,19, 20, 21,24).3 In case of unbalanced (and nonproportional) designs,4 these hypotheses are clearly different from one another, not only mathematically, but conceptually as well. For example, expected cell means of some hypotheses are mathematically weighted in one manner, those of some hypotheses in another manner, and those of still others are not weighted at all. As a result, very different meanings must be attached to these various hypotheses', they are not simply different approximations of some single ideal hypothesis, but rather must be considered as distinct enti ties. A brief discussion of each hypothesis should make this point more meaningful.