The fixed — effect fallacy in educational psychological research: A problem of generalizing to multiple populations

Abstract

A common statistical error in educational psychology involves the failure to treat language materials (e.g., words, sentences, prose passages) as a random effect in the analysis of variance. The fallacy of treating language items as a fixed effect limits the generalizability of research findings to the particular items used in an experiment, thus questioning the scientific worth of such studies. A review of social science citations suggests that few researchers are aware of the problem outside such areas as psycholinguistics, verbal learning, and cognitive psychology. A review of 35 studies in educational psychology revealed that the statistical error of failing to treat language items as a random effect was committed 32 times in these 35 studies. A possible methodological solution to the problem is discussed, along with several statistical solutions. The language‐as‐fixed‐effect fallacy is discussed as a special case of a more general problem involving the identification of relevant sampling variables representative of populations to which significant findings must be generalized if a study is to be scientifically worthwhile.