The High Cost of Complexity in Experimental Design and Data Analysis: Type I and Type II Error Rates in Multiway ANOVA

Abstract

The availability of statistical software packages has led to a sharp increase in use of complex research designs and complex statistical analyses in communication research. An informal examination of studies from 2 leading communication journals suggests that the analysis of variance (ANOVA) is often the statistic of choice, and a substantial proportion of published research reports using ANOVA employ complex (k ≥ 3) factorial designs, often involving multiple dependent variables. This article reports a series of Monte Carlo simulations which demonstrate that this complexity may come at a heavier cost than many communication researchers realize. As frequently used, complex factorial ANOVA yield Type I and Type II error rates that many communication scholars would likely consider unacceptable. Consequently, quality of statistical inference in many studies is highly suspect. Communication researchers are warned about problems associated with design and statistical complexity and solutions are suggested.