Most studies addressing lexical processing make use of factorial designs. For many researchers in this field of inquiry, a real experiment is a factorial experiment. Methods such as regression and factor analysis would not allow for hypothesis testing and would not contribute substantially to the advancement of scientific knowledge. Their use would be restricted to exploratory studies at best. This paper is an apology coming to the defense of regression designs for experiments including lexical distributional variables as predictors. In studies of the mental lexicon, we often are dealing with two kinds of predictors, to which I will refer as treatments and covariates. Stimulus-onset asynchrony (soa) is an example of a treatment. If we want to study the effect of a long versus a short soa, it makes sense to choose sensible values, say 200 ms versus 50 ms, and to run experiments with these two settings. If the researcher knows that the effect of soa is linear, and that it can be administered independently of the intrinsic properties of the items, then the optimal design testing for an effect of soa is factorial. One would loose power by using a regression design testing for an effect at a sequence of SOA intervals, say 50, 60, 70, . . . , 200 ms. This advantage of sampling at the extremes is well-known (see, e.g., Crawley, 2002, p. 67): the further apart the values of soa are, the larger the corresponding sum of squares, and the smaller the standard error for the slope. The advantage of designs with maximal contrasts for treatment predictors is often assumed to carry over to the study of lexical covariates such as frequency, length, neighborhood density, etc. In order to test for an effect of frequency, the traditional wisdom advises us to create a data set with very high-frequency words on the one hand, and very low-frequency words on the other hand. The problem that one runs into very quickly is that the set of highfrequency words will comprise short words with many neighbors, and that the low-frequency words will be long words with few neighbors. The massive correlations characterizing lexical properties create the problem that an effect of frequency could just as well be an effect of length or an effect of neighborhood density, or any combination of these variables. The traditional solution is to create a factorial contrast for frequency, while matching for the other predictors. This can be done by hand, or with the help of Maarten van Casteren’s mix program (Van Casteren and Davis, 2006). The aim of this contribution is to illustrate, by means of some simple simulations, that this matching process leads to a severe loss of power (following up on, e.g., Cohen, 1983; MacCallum et al., 2002). In all the simulations to follow, the dependent variable (RT) is a function of two numerical predictors, X1 (this could be log frequency, or the word’s imageability) and X2 (this could be number of orthographic neighbors, or word length), which both follow a standard normal
Read full abstract