Investigation of the robust rank-order test for non-normal populations with unequal variances: The case of reaction time.

Abstract

Abstract This article examines the performance of the robust rank-order (Fligner-Policello) test of treatment effects for populations with unequal variances. Both symmetric (normal) and skewed (ex-Gaussian) distributions are examined. The ex-Gaussian distribution is particularly relevant because it is representative of skewed distributions found in experimental psychology, most notably reaction time data. The results indicate that when testing the hypothesis of equal medians for the symmetric distribution, the Fligner-Policello test is conservative, whereas it performs inconsistently for the skewed population. In light of the findings, we discuss the various options presently available to researchers confronted with unequal variances and skewed data, including a discussion of whether testing for treatment effects for populations with unequal variances is of practical relevance. Experimental psychologists are often interested in testing hypotheses about treatment effects in their experimental designs. The commonly used significance tests using t or F statistics involve the assumptions that the populations are normally distributed and the population variances are equal. The violation of the latter assumption is often referred to as the Behrens-Fisher problem (Scheffe, 1970) in honour of two pioneering statisticians who drew attention to the violation of this assumption and provided an early solution. In this article, we consider the simultaneous violation of both normality and equal variances, and we evaluate a nonparametric statistical method that is designed for use when neither of these assumptions is met. Consider a common scenario in experimental research: A researcher is interested in the reaction times (recorded in milliseconds) in a perceptual judgement task. The design consists of two independent groups, wherein group A is a control group and group B is an experimental group. As expected with reaction time data, the distribution of scores in each group is skewed (mean minus median equals 92 milliseconds); however, the variance in the experimental group is 2.25 times larger than that in the control group. The researcher is interested in testing the equality of the two sample means or, alternatively, medians. Given that the samples are skewed, it is generally recommended that a nonparametric inference procedure, usually the Wilcoxon-Mann-Whitney test, be used (Harwell & Serlin, 1989). However, the Wilcoxon-Mann-Whitney test is inappropriate for cases with unequal variances (see, e.g., Harwell, Rubinstein, Hayes, & Olds, 1992; Zimmerman & Zumbo, 1993a; 1993b). In fact, it is noteworthy that the Wilcoxon-Mann-Whitney test behaves very much like the Student t test: When the larger variance is associated with the larger sample size, there is a depression of the Type I error rate; when the larger variance is associated with the smaller sample size, there is an elevation of the Type I error rate (Zimmerman & Zumbo, 1993a; 1993b). Siegel and Castellan (1988), however, recommend that in the case of non-normality and unequal variances, an alternative nonparametric inference procedure is the Fligner-Policello (1981) test, also known as the robust rank-order test. It should be noted that the Behrens-Fisher problem was originally formulated only for the case of normal population distributions (Scheffe, 1970). The scenario described in our example, non-normality and unequal variances, is therefore referred to in the statistical literature as the generalized Behrens-Fisher problem. Although solutions to the Behrens-Fisher problem are discussed in some textbooks on experimental design (e.g., Winer, Brown, & Michels, 1991), to our knowledge, the only such discussion of solutions to the generalized Behrens-Fisher problem is Siegel and Castellan's (1988; pp. 137-144) presentation of the robust rank-order test (Fligner & Policello, 1981). Therefore, the purposes of the present article are: (a) to present the Fligner-Policello test in more detail, noting some important points overlooked by Siegel and Castellan in their presentation, and (b) to examine the performance of the Fligner-Policello test when applied to a non-normal distribution representative of those found in experimental psychology, especially in reaction time data. …