Postscript: Some closing comments on "Alternatives to classical statistics" (normal theory).

Abstract

Upon reflection, we were struck by the enormity of the task at hand. It was impossible to treat all of the alternatives to normal theory in a matter of two to three hours. Alternatives come from many perspectives and clearly demonstrate that methodology is vibrant in the social sciences. The topics selected were, in part, circumstantial and partly driven by a consideration of the fundamental assumptions of classical (normal theory) statistics.However, one very important point was not clearly stated in the proceedings. This point is most clearly articulated by Jacob Cohen in his 1965 monograph entitled Some statistical issues in Psychological (cf., Schutz & Gessaroli, 1992):Statistical analysis is a tool, not a ritualistic religion. It is for use, not for reverence, and it should be approached in the spirit that it was made for psychologists rather than vice versa. As one of many tools in the psychologist's kit, it is frequently not relevant and is sometimes of considerable utility. It is certainly not as important as, nor can it even partly replace good ideas or well - conceived experimental strategems, although it may be virtually indispensable in testing out an idea or rounding out a good experiment. (p. 95)This quotation is important when discussing alternatives to normal theory. It is not enough to simply go through the statistical gyrations hoping to sanctify the findings with a p - value less than .05. We believe that what is lacking in psychological research is a strong theoretical foundation which drives the conceptualization of the random variables (i.e., the measures), the type of study and the statistical analyses. An excellent example of this is experiments involving reaction time or response time measures. Reaction time experiments were chosen simply because some excellent theoretical work has clarified the generating process of reaction times, and therefore the conceptualization of the random variable. But the approach to this psychological phenomenon also clearly highlights how the knowledge of the random variable is underutilized in Psychological experimentation.Most psychological studies have rather heuristic models of the process being studied; an exception to this is decision making and cognition where many explicit models (both quantitative and otherwise) are proposed. It has been argued quite convincingly that reaction times are a realization of the Ex - gaussian or Gamma distribution (distributionswhich have very long tails and are often not symmetrical). At this point, researchers usually design experiments to test the equality of means in various experimental conditions and use reaction time as the dependent variable. The data is then processed through ordinary least - squares ANOVA. From our point of view, this process of experimentation has two problems. First, the researcher is not making use of the knowledge of the generating process. That is, ordinary least squares breaks down quickly with asymmetric and long - tailed data. Second, conceptualizing the study according to mean differences between groups is discarding the intricate theory which went into arguing for Ex - gaussian or Gamma generating processes. Zumbo and McMorran (1991) propose that rather than examine mean differences in reaction time it may be beneficial to model the data in each condition as an Ex - gaussian or Gamma process by methods of maximum - likelihood estimation. At this point, a likelihood ratio test can be used to test the difference in the rate parameters (i.e., response time parameters) in the various conditions. The Zumbo and McMorran approach makes use of the detailed substantive theory for modelling the process and testing parameters in different conditions rather than simply testing mean differences between groups. Psychologists need to leave testing mean differences behind them.Reflections on the SymposiumThe papers in this symposium reflect a change in statistical practice and data analysis in the last three decades. …