Randomization tests: statistics for experimenters

Abstract

Randomization (permutation) tests free the experimenter from the constraints of random sampling, a known error distribution and equal variances, to give a direct answer to the question “how likely is such a large (or small) result if the applied treatments had no effect?” The “result” may be the difference in mean responses, a correlation coefficient or any other value of interest. A randomization test is not a different statistical test but a different, and always valid, method of determining statistical significance. The familiar t-test and F-test can be carried out by data permutation without any parametric assumptions being fulfilled. A particular advantage of this method is that unbalanced designs and missing values are easily accommodated. Even with only a small number of subjects the number of permutations will be large and a computer is necessary if the randomization test is to be of practical value. To make this method of determining statistical significance generally available an interactive microcomputer program, forming a comprehensive package for the design and analysis of experiments, has been prepared.