Another Look at the Lady Tasting Tea and Differences Between Permutation Tests and Randomisation Tests

Jesse Hemerik,Jelle J Goeman

doi:10.1111/insr.12431

Abstract

SummaryThe statistical literature is known to be inconsistent in the use of the terms ‘permutation test’ and ‘randomisation test’. Several authors successfully argue that these terms should be used to refer to two distinct classes of tests and that there are major conceptual differences between these classes. The present paper explains an important difference in mathematical reasoning between these classes: a permutation test fundamentally requires that the set of permutations has a group structure, in the algebraic sense; the reasoning behind a randomisation test is not based on such a group structure, and it is possible to use an experimental design that does not correspond to a group. In particular, we can use a randomisation scheme where the number of possible treatment patterns is larger than in standard experimental designs. This leads to exact p values of improved resolution, providing increased power for very small significance levels, at the cost of decreased power for larger significance levels. We discuss applications in randomised trials and elsewhere. Further, we explain that Fisher's famous Lady Tasting Tea experiment, which is commonly referred to as the first permutation test, is in fact a randomisation test. This distinction is important to avoid confusion and invalid tests.

Highlights

The statistical literature is very inconsistent in the use of the terms ‘permutation tests’ and ‘randomisation tests’ (Onghena, 2018; Rosenberger et al, 2019)
Because permutation-based randomisation tests do not require a group structure, it can be useful to consider a randomisation scheme that does not correspond to a group
Referring to the lady tasting tea experiment as an example of a permutation test, as is often done, can put readers on the wrong foot, because the reasoning underlying this experiment is not based on a group structure

Summary

Introduction

The statistical literature is very inconsistent in the use of the terms ‘permutation tests’ and ‘randomisation tests’ (Onghena, 2018; Rosenberger et al, 2019). The confusion surrounding differences between such tests is an important issue, because there are major differences between permutation tests and randomisation tests in the sense of, for example, Onghena (2018), Kempthorne and Doerfler (1969) and Rosenberger et al (2019), whose definitions we will follow here. Those authors use these terms to refer to strictly distinct classes of tests and discuss the terms in detail.

Objectives

Findings

Discussion

Conclusion