Identifying settings when permutation tests have nominal size with paired, binary-outcome, group randomized trials

Abstract

There is an increasing use of permutation tests for the analysis of paired group randomized trials (GRTs). However, a one-sample permutation test has a nominal size (i.e. actual size equals desired size) only if the pairwise differences have a distribution that is continuous and symmetric around zero. If there are differing cluster sizes within each pair, the assumption of symmetry is violated and the permutation test will be liberal. Furthermore, differences in cluster means of binary outcomes are discrete with positive probability mass at zero; if the distribution has point mass at zero, the permutation test will be conservative. We show that a one-sample permutation test based upon pairwise differences in cluster means of binary outcomes is likely to have nominal size if the study randomizes a reasonable number of pairs and each cluster contains a large number of subjects. Through simulation, we have quantified the deviation in Type I error rates to serve as a guide for future researchers wishing to design a paired GRT with binary outcomes.