Permutation-based group sequential analyses for cognitive neuroscience

Abstract

Cognitive neuroscientists have been grappling with two related experimental design problems. First, the complexity of neuroimaging data (e.g. often hundreds of thousands of correlated measurements) and analysis pipelines demands bespoke, non-parametric statistical tests for valid inference, and these tests often lack an agreed-upon method for performing a priori power analyses. Thus, sample size determination for neuroimaging studies is often arbitrary or inferred from other putatively but questionably similar studies, which can result in underpowered designs – undermining the efficacy of neuroimaging research. Second, when meta-analyses estimate the sample sizes required to obtain reasonable statistical power, estimated sample sizes can be prohibitively large given the resource constraints of many labs. We propose the use of sequential analyses to partially address both of these problems. Sequential study designs – in which the data is analyzed at interim points during data collection and data collection can be stopped if the planned test statistic satisfies a stopping rule specified a priori – are common in the clinical trial literature, due to the efficiency gains they afford over fixed-sample designs. However, the corrections used to control false positive rates in existing approaches to sequential testing rely on parametric assumptions that are often violated in neuroimaging settings. We introduce a general permutation scheme that allows sequential designs to be used with arbitrary test statistics. By simulation, we show that this scheme controls the false positive rate across multiple interim analyses. Then, performing power analyses for seven evoked response effects seen in the EEG literature, we show that this sequential analysis approach can substantially outperform fixed-sample approaches (i.e. require fewer subjects, on average, to detect a true effect) when study designs are sufficiently well-powered. To facilitate the adoption of this methodology, we provide a Python package “niseq” with sequential implementations of common tests used for neuroimaging: cluster-based permutation tests, threshold-free cluster enhancement, t-max, F-max, and the network-based statistic with tutorial examples using EEG and fMRI data.