Multi-level block permutation

Anderson M Winkler,Matthew A Webster,Diego Vidaurre,Thomas E Nichols,Stephen M Smith

doi:10.1016/j.neuroimage.2015.05.092

Anderson M Winkler, Matthew A Webster + Show 3 more

Open Access

https://doi.org/10.1016/j.neuroimage.2015.05.092

Copy DOI

Abstract

Under weak and reasonable assumptions, mainly that data are exchangeable under the null hypothesis, permutation tests can provide exact control of false positives and allow the use of various non-standard statistics. There are, however, various common examples in which global exchangeability can be violated, including paired tests, tests that involve repeated measurements, tests in which subjects are relatives (members of pedigrees) — any dataset with known dependence among observations. In these cases, some permutations, if performed, would create data that would not possess the original dependence structure, and thus, should not be used to construct the reference (null) distribution. To allow permutation inference in such cases, we test the null hypothesis using only a subset of all otherwise possible permutations, i.e., using only the rearrangements of the data that respect exchangeability, thus retaining the original joint distribution unaltered. In a previous study, we defined exchangeability for blocks of data, as opposed to each datum individually, then allowing permutations to happen within block, or the blocks as a whole to be permuted. Here we extend that notion to allow blocks to be nested, in a hierarchical, multi-level definition. We do not explicitly model the degree of dependence between observations, only the lack of independence; the dependence is implicitly accounted for by the hierarchy and by the permutation scheme. The strategy is compatible with heteroscedasticity and variance groups, and can be used with permutations, sign flippings, or both combined. We evaluate the method for various dependence structures, apply it to real data from the Human Connectome Project (HCP) as an example application, show that false positives can be avoided in such cases, and provide a software implementation of the proposed approach.

Highlights

In the context of hypothesis testing using the general linear model (GLM) (Scheffé, 1959; Searle, 1971), permutation tests can provide exact or approximately exact control of false positives, and allow the use of various non-standard statistics, all under weak and reasonable assumptions, mainly that the data are exchangeable under the null hypothesis, that is, that the joint distribution of the error terms remains unaltered after permutation
Permutations combined with sign flippings showed minimal power changes that were unrelated to the average Hamming distance, and with losses that were smaller than for just permutations or just sign flippings, suggesting that when both EE and ISE are valid for a given model, permutations with sign flippings can allow maximum efficiency
Since most studies — and most of those referenced in the previous paragraph — investigated only the relationship between one independent versus one dependent variable, for which no such correction is necessary, the results shown emulate well the risk of false positives in similar, real studies

Summary

Introduction

In the context of hypothesis testing using the general linear model (GLM) (Scheffé, 1959; Searle, 1971), permutation tests can provide exact or approximately exact control of false positives, and allow the use of various non-standard statistics, all under weak and reasonable assumptions, mainly that the data are exchangeable under the null hypothesis, that is, that the joint distribution of the error terms remains unaltered after permutation. Permutation tests that compare, for instance, groups of subjects, are of great value for neuroimaging (Holmes et al, 1996; Nichols and Holmes, 2002), and in Winkler et al (2014), extensions were presented to more broadly allow tests in the form of a GLM, and to account for certain types of well structured non-independence between observations, which ordinarily would preclude the use of permutation methods. The use of EBs allows for variances to be heterogeneous, provided that the groups of observations sharing the same variance (i.e., variance groups, VGs) (Woolrich et al, 2004) are compatible with the EBs; for within-block exchangeability the VGs must coincide with the blocks, and for whole-block exchangeability they must include one or more observations from each block in a consistent order

Results

Discussion

Conclusion