The impact of sample size on the reproducibility of voxel-based lesion-deficit mappings

Diego L Lorca-Puls,Andrea Gajardo-Vidal,Jitrachote White,Mohamed L Seghier,Alexander P Leff,David W Green,Jenny T Crinion,Philipp Ludersdorfer,Thomas M.H Hope,Howard Bowman,Cathy J Price

doi:10.1016/j.neuropsychologia.2018.03.014

Diego L Lorca-Puls, Andrea Gajardo-Vidal + Show 9 more

Open Access

https://doi.org/10.1016/j.neuropsychologia.2018.03.014

Copy DOI

Abstract

This study investigated how sample size affects the reproducibility of findings from univariate voxel-based lesion-deficit analyses (e.g., voxel-based lesion-symptom mapping and voxel-based morphometry). Our effect of interest was the strength of the mapping between brain damage and speech articulation difficulties, as measured in terms of the proportion of variance explained. First, we identified a region of interest by searching on a voxel-by-voxel basis for brain areas where greater lesion load was associated with poorer speech articulation using a large sample of 360 right-handed English-speaking stroke survivors. We then randomly drew thousands of bootstrap samples from this data set that included either 30, 60, 90, 120, 180, or 360 patients. For each resample, we recorded effect size estimates and p values after conducting exactly the same lesion-deficit analysis within the previously identified region of interest and holding all procedures constant. The results show (1) how often small effect sizes in a heterogeneous population fail to be detected; (2) how effect size and its statistical significance varies with sample size; (3) how low-powered studies (due to small sample sizes) can greatly over-estimate as well as under-estimate effect sizes; and (4) how large sample sizes (N ≥ 90) can yield highly significant p values even when effect sizes are so small that they become trivial in practical terms. The implications of these findings for interpreting the results from univariate voxel-based lesion-deficit analyses are discussed.

Highlights

Using data from a large sample of stroke patients, we firstly estimated the magnitude of a lesion-deficit mapping of interest and formally investigated how effect size and its statistical significance varies with sample size
In Analysis 2, we investigated how sample size affected the reproducibility of the lesion-deficit mapping within the region of interest identified in Analysis 1
Poorer speech articulation was significantly associated with greater lesion load in 549 voxels (= 4.4 cm3 in size; see Table 3)

Summary

Introduction

There is a great deal of evidence showing how both false positive and false negative results increase as sample size decreases (Bakker et al, 2012; Button et al, 2013a; Chen et al, 2018; Cremers et al, 2017; Ingre, 2013; Ioannidis, 2008) and how inadequate statistical power can lead to replication failures (Anderson et al, 2017; Bakker et al, 2012; Perugini et al, 2014; Simonsohn et al, 2014a; Szucs and Ioannidis, 2017). Using data from a large sample of stroke patients, we firstly estimated the magnitude of a lesion-deficit mapping of interest and formally investigated how effect size and its statistical significance varies with sample size. Statistically significant findings when sample sizes are large can hide the fact that the effect under investigation might be of little importance in practical terms, or, even worse, the result of random chance alone and thereby a false

Objectives

Methods

Results

Discussion

Conclusion