Multi-site Neuroimaging Studies Research Articles

Characterization of the effects of outliers on ComBat harmonization for removing inter-site data heterogeneity in multisite neuroimaging studies.

Data harmonization is a key step widely used in multisite neuroimaging studies to remove inter-site heterogeneity of data distribution. However, data harmonization may even introduce additional inter-site differences in neuroimaging data if outliers are present in the data of one or more sites. It remains unclear how the presence of outliers could affect the effectiveness of data harmonization and consequently the results of analyses using harmonized data. To address this question, we generated a normal simulation dataset without outliers and a series of simulation datasets with outliers of varying properties (e.g., outlier location, outlier quantity, and outlier score) based on a real large-sample neuroimaging dataset. We first verified the effectiveness of the most commonly used ComBat harmonization method in the removal of inter-site heterogeneity using the normal simulation data, and then characterized the effects of outliers on the effectiveness of ComBat harmonization and on the results of association analyses between brain imaging-derived phenotypes and a simulated behavioral variable using the simulation datasets with outliers. We found that, although ComBat harmonization effectively removed the inter-site heterogeneity in multisite data and consequently improved the detection of the true brain-behavior relationships, the presence of outliers could damage severely the effectiveness of ComBat harmonization in the removal of data heterogeneity or even introduce extra heterogeneity in the data. Moreover, we found that the effects of outliers on the improvement of the detection of brain-behavior associations by ComBat harmonization were dependent on how such associations were assessed (i.e., by Pearson correlation or Spearman correlation), and on the outlier location, quantity, and outlier score. These findings help us better understand the influences of outliers on data harmonization and highlight the importance of detecting and removing outliers prior to data harmonization in multisite neuroimaging studies.

The impact of harmonization on radiomic features in Parkinson's disease and healthy controls: A multicenter study.

Radiomics is a challenging development area in imaging field that is greatly capturing interest of radiologists and neuroscientists. However, radiomics features show a strong non-biological variability determined by different facilities and imaging protocols, limiting the reproducibility and generalizability of analysis frameworks. Our study aimed to investigate the usefulness of harmonization to reduce site-effects on radiomics features over specific brain regions. We selected T1-weighted magnetic resonance imaging (MRI) by using the MRI dataset Parkinson’s Progression Markers Initiative (PPMI) from different sites with healthy controls (HC) and Parkinson’s disease (PD) patients. First, the investigation of radiomics measure discrepancies were assessed on healthy brain regions-of-interest (ROIs) via a classification pipeline based on LASSO feature selection and support vector machine (SVM) model. Then, a ComBat-based harmonization approach was applied to correct site-effects. Finally, a validation step on PD subjects evaluated diagnostic accuracy before and after harmonization of radiomics data. Results on healthy subjects demonstrated a dependence from site-effects that could be corrected with ComBat harmonization. LASSO regressor after harmonization was unable to select any feature to distinguish controls by site. Moreover, harmonized radiomics features achieved an area under the receiving operating characteristic curve (AUC) of 0.77 (compared to AUC of 0.71 for raw radiomics measures) in distinguish Parkinson’s patients from HC. We found a not-negligible site-effect studying radiomics of HC pre- and post-harmonization of features. Our validation study on PD patients demonstrated a significant influence of non-biological noise source in diagnostic performances. Finally, harmonization of multicenter radiomic data represent a necessary step to make analysis pipelines reliable and replicable for multisite neuroimaging studies.

Variability and reproducibility of multi-echo T2 relaxometry: Insights from multi-site, multi-session and multi-subject MRI acquisitions.

Quantitative magnetic resonance imaging (qMRI) can increase the specificity and sensitivity of conventional weighted MRI to underlying pathology by comparing meaningful physical or chemical parameters, measured in physical units, with normative values acquired in a healthy population. This study focuses on multi-echo T2 relaxometry, a qMRI technique that probes the complex tissue microstructure by differentiating compartment-specific T2 relaxation times. However, estimation methods are still limited by their sensitivity to the underlying noise. Moreover, estimating the model's parameters is challenging because the resulting inverse problem is ill-posed, requiring advanced numerical regularization techniques. As a result, the estimates from distinct regularization strategies are different. In this work, we aimed to investigate the variability and reproducibility of different techniques for estimating the transverse relaxation time of the intra- and extra-cellular space () in gray (GM) and white matter (WM) tissue in a clinical setting, using a multi-site, multi-session, and multi-run T2 relaxometry dataset. To this end, we evaluated three different techniques for estimating the T2 spectra (two regularized non-negative least squares methods and a machine learning approach). Two independent analyses were performed to study the effect of using raw and denoised data. For both the GM and WM regions, and the raw and denoised data, our results suggest that the principal source of variance is the inter-subject variability, showing a higher coefficient of variation (CoV) than those estimated for the inter-site, inter-session, and inter-run, respectively. For all reconstruction methods studied, the CoV ranged between 0.32 and 1.64%. Interestingly, the inter-session variability was close to the inter-scanner variability with no statistical differences, suggesting that is a robust parameter that could be employed in multi-site neuroimaging studies. Furthermore, the three tested methods showed consistent results and similar intra-class correlation (ICC), with values superior to 0.7 for most regions. Results from raw data were slightly more reproducible than those from denoised data. The regularized non-negative least squares method based on the L-curve technique produced the best results, with ICC values ranging from 0.72 to 0.92.

Inter-scanner reproducibility of brain volumetry: influence of automated brain segmentation software

BackgroundThe inter-scanner reproducibility of brain volumetry is important in multi-site neuroimaging studies, where the reliability of automated brain segmentation (ABS) tools plays an important role. This study aimed to evaluate the influence of ABS tools on the consistency and reproducibility of the quantified brain volumetry from different scanners.MethodsWe included fifteen healthy volunteers who were scanned with 3D isotropic brain T1-weighted sequence on three different 3.0 Tesla MRI scanners (GE, Siemens and Philips). For each individual, the time span between image acquisitions on different scanners was limited to 1 h. All the T1-weighted images were processed with FreeSurfer v6.0, FSL v5.0 and AccuBrain® with default settings to obtain volumetry of brain tissues (e.g. gray matter) and substructures (e.g. basal ganglia structures) if available. Coefficient of variation (CV) was calculated to test inter-scanner variability in brain volumetry of various structures as quantified by these ABS tools.ResultsThe mean inter-scanner CV values per brain structure among three MRI scanners ranged from 6.946 to 12.29% (mean, 9.577%) for FreeSurfer, 7.245 to 20.98% (mean, 12.60%) for FSL and 1.348 to 8.800% (mean value, 3.546%) for AccuBrain®. In addition, AccuBrain® and FreeSurfer achieved the lowest mean values of region-specific CV between GE and Siemens scanners (from 0.818 to 5.958% for AccuBrain®, and from 0.903 to 7.977% for FreeSurfer), while FSL-FIRST had the lowest mean values of region-specific CV between GE and Philips scanners (from 2.603 to 16.310%). AccuBrain® also had the lowest mean values of region-specific CV between Siemens and Philips scanners (from 1.138 to 6.615%).ConclusionThere is a large discrepancy in the inter-scanner reproducibility of brain volumetry when using different processing software. Image acquisition protocols and selection of ABS tool for brain volumetry quantification have impact on the robustness of results in multi-site studies.

Intensity warping for multisite MRI harmonization

In multisite neuroimaging studies there is often unwanted technical variation across scanners and sites. These “scanner effects” can hinder detection of biological features of interest, produce inconsistent results, and lead to spurious associations. We propose mica (multisite image harmonization by cumulative distribution function alignment), a tool to harmonize images taken on different scanners by identifying and removing within-subject scanner effects. Our goals in the present study were to (1) establish a method that removes scanner effects by leveraging multiple scans collected on the same subject, and, building on this, (2) develop a technique to quantify scanner effects in large multisite studies so these can be reduced as a preprocessing step. We illustrate scanner effects in a brain MRI study in which the same subject was measured twice on seven scanners, and assess our method’s performance in a second study in which ten subjects were scanned on two machines. We found that unharmonized images were highly variable across site and scanner type, and our method effectively removed this variability by aligning intensity distributions. We further studied the ability to predict image harmonization results for a scan taken on an existing subject at a new site using cross-validation.

Assessment of quantitative magnetic resonance imaging metrics in the brain through the use of a novel phantom

ABSTRACTObjective: Multisite and longitudinal neuroimaging studies are important in uncovering trajectories of recovery and neurodegeneration following traumatic brain injury (TBI) and concussion through the use of diffusion tensor imaging (DTI) and other imaging modalities. This study assessed differences in anisotropic diffusion measurement across four scanners using a human and a novel phantom developed in conjunction with the Chronic Effects of Neurotrauma Consortium.Method: Human scans provided measurement within biological tissue, and the novel physical phantom provided measures of anisotropic intra-tubular diffusion to serve as a model for intra-axonal water diffusion. Intra- and inter-scanner measurement variances were compared, and the impact on effect size was calculated.Results: Intra-scanner test–retest reliability estimates for fractional anisotropy (FA) demonstrated relative stability over testing intervals. The human tissue and phantom showed similar FA ranges, high linearity and large within-device effect sizes. However, inter-scanner measures of FA indicated substantial differences, some of which exceeded typical DTI effect sizes in mild TBI.Conclusion: The diffusion phantom may be used to better elucidate inter-scanner variability in DTI-based measurement and provides an opportunity to better calibrate results obtained from scanners used in multisite and longitudinal studies. Novel solutions are being evaluated to understand and potentially overcome these differences.

Statistical harmonization corrects site effects in functional connectivity measurements from multi‐site fMRI data

Acquiring resting-state functional magnetic resonance imaging (fMRI) datasets at multiple MRI scanners and clinical sites can improve statistical power and generalizability of results. However, multi-site neuroimaging studies have reported considerable nonbiological variability in fMRI measurements due to different scanner manufacturers and acquisition protocols. These undesirable sources of variability may limit power to detect effects of interest and may even result in erroneous findings. Until now, there has not been an approach that removes unwanted site effects. In this study, using a relatively large multi-site (4 sites) fMRI dataset, we investigated the impact of site effects on functional connectivity and network measures estimated by widely used connectivity metrics and brain parcellations. The protocols and image acquisition of the dataset used in this study had been homogenized using identical MRI phantom acquisitions from each of the neuroimaging sites; however, intersite acquisition effects were not completely eliminated. Indeed, in this study, we found that the magnitude of site effects depended on the choice of connectivity metric and brain atlas. Therefore, to further remove site effects, we applied ComBat, a harmonization technique previously shown to eliminate site effects in multi-site diffusion tensor imaging (DTI) and cortical thickness studies. In the current work, ComBat successfully removed site effects identified in connectivity and network measures and increased the power to detect age associations when using optimal combinations of connectivity metrics and brain atlases. Our proposed ComBat harmonization approach for fMRI-derived connectivity measures facilitates reliable and efficient analysis of retrospective and prospective multi-site fMRI neuroimaging studies.

Harmonization of cortical thickness measurements across scanners and sites

With the proliferation of multi-site neuroimaging studies, there is a greater need for handling non-biological variance introduced by differences in MRI scanners and acquisition protocols. Such unwanted sources of variation, which we refer to as “scanner effects”, can hinder the detection of imaging features associated with clinical covariates of interest and cause spurious findings. In this paper, we investigate scanner effects in two large multi-site studies on cortical thickness measurements across a total of 11 scanners. We propose a set of tools for visualizing and identifying scanner effects that are generalizable to other modalities. We then propose to use ComBat, a technique adopted from the genomics literature and recently applied to diffusion tensor imaging data, to combine and harmonize cortical thickness values across scanners. We show that ComBat removes unwanted sources of scan variability while simultaneously increasing the power and reproducibility of subsequent statistical analyses. We also show that ComBat is useful for combining imaging data with the goal of studying life-span trajectories in the brain.

Removing inter-subject technical variability in magnetic resonance imaging studies

Magnetic resonance imaging (MRI) intensities are acquired in arbitrary units, making scans non-comparable across sites and between subjects. Intensity normalization is a first step for the improvement of comparability of the images across subjects. However, we show that unwanted inter-scan variability associated with imaging site, scanner effect, and other technical artifacts is still present after standard intensity normalization in large multi-site neuroimaging studies. We propose RAVEL (Removal of Artificial Voxel Effect by Linear regression), a tool to remove residual technical variability after intensity normalization. As proposed by SVA and RUV [Leek and Storey, 2007, 2008, Gagnon-Bartsch and Speed, 2012], two batch effect correction tools largely used in genomics, we decompose the voxel intensities of images registered to a template into a biological component and an unwanted variation component. The unwanted variation component is estimated from a control region obtained from the cerebrospinal fluid (CSF), where intensities are known to be unassociated with disease status and other clinical covariates. We perform a singular value decomposition (SVD) of the control voxels to estimate factors of unwanted variation. We then estimate the unwanted factors using linear regression for every voxel of the brain and take the residuals as the RAVEL-corrected intensities. We assess the performance of RAVEL using T1-weighted (T1-w) images from more than 900 subjects with Alzheimer's disease (AD) and mild cognitive impairment (MCI), as well as healthy controls from the Alzheimer's Disease Neuroimaging Initiative (ADNI) database. We compare RAVEL to two intensity-normalization-only methods: histogram matching and White Stripe. We show that RAVEL performs best at improving the replicability of the brain regions that are empirically found to be most associated with AD, and that these regions are significantly more present in structures impacted by AD (hippocampus, amygdala, parahippocampal gyrus, enthorinal area, and fornix stria terminals). In addition, we show that the RAVEL-corrected intensities have the best performance in distinguishing between MCI subjects and healthy subjects using the mean hippocampal intensity (AUC=67%), a marked improvement compared to results from intensity normalization alone (AUC=63% and 59% for histogram matching and White Stripe, respectively). RAVEL is promising for many other imaging modalities.

Reliability of an fMRI paradigm for emotional processing in a multisite longitudinal study.

Multisite neuroimaging studies can facilitate the investigation of brain-related changes in many contexts, including patient groups that are relatively rare in the general population. Though multisite studies have characterized the reliability of brain activation during working memory and motor functional magnetic resonance imaging tasks, emotion processing tasks, pertinent to many clinical populations, remain less explored. A traveling participants study was conducted with eight healthy volunteers scanned twice on consecutive days at each of the eight North American Longitudinal Prodrome Study sites. Tests derived from generalizability theory showed excellent reliability in the amygdala ( Eρ2 = 0.82), inferior frontal gyrus (IFG; Eρ2 = 0.83), anterior cingulate cortex (ACC; Eρ2 = 0.76), insula ( Eρ2 = 0.85), and fusiform gyrus ( Eρ2 = 0.91) for maximum activation and fair to excellent reliability in the amygdala ( Eρ2 = 0.44), IFG ( Eρ2 = 0.48), ACC ( Eρ2 = 0.55), insula ( Eρ2 = 0.42), and fusiform gyrus ( Eρ2 = 0.83) for mean activation across sites and test days. For the amygdala, habituation ( Eρ2 = 0.71) was more stable than mean activation. In a second investigation, data from 111 healthy individuals across sites were aggregated in a voxelwise, quantitative meta-analysis. When compared with a mixed effects model controlling for site, both approaches identified robust activation in regions consistent with expected results based on prior single-site research. Overall, regions central to emotion processing showed strong reliability in the traveling participants study and robust activation in the aggregation study. These results support the reliability of blood oxygen level-dependent signal in emotion processing areas across different sites and scanners and may inform future efforts to increase efficiency and enhance knowledge of rare conditions in the population through multisite neuroimaging paradigms.

Reliability of functional magnetic resonance imaging activation during working memory in a multi-site study: Analysis from the North American Prodrome Longitudinal Study

Multi-site neuroimaging studies offer an efficient means to study brain functioning in large samples of individuals with rare conditions; however, they present new challenges given that aggregating data across sites introduces additional variability into measures of interest. Assessing the reliability of brain activation across study sites and comparing statistical methods for pooling functional data are critical to ensuring the validity of aggregating data across sites. The current study used two samples of healthy individuals to assess the feasibility and reliability of aggregating multi-site functional magnetic resonance imaging (fMRI) data from a Sternberg-style verbal working memory task. Participants were recruited as part of the North American Prodrome Longitudinal Study (NAPLS), which comprises eight fMRI scanning sites across the United States and Canada. In the first study sample (n=8), one participant from each home site traveled to each of the sites and was scanned while completing the task on two consecutive days. Reliability was examined using generalizability theory. Results indicated that blood oxygen level-dependent (BOLD) signal was reproducible across sites and was highly reliable, or generalizable, across scanning sites and testing days for core working memory ROIs (generalizability ICCs=0.81 for left dorsolateral prefrontal cortex, 0.95 for left superior parietal cortex). In the second study sample (n=154), two statistical methods for aggregating fMRI data across sites for all healthy individuals recruited as control participants in the NAPLS study were compared. Control participants were scanned on one occasion at the site from which they were recruited. Results from the image-based meta-analysis (IBMA) method and mixed effects model with site covariance method both showed robust activation in expected regions (i.e. dorsolateral prefrontal cortex, anterior cingulate cortex, supplementary motor cortex, superior parietal cortex, inferior temporal cortex, cerebellum, thalamus, basal ganglia). Quantification of the similarity of group maps from these methods confirmed a very high (96%) degree of spatial overlap in results. Thus, brain activation during working memory function was reliable across the NAPLS sites and both the IBMA and mixed effects model with site covariance methods appear to be valid approaches for aggregating data across sites. These findings indicate that multi-site functional neuroimaging can offer a reliable means to increase power and generalizability of results when investigating brain function in rare populations and support the multi-site investigation of working memory function in the NAPLS study, in particular.

Knowledge-Guided Robust MRI Brain Extraction for Diverse Large-Scale Neuroimaging Studies on Humans and Non-Human Primates

Accurate and robust brain extraction is a critical step in most neuroimaging analysis pipelines. In particular, for the large-scale multi-site neuroimaging studies involving a significant number of subjects with diverse age and diagnostic groups, accurate and robust extraction of the brain automatically and consistently is highly desirable. In this paper, we introduce population-specific probability maps to guide the brain extraction of diverse subject groups, including both healthy and diseased adult human populations, both developing and aging human populations, as well as non-human primates. Specifically, the proposed method combines an atlas-based approach, for coarse skull-stripping, with a deformable-surface-based approach that is guided by local intensity information and population-specific prior information learned from a set of real brain images for more localized refinement. Comprehensive quantitative evaluations were performed on the diverse large-scale populations of ADNI dataset with over 800 subjects (55∼90 years of age, multi-site, various diagnosis groups), OASIS dataset with over 400 subjects (18∼96 years of age, wide age range, various diagnosis groups), and NIH pediatrics dataset with 150 subjects (5∼18 years of age, multi-site, wide age range as a complementary age group to the adult dataset). The results demonstrate that our method consistently yields the best overall results across almost the entire human life span, with only a single set of parameters. To demonstrate its capability to work on non-human primates, the proposed method is further evaluated using a rhesus macaque dataset with 20 subjects. Quantitative comparisons with popularly used state-of-the-art methods, including BET, Two-pass BET, BET-B, BSE, HWA, ROBEX and AFNI, demonstrate that the proposed method performs favorably with superior performance on all testing datasets, indicating its robustness and effectiveness.

Unbiased tensor-based morphometry: Improved robustness and sample size estimates for Alzheimer's disease clinical trials

Various neuroimaging measures are being evaluated for tracking Alzheimer's disease (AD) progression in therapeutic trials, including measures of structural brain change based on repeated scanning of patients with magnetic resonance imaging (MRI). Methods to compute brain change must be robust to scan quality. Biases may arise if any scans are thrown out, as this can lead to the true changes being overestimated or underestimated. Here we analyzed the full MRI dataset from the first phase of Alzheimer's Disease Neuroimaging Initiative (ADNI-1) from the first phase of Alzheimer's Disease Neuroimaging Initiative (ADNI-1) and assessed several sources of bias that can arise when tracking brain changes with structural brain imaging methods, as part of a pipeline for tensor-based morphometry (TBM). In all healthy subjects who completed MRI scanning at screening, 6, 12, and 24months, brain atrophy was essentially linear with no detectable bias in longitudinal measures. In power analyses for clinical trials based on these change measures, only 39AD patients and 95 mild cognitive impairment (MCI) subjects were needed for a 24-month trial to detect a 25% reduction in the average rate of change using a two-sided test (α=0.05, power=80%). Further sample size reductions were achieved by stratifying the data into Apolipoprotein E (ApoE) ε4 carriers versus non-carriers. We show how selective data exclusion affects sample size estimates, motivating an objective comparison of different analysis techniques based on statistical power and robustness. TBM is an unbiased, robust, high-throughput imaging surrogate marker for large, multi-site neuroimaging studies and clinical trials of AD and MCI.

Electronic Data Capture, Representation, and Applications for Neuroimaging

With growing emphasis on data sharing across neuroimaging studies in humans, both within and across institutions and studies, there is a growing awareness of the need for clear connections between the images and the other data about the subject, and between the images and the behavioral or physiological data that is relevant for analysis (Van Horn and Toga, 2009a). Means for electronic data capture, storage, organization, and levels of interactivity are required for the large and rich datasets which modern neuroimaging studies now generate and seek to analyze (Helmer et al., 2011). Beyond brain imaging volumes, researchers are including detailed meta-data on scan data types, cognitive protocols, phenomics, and graphical renderings as part of shareable dataset. Capturing these data using automated and semi-automated means represents a particular challenge for subsequent examination, mining, and visualization (Figure ​(Figure1).1). The aim of this Frontiers in Neuroinformatics special topic was developed to sample from the current state of the art in automated and semi-automated data collection methods, data management, and their practical applications in neuroimaging research and related studies. Figure 1 Electronic data capture and efficient representation of neuroimaging enables comprehensive data mining and compelling visualization which, in turn, contributes to greater data sharing and openness in the brain sciences. These articles, from experts in data capture, management, and visualization, provide an overview of the ways different research teams have addressed similar issues across the various stages of data capture in large-scale neuroimaging studies. Articles focus on how databases are populated, how to make sure that the information so gathered is accurate, and how to manage it so that it can be successfully communicated to others. Specifically, these articles address: Automated and semi-automated data capture Two of the papers deal directly with specific methods for data collection during the study. Voyvodic et al. (this issue) present a software package that has been applied in several studies both locally and across multiple sites. The CIGAL software makes a point of capturing behavioral and physiological data during experimental tasks in a way that is interpretable both during the experiment and afterward, and usable in single and multi-site neuroimaging or other behavioral studies. The timing of the various events during the experiment can be extracted from the resulting text files for storage in a database, translation into XML, or use in analysis pipelines. Both the behavioral/physiological data capture of CIGAL and the clinical assessments capture of CARAT (Turner et al., 2010), make a point of representing the data in a way that is research-friendly and works with arbitrary data management systems. The CARAT package for collecting clinical measurements allows the user to connect to various databases to store the demographic and other data automatically, without need for transferring data from paper records.

Optimizing power to track brain degeneration in Alzheimer's disease and mild cognitive impairment with tensor-based morphometry: An ADNI study of 515 subjects

Tensor-based morphometry (TBM) is a powerful method to map the 3D profile of brain degeneration in Alzheimer's disease (AD) and mild cognitive impairment (MCI). We optimized a TBM-based image analysis method to determine what methodological factors, and which image-derived measures, maximize statistical power to track brain change. 3D maps, tracking rates of structural atrophy over time, were created from 1030 longitudinal brain MRI scans (1-year follow-up) of 104 AD patients (age: 75.7 ± 7.2 years; MMSE: 23.3 ± 1.8, at baseline), 254 amnestic MCI subjects (75.0 ± 7.2 years; 27.0 ± 1.8), and 157 healthy elderly subjects (75.9 ± 5.1 years; 29.1 ± 1.0), as part of the Alzheimer's Disease Neuroimaging Initiative (ADNI). To determine which TBM designs gave greatest statistical power, we compared different linear and nonlinear registration parameters (including different regularization functions), and different numerical summary measures derived from the maps. Detection power was greatly enhanced by summarizing changes in a statistically-defined region-of-interest (ROI) derived from an independent training sample of 22 AD patients. Effect sizes were compared using cumulative distribution function (CDF) plots and false discovery rate methods. In power analyses, the best method required only 48 AD and 88 MCI subjects to give 80% power to detect a 25% reduction in the mean annual change using a two-sided test (at α = 0.05). This is a drastic sample size reduction relative to using clinical scores as outcome measures (619 AD/6797 MCI for the ADAS-Cog, and 408 AD/796 MCI for the Clinical Dementia Rating sum-of-boxes scores). TBM offers high statistical power to track brain changes in large, multi-site neuroimaging studies and clinical trials of AD.

Quantitative evaluation of automated skull‐stripping methods applied to contemporary and legacy images: Effects of diagnosis, bias correction, and slice location

Performance of automated methods to isolate brain from nonbrain tissues in magnetic resonance (MR) structural images may be influenced by MR signal inhomogeneities, type of MR image set, regional anatomy, and age and diagnosis of subjects studied. The present study compared the performance of four methods: Brain Extraction Tool (BET; Smith [2002]: Hum Brain Mapp 17:143-155); 3dIntracranial (Ward [1999] Milwaukee: Biophysics Research Institute, Medical College of Wisconsin; in AFNI); a Hybrid Watershed algorithm (HWA, Segonne et al. [2004] Neuroimage 22:1060-1075; in FreeSurfer); and Brain Surface Extractor (BSE, Sandor and Leahy [1997] IEEE Trans Med Imag 16:41-54; Shattuck et al. [2001] Neuroimage 13:856-876) to manually stripped images. The methods were applied to uncorrected and bias-corrected datasets; Legacy and Contemporary T1-weighted image sets; and four diagnostic groups (depressed, Alzheimer's, young and elderly control). To provide a criterion for outcome assessment, two experts manually stripped six sagittal sections for each dataset in locations where brain and nonbrain tissue are difficult to distinguish. Methods were compared on Jaccard similarity coefficients, Hausdorff distances, and an Expectation-Maximization algorithm. Methods tended to perform better on contemporary datasets; bias correction did not significantly improve method performance. Mesial sections were most difficult for all methods. Although AD image sets were most difficult to strip, HWA and BSE were more robust across diagnostic groups compared with 3dIntracranial and BET. With respect to specificity, BSE tended to perform best across all groups, whereas HWA was more sensitive than other methods. The results of this study may direct users towards a method appropriate to their T1-weighted datasets and improve the efficiency of processing for large, multisite neuroimaging studies.