ROC-based assessments of 3D cortical surface-matching algorithms

Abstract

Algorithms for the semi-automated analysis of brain surfaces have recently received considerable attention, and yet, they rarely receive a rigorous assessment of their performance. We present a method for the quantitative assessment of performance across differing surface analysis algorithms and across various modifications of a single algorithm. The sensitivity and specificity of an algorithm for detecting known deformations added synthetically to the brains being studied are assessed using curves for Receiver Operating Characteristics (ROC). We also present a method for the isolation of sources of variance in MRI data sets that can contribute to degradation in performance of surface-matching algorithms. Isolation of these sources of variance allows determination of whether degradation in performance of surface-matching algorithms derives primarily from errors in registration of brains to a common coordinate space, from errors in placement of the known deformation, or from interindividual or between-group variability in morphology of the cortical surface. We apply these methods to the study of surface-matching algorithms that are based on fluid flow (FF) deformation, geodesic (GD) interpolation, or nearest neighbor (NN) proximity. We show that the performances of surface-matching algorithms depend on the presence of interindividual and between-group variability in the surfaces surrounding the cortical deformation. We also show that, in general, the FF algorithm performs as well as or better than the GD and NN algorithms. The large variance in identifying point correspondences across brain surfaces using the GD and the NN algorithms suggests strongly that these point correspondences are less valid than those determined by the FF algorithm. The GD and NN algorithms, moreover, are both vulnerable to detecting false-positive activations at points of high curvature, particularly along large fissures, cisterns, and cortical sulci.