Reliability of using single specialist annotation for designing and evaluating automatic segmentation methods: A skull stripping case study

Abstract

Manual annotation by specialists has been largely used as a gold standard for evaluation of automatic segmentation methods in medical imaging and also for designing datadriven approaches. The use of single-specialist manual annotation in this context may lead to a poor assessment and/or training of automatic segmentation methods. This work investigates the limitations of using single-specialist manual annotation as a gold standard. This investigation is framed within a common case study: application of skull stripping to magnetic resonance (MR) images. Twelve MR volumes acquired with scanners of different vendors and at different magnetic field strengths were manually annotated. Each volume was annotated twice. Nine publicly available automatic skull stripping methods were applied to these twelve volumes. The Dice coefficient overlap metric was used to rank the automatic skull stripping methods. Our results show differences (p < 0.05) that depended on the manual annotation reference used to compute the metrics. The ranking of the best skull stripping methods changed with the gold standard. Manual annotation also plays a fundamental role in data-driven approaches, such as convolutional neural networks (CNNs). An auto-context CNN was evaluated using two public datasets in addition to our twelve volumes, to assess manual annotation influence both during training and validation of the algorithm. Our results indicate that the CNN has a strong bias towards the manual annotation used during training, potentially resulting in poor generalization of the model. These results highlight the importance of having robust gold standards, and raise the need for developing better methodologies for generating gold standards.