Validate your white matter tractography algorithms with a reappraised ISMRM 2015 Tractography Challenge scoring system

Abstract

Since 2015, research groups have sought to produce the ne plus ultra of tractography algorithms using the ISMRM 2015 Tractography Challenge as evaluation. In particular, since 2017, machine learning has made its entrance into the tractography world. The ISMRM 2015 Tractography Challenge is the most used phantom during tractography validation, although it contains limitations. Here, we offer a new scoring system for this phantom, where segmentation of the bundles is now based on manually defined regions of interest rather than on bundle recognition. Bundles are now more reliably segmented, offering more representative metrics for future users. New code is available online. Scores of the initial 96 submissions to the challenge are updated. Overall, conclusions from the 2015 challenge are confirmed with the new scoring, but individual tractogram scores have changed, and the data is much improved at the bundle- and streamline-level. This work also led to the production of a ground truth tractogram with less broken or looping streamlines and of an example of processed data, all available on the Tractometer website. This enhanced scoring system and new data should continue helping researchers develop and evaluate the next generation of tractography techniques.