Crowdsourcing Deep Learning Algorithms to Automatically Contour GI Luminal Organs on Serial MRIs

Abstract

A major barrier to a sustainable real-time adaptive MR-guided radiotherapy workflow is the time-consuming process of contouring the target and organs-at-risk (OARs) before the delivery of each fraction. While auto-contouring algorithms perform relatively well for many solid organs, the performance on luminal organs in the abdomen remain difficult due to the variability between patients and variability in daily shape and position. The purpose of this study is to evaluate the performance of crowdsourced deep learning algorithms to automatically contour GI luminal organs on serial MRIs. The stomach, small intestines, and large intestines were manually contoured on MRIs from patients who had undergone radiotherapy on an MR-Linac by a team of radiation therapists and medical physicists and were verified by a board-certified radiation oncologist. The MRIs and the contours were de-identified and uploaded to Kaggle, an online machine learning competition platform with portion of the data open to the public as training data and the remaining data hidden as a test set. Prize money was offered to teams submitting the best auto-contouring algorithms based on the Dice coefficient and Hausdorff distance evaluation metrics. The average performance of the winning algorithm and of manual contours were compared using unpaired t-test. Four hundred sixty-seven MRIs were collected from 107 patients who underwent 1-5 serial MRI sessions between 2015 and 2019. The most common anatomic site of treatment was the pancreas with 41 patients, followed by the liver with 38 patients. The manual contours of the stomach, small intestines, and large intestines on 4 representative MRIs had mean and standard deviation Dice coefficient of 0.90 +/- 0.02, 0.76 +/- 0.04, and 0.85 +/- 0.03 respectively and Hausdorff distance of 18.0 +/- 6.9, 35.5 +/- 12.6, and 32.3 +/- 12.3 mm respectively. The Kaggle competition was held from April to July 2022 and 1548 teams submitted algorithms for evaluation. The auto-contouring performance of the winning solution on the stomach, small intestines, and large intestines, when evaluated on a hold-out test set with 188 MRIs, had mean and standard deviation Dice coefficient of 0.92 +/- 0.04, 0.80 +/- 0.09, 0.85 +/- 0.08 respectively and Hausdorff distance of 15.2 +/- 11.2, 33.9 +/- 15.2, 34.8 +/- 20.3 mm respectively. Unpaired t-test was performed to compare the average performance across three organs of the human (N = 120) and of the algorithm (N = 564). The results suggest that average algorithm performance was statistically superior to manual contours for Dice (p = 0.01), yet not for Hausdorff (p = 0.64). Crowdsourced deep learning algorithms to auto-contour GI luminal organs on serial MRIs perform superiorly compared to manual contours when using a Dice coefficient metric but not when using a Haudorff distance metric. These auto-contouring algorithms may be used to efficiently adapt radiotherapy plans according to the anatomy of the day for patients with abdominal tumors on MR-Linacs.