We are One: An Investigation of Bias in Deep Learning Autocontouring

Abstract

<h3>Purpose/Objective(s)</h3> Deep Learning based Autocontouring (DLC) has demonstrated consistent and accurate organ delineation, leading to time-savings in clinical contouring. Data used in development and testing of DLC algorithms have been largely sourced from single geographic populations, which leads to a potential risk of population-based bias being learnt by the algorithms. The goal of this study is to evaluate this risk by determining the impact ethnicity and or differing geographic populations have on performance of an autocontouring system. <h3>Materials/Methods</h3> 40 Head Neck CT scans were collected from four clinics, located in Europe (n=2) and Asia (n=2). A single observer manually delineated 15 organs-at-risk (OAR) in every dataset. A DLC solution then processed all CTs to produce 15 OARs. The autocontoured OARs were compared to the manual delineations using 3D Dice Similarity Coefficient (3D DSC), Added Path Length (APL) and 2D 95% Hausdorff Distance measures (2D 95% HD). The null hypothesis was that quantitative performance on all populations is the same. A Kruskall-Wallis test (KWT) was performed per similarity measure and per organ was used to test this hypothesis for the four sites. <h3>Results</h3> The KWT test found no statistically significant difference for the majority of structures across all similarity metrics. With a Bonferroni Correction applied to reduce this risk of Type I error with multiple tests, all structures showed no statistically significant difference across the populations except the Pharyngeal Constrictor Muscle. <h3>Conclusion</h3> No statistically significant differences in performance of deep learning contouring across populations were found for most structures. From this study, there appears to be no bias across populations, demonstrating its relevance internationally. This investigation of potential geographic or ethnic bias should extend to more patients, populations and anatomical regions in future.