Comparison of Diagnostic Performance in Thyroid Nodules on US: Deep Convolutional Neural Network Models vs Endocrinologists With Various Experiences

Abstract

Abstract Objectives: To diagnose thyroid cancer, ultrasonography is a primary tool, but diagnostic accuracy varies according to the proficiency of clinicians. The aim of this study was to compare diagnostic performance between deep convolutional neural network (CNN) models and endocrinologist with various experiences. Methods: Patients who underwent fine needle aspiration at endocrinology department in Seoul National University Hospital, between April 2014 and June 2019, were reviewed. Among them, thyroid nodules which were pathologically confirmed by surgery and maximal diameter greater than 1cm were included. Ultrasonography images of thyroid nodules were reviewed by 13 endocrinologists with various experiences: 0 month (E0, n=8), 1 year (E1, n=2), and >5 years (E5, n=3). Results: Of total 451 thyroid nodules, 66.5% was diagnosed as cancer and 83.7% was papillary thyroid cancer (PTC). Sensitivity and specificity of CNN were 85.3% and 63.6%, respectively, and its area under the curve (AUC) was 0.855. Compared to CNN, mean accuracy of E0 group was significantly lower (Accuracy 68.7% vs 78.0%, P <0.001), and after CNN-assistance, that of E0 was significantly improved (68.7% [before] vs 73.93% [after], P = 0.008). E1 and E5 groups showed similar diagnostic performance to CNN, and CNN-assistance did not change it. Next, subgroup analysis was performed according to the histologic subtypes. AUC of CNN in PTC (0.925) was much higher than that of other cancers including FTC (0.529). Interestingly, CNN-assistance significantly improved diagnostic performance for PTC not only in beginners (E0), but also a subset of experienced endocrinologist (E1 and E5). Conclusions: CNN has good diagnostic performance in the diagnosis of PTC. Endocrinologist with lower experience in ultrasonography, CNN-assistance is beneficial for improving diagnostic performance especially in PTC.