External Validation and Radiologist Comparison of a Deep Learning Model (DLM) to Identify Extranodal Extension (ENE) in Head and Neck Squamous Cell Carcinoma (HNSCC) with Pretreatment Computed Tomography (CT) Imaging

Abstract

For HNSCC patients, pretreatment identification of ENE would aid risk stratification and guide management. ENE identification using diagnostic imaging is currently inaccurate, with reported areas under the receiver operating characteristic curve (AUC) <.70 and accuracies <70%. We previously developed and internally validated a 3D convolutional neural network-based DLM trained on a HNSCC lymph node database derived from preoperative, diagnostic, contrast-enhanced CT scans from patients who underwent lymph node dissection at a single institution. An AUC of 0.91 was achieved on the internal test set (Institution 1). In this study, we sought to externally validate our model and directly compare its performance with diagnostic radiologists. We obtained de-identified, preoperative, contrast-enhanced HNSCC CT scans and corresponding pathology reports from a novel set of 82 HNC patients who underwent lymph node dissection at a different institution (Institution 2). From these patients, 130 lymph nodes were segmented and annotated as ENE(+) or ENE(-) with a high degree of certainty based on pathology reports. The previously developed DLM was used to predict ENE on the external dataset. Performance was evaluated with AUC, accuracy, sensitivity, and specificity. DLM performance was compared to the ENE predictions from two fellowship-trained, board-certified neuroradiologists (NR1 and NR2). Of 130 lymph nodes, there were 21 (16.2%) with ENE. The DLM achieved AUC of 0.84 (95%CI: 0.75-0.93) and accuracy 83.1% for ENE identification on the external validation set (Table). Neuroradiologists achieved AUCs of 0.70 (95%CI: 0.59-0.82; NR1) and 0.71 (95%CI: 0.60-0.82; NR2) and accuracies of 76.2% (NR1) and 73.8% (NR2) with a Cohen’s Kappa coefficient of inter-rater agreement 0.43. The DLM showed high discriminatory performance in identifying ENE on a blinded external dataset. The DLM performance was superior to two diagnostic neuroradiologists. Our findings suggest that a DLM can accurately detect ENE in HNSCC patients across different populations at the level of trained clinicians. Prospective testing is being planned to determine how to optimally integrate this model into the clinic.Abstract 146; Table 1Performance MetricExtranodal Extension (ENE)Internal Test Set (Institution 1, n = 98)External Validation Set (Institution 2, n = 130)DLMDLMNeuroradiologist 1Neuroradiologist 2AUC.91 (.86 - .96).84 (.75 - .93).70 (.59 -.82).71 (.60 - .82)Accuracy85.7%83.1%76.2%73.8%Sensitivity.88.71.62.67Specificity.85.85.79.75Youden Index.73.56.41.42 Open table in a new tab