Deep Learning Surpasses Gastrointestinal Experts at Classifying Endoscopic Severity in Patients with Ulcerative Colitis

Abstract

Background: Evaluation of endoscopic disease severity is a key component in the management of patients with ulcerative colitis (UC). However, endoscopic assessment suffers from substantial intra- and inter-observer variation of up to 75%, thereby limiting the reliability of individual assessments. Our aim was to develop a deep learning (DL) model capable of distinguishing active from healed mucosa, as well as to differentiate between different degrees of endoscopic disease activity. Methods: 1,484 unique endoscopic images from 467 patients were extracted for classification. Two experts classified all images independently of one another according to the Mayo endoscopic subscore (MES). In cases of disagreement, a third expert classified the images. Convolutional neural networks, including InceptionNetV3 and EfficientNetB0-B4, were considered in the development of our DL model. Five-fold cross-validation was used to develop and select the best model. Unseen test datasets were used to evaluate the models. The accuracy, sensitivity, specificity, positive and negative predictive values, and Cohen’s kappa were used to evaluate the final models. Findings: In the most difficult task – distinguishing between all four categories of MES – our final model achieved a mean accuracy of 0·82, a mean AUC of 0·99, test accuracy of 0·84, sensitivity of 0·88, specificity of 0·81 and a weighted Cohen’s kappa of 0·83 (p<0·001 compared to the experts). Interpretation: We propose a new, standardised way of evaluating endoscopic images from UC patients for both clinical and academic purposes. We demonstrate how our DL model is highly capable of distinguishing between all four MES levels of activity. This model will optimize and standardize the evaluation of disease severity measured by the MES across all centres and hospitals, no matter their level of medical expertise. Funding: None received. Declaration of Interest: B. Lo, Z. Liu, C. Igel has nothing to declare I. Vind has received either consulting or lecture fees from Tillotts Pharma AB, AbbVie A/S, Janssen-Cilag A/S, Takeda Pharma A/S. F Bendtsen have received consulting fees, lecture fess or research funds from Takeda Pharma A/S, Norgine Danmark A/S, Ferring Pharmaceuticals A/S J Burisch has received consulting fees from Celgene, Janssen-Cilag, AbbVie, Vifor Pharma, Jansen and Ferring; lecture fees from Abbvie, Pfizer, MSD, Pharmacosmos and Takeda Pharma, and unrestricted grant support from Takeda Pharma and Tillotts Pharma. Ethical Approval: This study was approved by the local hospital board as a quality insurance/improvement study.