PurposeTo develop and validate machine learning (ML) models to predict choroidal nevus transformation to melanoma based on multimodal imaging at initial presentation. DesignRetrospective multicenter study. ParticipantsPatients diagnosed with choroidal nevus on the Ocular Oncology Service at Wills Eye Hospital (2007-2017) or Mayo Clinic Rochester (2015-2023). MethodsMultimodal imaging was obtained, including fundus photography, fundus autofluorescence, spectral domain OCT, and B-scan ultrasonography. ML models were created (XGBoost, LGBM, Random Forest, Extra Tree) and optimized for area under receiver operating curve (AUROC). The Wills Eye Hospital cohort was utilized for training and testing (80% training-20% testing) with 5-fold cross validation. The Mayo Clinic cohort provided external validation. Model performance was characterized by AUROC and area under the precision recall curve (AUPRC). Models were interrogated using SHapley Additive exPlanations (SHAP) to identify the features most predictive of conversion from nevus to melanoma. Differences in AUROC and AUPRC between models were tested using 10,000 bootstrap samples with replacement and results. Main Outcome MeasuresAUROC and AUPRC for each ML model. ResultsThere were 2,870 nevi included in the study, with conversion to melanoma confirmed in 128 cases. Simple AI Nevus Transformation System (SAINTS; XGBoost) was the top performing model in the test cohort [pooled AUROC 0.864 (95% confidence interval (CI): 0.864-0.865), pooled AUPRC 0.244 (0.243-0.246)] and in the external validation cohort [pooled AUROC 0.931 (95% CI: 0.930-0.931), pooled AUPRC 0.533 (0.531-0.535)]. Other models also had good discriminative performance: LGBM (test set pooled AUROC 0.831, validation set pooled AUROC 0.815), Random Forest (test set pooled AUROC 0.812, validation set pooled AUROC 0.866), and Extra Tree (test set pooled AUROC 0.826, validation set pooled AUROC 0.915). A model including only nevi with at least 5 years of follow up demonstrated the best performance in AUPRC (test: pooled 0.592 (95% CI: 0.590-0.594); validation: pooled 0.656 (95% CI: 0.655-0.657)). The top five features in SAINTS by SHAP values were: tumor thickness, largest tumor basal diameter, tumor shape, distance to optic nerve, and subretinal fluid extent. ConclusionsWe demonstrate accuracy and generalizability of a ML model for predicting choroidal nevus transformation to melanoma based on multimodal imaging.
Read full abstract