Performance of Automated Machine Learning in Predicting Outcomes of Pneumatic Retinopexy

Abstract

ObjectiveAutomated Machine Learning (AutoML) has emerged as a novel tool for medical professionals lacking coding experience, enabling them to develop predictive models for treatment outcomes. This study evaluated the performance of AutoML tools in developing models predicting the success of pneumatic retinopexy (PR) in treatment of rhegmatogenous retinal detachment (RRD). These models were then compared with custom models created by ML experts. DesignRetrospective multicenter study. Population539 consecutive patients with primary RRD that underwent PR by a vitreoretinal fellow at six training hospitals between 2002 and 2022. MethodsWe used two AutoML platforms, MATLAB Classification Learner and Google Cloud AutoML. Additional models were developed by computer scientists. We included patient demographics and baseline characteristics, including lens and macula status, RRD size, number and location of breaks, presence of vitreous hemorrhage and lattice degeneration, and physicians’ experience. The dataset was split into a training (n = 483) and test set (n = 56). The training set, with a 2:1 success-to-failure ratio, was used to train the MATLAB models. Since Google Cloud AutoML requires a minimum of 1000 samples, the training set was tripled to create a new set with 1449 datapoints. Additionally, balanced datasets with a 1:1 success-to-failure ratio were created using Python. Main Outcome MeasuresSingle-procedure anatomic success rate, as predicted by the ML models. F2-scores and area under the receiver operating curve (AUROC) were used as primary metrics to compare models. ResultsThe best performing AutoML model (F2-score=0.85, AUROC=0.90; MATLAB), showed similar performance to the custom model (0.92, 0.86) when trained on the balanced datasets. However, training the AutoML model with imbalanced data yielded misleadingly high AUROC (0.81) despite low F2-score (0.2) and sensitivity (0.17). ConclusionsWe demonstrated the feasibility of using AutoML as an accessible tool for medical professionals to develop models from real-world data. Such models can ultimately aid in the clinical decision-making, contributing to better patient outcomes. However, outcomes can be misleading or unreliable if used naively. Limitations exist, particularly if datasets contain missing variables or are highly imbalanced. Proper model selection and data preprocessing can improve the reliability of AutoML tools.