OTHR-15. Assessment of TRIPOD adherence in articles developing machine learning models for differentiation of glioma from brain metastasis

Abstract

PurposeMachine learning (ML) applications in predictive models in neuro-oncology have become an increasingly investigated subject of research. For their incorporation into clinical practice, rigorous assessment is needed to reduce bias. Several reports have indicated utility of ML applications in differentiation of glioma from brain metastasis. However, a systematic assessment of quality of methodology and reporting in these studies has not been done yet. We examined the adherence of 29 published reports in this field to the TRIPOD statement, which is similar to CLAIM checklist.Materials and MethodsOur systematic review was conducted in accordance with PRISMA guidelines. Ovid Embase, Ovid MEDLINE, Cochrane trials (CENTRAL) and Web of science core-collection were searched. Keywords included artificial intelligence, machine learning, deep learning, radiomics, magnetic resonance imaging, glioma, and glioblastoma. Assessment of TRIPOD adherence in 29 eligible studies was performed. Individual item performance was assessed by adherence index (ADI), the ratio of mean achieved score to maximum score per TRIPOD item.ResultsIn a preliminary analysis of 8 studies, the average TRIPOD adherence score was 0.48 (14.25/30 items fulfilled) with individual scores ranging from 0.27 (8/30) to 0.60 (18/30). Best overall item performance, with an ADI of 1, was seen in item 3 (Background/Objectives), 16 (Model performance) and 19 (Interpretation). Poorest performance was detected in item 1 (Title) and 2 (), followed by item 9 (Missing Data) with ADI of 0, 0 and 0.13, respectively.ConclusionPreliminary results underline the lack of reproducibility in ML studies on distinction between glioma and brain metastasis. An average TRIPOD adherence score of 0.48 indicates insufficient quality of reporting and outlines the need for increased utilization of quality scoring systems in study documentation. Systematic evaluation of quality score adherence will allow us to identify common flaws in this field for enabling translation of models into clinical workflow.