412 Outcome Forecasting in Aneurysmal Subarachnoid Hemorrhage: Comparison of a Neural Network Model With Conventional Logistic Regression

Abstract

INTRODUCTION: Interest in machine-learning-(ML)-based predictive modeling has led to the development of models predicting aneurysmal subarachnoid hemorrhage (aSAH) outcome, including the Nijmegen acute Subarachnoid Hemorrhage calculator (Nutshell). Generalizability of such models to external data and their performance in comparison to more established conventional logistic-regression-based models, such as the Subarachnoid-Hemorrhage-International-Trialists-(SAHIT) calculator tool, remains unclear. METHODS: A prospectively-maintained database of aSAH patients presenting consecutively to our institution in the 2013-2018 period was used. The web-based Nutshell (https://nutshell-tool.com/) and SAHIT (http://sahitscore.com) calculators were used to derive the risks of poor long-term (12-18 month) outcome and 30-day mortality. Discrimination and calibration were evaluated using area under the curve (AUC) and calibration plots. Literature on relevant ML models was surveyed for a synopsis. RESULTS: The SAHIT models outperformed the Nutshell tool in predicting long-term functional outcome (AUC: 0.786 vs. 0.689, p = 0.025) and 30-day mortality (AUC: 0.810 vs. 0.636, p < 0.001). Calibration properties were additionally more favorable for the SAHIT models. Most published aneurysm-related ML-based outcome models lack external validation and usable testing platforms. In total, 24 studies utilizing ML models were identified in our literature review, evaluating various aSAH-related and aneurysm treatment outcomes. Only two studies pursued external validation, and only five provided a testing platform for their model to be used by external researchers (e.g. scoring system, calculator, decision tree etc.). CONCLUSION: The Nutshell tool demonstrated limited performance upon external validation in comparison to the SAHIT models. External validation and the dissemination of testing platforms for ML models must be emphasized.