Machine learning algorithms performed no better than regression models for prognostication in traumatic brain injury

Ewout W Steyerberg,Hester F Lingsma,Ben Van Calster,Daan Nieboer,Ari Ercole,Benjamin Gravesteijn,David W Nelson

doi:10.1016/j.jclinepi.2020.03.005

Abstract

ObjectiveWe aimed to explore the added value of common machine learning (ML) algorithms for prediction of outcome for moderate and severe traumatic brain injury. Study Design and SettingWe performed logistic regression (LR), lasso regression, and ridge regression with key baseline predictors in the IMPACT-II database (15 studies, n = 11,022). ML algorithms included support vector machines, random forests, gradient boosting machines, and artificial neural networks and were trained using the same predictors. To assess generalizability of predictions, we performed internal, internal-external, and external validation on the recent CENTER-TBI study (patients with Glasgow Coma Scale <13, n = 1,554). Both calibration (calibration slope/intercept) and discrimination (area under the curve) was quantified. ResultsIn the IMPACT-II database, 3,332/11,022 (30%) died and 5,233(48%) had unfavorable outcome (Glasgow Outcome Scale less than 4). In the CENTER-TBI study, 348/1,554(29%) died and 651(54%) had unfavorable outcome. Discrimination and calibration varied widely between the studies and less so between the studied algorithms. The mean area under the curve was 0.82 for mortality and 0.77 for unfavorable outcomes in the CENTER-TBI study. ConclusionML algorithms may not outperform traditional regression approaches in a low-dimensional setting for outcome prediction after moderate or severe traumatic brain injury. Similar to regression-based prediction models, ML algorithms should be rigorously validated to ensure applicability to new populations.

Highlights

Traumatic brain injury (TBI) is a common disease, with a significant societal burden [1]: TBI is estimated to be responsible for around 300 hospital admissions and 12 deaths per 100,000 persons per year in Europe [2]
Prognostic models, which predict outcome for a patient, given a particular combination of baseline characteristics, are important: they may give us insight in mechanisms of disease that lead to poor outcome and allow for risk-based stratification of patients for logistic, research, and clinical reasons
Performance was assessed in terms of both discrimination and calibration, which are both important characteristics to be assessed in algorithm validation [22,24,39]

Summary

Introduction

Traumatic brain injury (TBI) is a common disease, with a significant societal burden [1]: TBI is estimated to be responsible for around 300 hospital admissions and 12 deaths per 100,000 persons per year in Europe [2]. Prognostic models, which predict outcome for a patient, given a particular combination of baseline characteristics, are important: they may give us insight in mechanisms of disease that lead to poor outcome and allow for risk-based stratification of patients for logistic, research, and clinical reasons. A large number of prediction models have been developed to predict outcome for patients with TBI, mostly using traditional regression techniques [4]. These models have not yet been widely implemented in clinical practice. More flexible machine learning (ML) algorithms have enjoyed enthusiasm as potentially promising techniques to improve outcome prognostication [5].

Results

Discussion

Conclusion