S26 Comparison of Machine and Deep Learning Methods With the Mayo Clinical Risk Score in Predicting 1-Year Mortality in Primary Biliary Cholangitis Patients

Abstract

Introduction: To compare machine learning (ML) and deep learning (DL) approaches to predict survival outcome in patients at 1- year with Primary Biliary Cholangitis (PBC) and to compare with the Mayo Clinical risk score Methods: The data is taken from a randomized trial in PBC patients conducted between 1974 and 1984 by the Mayo Clinic. The first 312 cases in the dataset were included out of a total of 424 PBC patients referred. The rest that contained subjects with >30% missing data, were excluded. We evaluated the following models: 1) The 5-covariate Mayo model: log(bilirubin), albumin, log(prothrombin time), edema, and age, 2) 17-covariate Cox proportional hazards model by componentwise likelihood based boosting with stepnumber 10 and penalty number 100, 3) 17-covariate Deephit: DeepHit: trains a neural network to learn the estimated joint distribution of survival time and event, while capturing the right-censored nature inherent in survival data. Analysis was done with frac 0.3, relu activation, 0.1 dropout, 100L epochs and a batch size of 32L, 4) 17-covariate Multitask logistic regression model with ranking based feature selection to predict survival using a logistic regression model and the parameters from each model are estimated simultaneously in the maximization of the joint likelihood function, 5) random survival forest with 1000 trees and 6) Support vector machine. For the ML & DL models all 17-covariates were include in the analysis. Analyses were done in RStudio, and missing values were imputed using missRanger package.1 The data is split into 80 percent training and 20 precent validation. All models were 5-fold cross validated. Prediciton statistics was calculated for each model developed. Finally, the ML & DL models were compared with the 5-covariate model of the Mayo Clinic Risk score. Results: Multitask logistic regression model showed the highest AUC (1.00) with a Harrell’s C-statistic of 0.90. Cox boost and random survival forest had equal AUCs (0.87) and C-indices (0.84). But the 5-covariate model of Mayo clinic Risk score had a higher AUC when compared to cox boost, random survival forest, support vector machine or even Deep hit. Conclusion: Machine learning methods offered limited improvement over the Mayo Clinical risk score except for the Multitask logistic regression model in predicting PBC survival outcome at 1-year (Table). Table 1. - Performance of Risk Scores from the 5-Covariate Model in comparison to the machine learning and deep learning models at 1-year S.No. Model specification AUC 95% CI Harrell’s C-statistic 1. 5-covariate model of Mayo clinic Risk score 0.88 0.85-0.91 0.79 2. Cox boost 0.87 0.84-0.90 0.84 3. Deep hit 0.70 0.66-0.74 0.30 4. Multitask logistic regression 1.00 1-1 0.90 5. Random survival forest 0.87 0.84-0.90 0.84 6. Support Vector Machine 0.56 0.52-0.61 0.43