Machine Learning as an Adjunct to Clinical Decision Making in Alcohol Dependence Treatment

Abstract

This thesis investigated the potential for a machine learning (ML) model, instantiated as a Decision Support System (DSS), to assist psychologists when making prospective predictions for alcohol dependence treatment outcomes. Predictions were made for patients undertaking a 12 week, abstinence-based, Cognitive Behavioural Therapy (CBT) program assisted by voluntary adjunctive medication (acamprosate and/or naltrexone). Success was defined as attending all treatment sessions while maintaining abstinence. A series of studies examined: the ability of clinical staff (N▄ 10) to predict treatment outcome on the basis of clinical data alone (Study 1, 50 consecutively treated patients); ML models to predict outcome for the same 50 patients, trained on 780 previously treated patients (Study 2); prospective intuitive psychologist predictions for 220 consecutive patients (Study 3); prospective ML model predictions for the same 220 patients, trained on 1016 previously treated patients (Study 4); and the clinically integrated application and efficacy of a DSS based on a naive Bayesian model to assist psychologists when predicting patient outcome (Study 5). Initially, the mean aggregate accuracy of clinicians when predicting with patient data alone (Study 1, 56.10%), was not significantly (pg.05) different to expected by chance as was the mean aggregate accuracy (Study 2, 58.57%) for ML models on the same 50 patients. However, two clinicians and six ML models were significantly (pl.05) more accurate than expected by chance alone. The maximum accuracy achieved by a psychologist was 66%, and was 78% for a ML model. When prospective predictions were made for 220 patients the mean accuracy of psychologists (Study 3, 56.36%) was not significantly different (pg.05) to chance alone whereas the mean accuracy of the ML models (Study 4, 63.95%) was significantly different (pl.05) for all but the two least accurate models. The 10.59% higher accuracy of the ML models meant that there was a significant (pl.05) difference in accuracy. The best ML model achieved an accuracy of 70.91%. This suggested that ML models were accurate enough to assist psychologists when making predictions to be suitable for a DSS. Psychologist probability estimates for the percentage chance of a successful patient outcome were significantly correlated with outcomes when making intuitive predictions although the correlation (rpb ▄ .163) was low. The findings for the ML models were also encouraging with a significant relationship between probability estimates and outcome at a weak to moderate strength (rpb ▄ .133-.251). Furthermore there was little evidence for overconfidence in psychologistsr predictions as evinced in previously published studies. These results taken together suggest that psychologists could potentially decide when to accept DSS advice in a principled fashion and would be less likely to reject the advice due to high confidence levels. A naive Bayesian model was integrated as a DSS into the normal clinical practice workflow using an intuitive and easy to use graphical interface developed by the author (Study 5). Psychologists initially made an intuitive prediction for their patients after the first session of treatment and had the option to request a DSS prediction. Predictions were requested for 57 of the 106 patients treated during this study. After viewing the DSS prediction psychologists were offered a chance to review their initial choice. The initial accuracy of the DSS (49.12%) was hindered by clinical challenges that arose out of the lin vivor evaluation in the context of a busy public hospital. When tested using an lidealr cleaned data-set post study the potential accuracy of the DSS was 59.65%. However, it performed statically no worse than psychologists (64.91%). A combined voting system, choosing the prediction with the highest estimated probability, would have been the most accurate (66.67%). However, psychologists did not alter any predictions in the final study after viewing the DSS prognosis. Given that the DSS fulfilled the requirements found for successful implementation in medical settings the unique requirements of psychological therapy must be further examined before the successful future deployment of a DSS into a behavioural treatment environment. The variables identified as most important for prediction were significantly different for psychologists and ML feature selection approaches. Furthermore, categories of variables previously unmeasured at the clinical site were identified as important by psychologists including social support, commitment/motivation and short-term drinking history before treatment. Capturing these variables could potentially improve ML accuracy. This thesis demonstrated proof of concept and provided early efficacy data for improving prediction of treatment outcome for alcohol dependence, using novel ML approaches combined with a DSS.