Substance Misuse Algorithm for Referral to Treatment Using Artificial Intelligence (SMART-AI): Multi-Modal Validation with Interpretation and Bias Assessment

Abstract

Background: Substance misuse is a heterogeneous and complex set of behavioral conditions that are highly prevalent in hospital settings and frequently co-occur. Few solutions exist to comprehensively and reliably identify these conditions hospital-wide to prioritize care and guide treatment. The aim is apply natural language processing (NLP) to admission notes in the electronic health record (EHR) to accurately screen for substance misuse. Methods: The reference dataset was derived from a hospital-wide program that used structured diagnostic interviews to manually screen admitted patients over 26 months (n=54,915). Temporal validation was provided over the subsequent 12 months (n=16,917) and external validation at a separate health system (n=1,991). The Alcohol Use Disorder Identification Test and Drug Abuse Screening Tool served as reference standards. The first 24 hours of notes in the EHR were mapped to standardized medical vocabulary and fed into neural network models. The primary outcome was discrimination for alcohol misuse, opioid misuse, or non-opioid drug misuse. Discrimination was assessed by the area under the receiver operating curve (AUROC). Findings: The model was trained on a cohort that had 3.5% (n=1,921) with any type of substance misuse. Nearly 11% of patients with substance misuse had more than one type of misuse. The multi-label convolutional neural network classifier had an average AUROC of 0.97 (95% CI: 0.96, 0.98) during temporal validation for all types of substance misuse. The model was well calibrated and demonstrated good face validity with model features containing explicit mentions of aberrant drug-taking behavior. The false-negative and false-positive rates were similar between non-Hispanic Black and non-Hispanic White groups. In external validation, the AUROC for alcohol and opioid misuse remained above 0.85. Interpretation: We developed a novel and accurate approach to leveraging the first 24 hours of EHR notes for screening multiple types of substance misuse. Funding Information: Research reported in this publication was supported by the National Institute On Drug Abuse of the National Institutes of Health under Award Numbers R01-DA051464 (MA), K23-AA024503 (MA), UL1-TR002389 (NK), KL2-TR002387 (NK), R01-DA041071 (NK), UG1-DA049467 (NK), R01-LM010090 (DD), R01-LM012973 (DD), K12-HS-026385 (HT), and R01-GM123193 (MMC). Declaration of Interests: Dr. Churpek has a patent pending (ARCD. P0535US.P2) for risk stratification algorithms for hospitalized patients, and has received research support from EarlySense (Tel Aviv, Israel). All other authors have nothing to declare. Ethics Approval Statement: This study was approved by the Institutional Review Board at RUMC and LUMC.