Predictive structured\u2013unstructured interactions in EHR models: A case study of suicide prediction

Ilkin Bayramli,Ben Y Reis,Yuval Barak-Corren,Matthew K Nock,Victor Castro,Jordan W Smoller,Emily M Madsen

doi:10.1038/s41746-022-00558-0

Abstract

Clinical risk prediction models powered by electronic health records (EHRs) are becoming increasingly widespread in clinical practice. With suicide-related mortality rates rising in recent years, it is becoming increasingly urgent to understand, predict, and prevent suicidal behavior. Here, we compare the predictive value of structured and unstructured EHR data for predicting suicide risk. We find that Naive Bayes Classifier (NBC) and Random Forest (RF) models trained on structured EHR data perform better than those based on unstructured EHR data. An NBC model trained on both structured and unstructured data yields similar performance (AUC = 0.743) to an NBC model trained on structured data alone (0.742, p = 0.668), while an RF model trained on both data types yields significantly better results (AUC = 0.903) than an RF model trained on structured data alone (0.887, p < 0.001), likely due to the RF model’s ability to capture interactions between the two data types. To investigate these interactions, we propose and implement a general framework for identifying specific structured-unstructured feature pairs whose interactions differ between case and non-case cohorts, and thus have the potential to improve predictive performance and increase understanding of clinical risk. We find that such feature pairs tend to capture heterogeneous pairs of general concepts, rather than homogeneous pairs of specific concepts. These findings and this framework can be used to improve current and future EHR-based clinical modeling efforts.

Highlights

In recent years there has been a proliferation of clinical prediction models powered by electronic health records (EHRs)
We found that models trained only on features derived from structured-data perform better than models trained only on features derived from unstructured data
Combining unstructured data with structured data provided almost no performance benefit with the Naive Bayes Classifier (NBC) model, whereas the Balanced Random Forest Classifier (BRFC) model showed a significant increase in area under the receiver operating curve (AUC)

Summary

Introduction

In recent years there has been a proliferation of clinical prediction models powered by electronic health records (EHRs). Many prediction models rely primarily on structured data from the EHR, which typically includes diagnostic, laboratory, medication, and procedure codes. Unstructured EHR data have been used for clinical predictive tasks, both as a standalone feature-set and in combination with structured data. It is important to understand the interactions between these two data types and identify instances where the nature of these interactions differs between case and non-case populations. Such differences can be valuable for deepening our understanding of clinical risk and for improving clinical risk prediction in models that are able to capture these interactions

Objectives

Methods

Findings

Conclusion