Development and Validation of a Natural Language Processing Model to Identify Low-Risk Pulmonary Embolism in Real Time to Facilitate Safe Outpatient Management

Abstract

ObjectiveThis study aimed to (1) develop and validate a natural language processing (NLP) model to identify presence of PE based on real-time radiology reports and (2) identify low-risk PE patients based on previously validated risk stratification scores using variables extracted from the electronic health record (EHR) at the time of diagnosis. The combination of these approaches yielded an NLP-based clinical decision support tool that can identify patients presenting to the emergency department (ED) with low-risk PE as candidates for outpatient management. Research QuestionCan an NLP-based clinical decision support tool that uses real-time radiology reports and EHR data accurately identify patients presenting to the ED with low-risk PE? Study Designs and MethodsData were curated from all patients who received a PE-protocol computed tomography pulmonary angiogram (CTPA) imaging study in the ED of a three-hospital academic health system between June 1, 2018 and December 31, 2020 (n = 12,183). The “preliminary” radiology reports from these imaging studies made available to ED providers at the time of diagnosis were adjudicated as positive or negative for PE by the clinical team. The reports were then divided into development, internal validation, and temporal validation cohorts in order to train, test, and validate an NLP model that could identify presence of PE based on unstructured text. For risk stratification, patient and encounter-level data elements were curated from the EHR and used to compute a real-time simplified pulmonary embolism severity (sPESI) score at the time of diagnosis. Chart abstraction was performed on all low-risk PE patients admitted for inpatient management. ResultsWhen applied to the internal validation and temporal validation cohorts, the NLP model identified presence of PE from radiology reports with an AUROC of 0.99, sensitivity 0.86-0.87, and specificity 0.99. Across cohorts, 10.5% of PE-CTPA studies were positive for PE, of which 22.2% were classified as low-risk by sPESI score. Of all low-risk PE patients, 74.3% were admitted for inpatient management. InterpretationThis study demonstrates that an NLP-based model utilizing real-time radiology reports can accurately identify patients with PE. Further, this model, used in combination with a validated risk stratification score (sPESI), provides a clinical decision support tool that accurately identifies patients in the ED with low-risk PE as candidates for outpatient management.