Abstract
Recent advancements in deep learning have led to a resurgence of medical imaging and Electronic Medical Record (EMR) models for a variety of applications, including clinical decision support, automated workflow triage, clinical prediction and more. However, very few models have been developed to integrate both clinical and imaging data, despite that in routine practice clinicians rely on EMR to provide context in medical imaging interpretation. In this study, we developed and compared different multimodal fusion model architectures that are capable of utilizing both pixel data from volumetric Computed Tomography Pulmonary Angiography scans and clinical patient data from the EMR to automatically classify Pulmonary Embolism (PE) cases. The best performing multimodality model is a late fusion model that achieves an AUROC of 0.947 [95% CI: 0.946–0.948] on the entire held-out test set, outperforming imaging-only and EMR-only single modality models.
Highlights
Pulmonary Embolism (PE) is a serious medical condition that hospitalizes 300,000 people in the United States every year[1]
With the approval of Stanford Institutional Review Board (IRB), we retrieved 108,991 studies from patients that had CTPA performed under the pulmonary embolism protocol between 2000 and 2016 at Stanford University Medical Center (SUMC)
We found that the fusion model achieved state-of-the-art Area under the receiver operating characteristic curve (AUROC) of 0.962 [0.961–0.963] for detecting clinically important central and segmental PE which was significantly better than either the pixel-based (0.833 [0.830–0.835]) or Electronic Medical Record (EMR)-based (0.921 [0.919–0.923]) models alone
Summary
Pulmonary Embolism (PE) is a serious medical condition that hospitalizes 300,000 people in the United States every year[1]. A survey showed that more than 85% of radiologists consider clinical context as vital for radiological exam interpretation[14] This holds true in the use case of pulmonary embolism diagnosis on CTPA where clinical context and prior imaging results are considered important for imaging decisions. The purpose of this study is to build and compare multimodal fusion models that combine information from both CT scans and Electronic Medical Record (EMR) to automatically diagnose the presence of PE Leveraging both clinical and imaging data by using a variety of fusion approaches could lead to a contextually relevant model which reduces PE misdiagnosis rate and delay in diagnosis, and inform future work by exploring optimal data selection and fusion strategies.
Sign-in/Register to view highlights & summary Sign-in/Register to access full text options 
Talk to us
Join us for a 30 min session where you can share your feedback and ask us any queries you have
Schedule a callDisclaimer: All third-party content on this website/platform is and will remain the property of their respective owners and is provided on "as is" basis without any warranties, express or implied. Use of third-party content does not indicate any affiliation, sponsorship with or endorsement by them. Any references to third-party content is to identify the corresponding services and shall be considered fair use under The CopyrightLaw.
