Hybrid Artificial Intelligence Outcome Prediction Using Feature Extraction from Stress Perfusion Cardiac Magnetic Resonance Images and Electronic Health Records

Abstract

Abstract Background Prediction of mortality and cardiovascular (CVS) events in coronary artery disease (CAD) has been conventionally achieved using clinical risk factors. Conventional qualitative visual assessment and quantitative analysis of stress perfusion cardiac magnetic resonance (SP-CMR) has previously been shown to have independent prognostic value (1), but the relationship between hidden or recognised imaging features and mortality outcome is still not well understood. Purpose To use deep learning with hybrid neural networks to combine image pixel data and clinical information for outcome prediction. Methods A retrospective study was performed on patients who had SP-CMR between 2011 and 2021. Ethical approval was obtained under the Research Ethics Committee in compliance with the Declaration of Helsinki. Clinical characteristics and CVS risk factors were extracted by using CogStack (2), a natural language processing (NLP) application to extract and analyse clinical data. SP-CMR images were extracted from patients who had mortality events, and from equally random sample without the events for class balance. Data were split into 70% for training, 15% for validation, and 15% for testing. A hybrid neural network (HNN), which combined convolutional neural network (CNN) and multilayer perceptron (MLP), was developed to predict the mortality outcome. The CNN and MLP were trained to extract image features and clinical data, respectively, followed by prediction from concatenated features. This prediction was compared with two models, one CNN model trained on images only and another linear model trained on clinical data including CMR reports findings. DeLong test was used to compare AUC and F1 scores. All deep learning techniques was performed using Python 3.7 and Tensorflow library. Results A total of 4,188 patients were identified, a subset of 500 patients were included in training the networks, including 220 patients with positive mortality events (44%). There were more males than females (73% vs 27%), approximately 36% of patients had history of myocardial infarction (MI), and there was a significant number of patients who had positive ischaemia on SP-CMR (40%), and/or evidence of ischaemic scar (48%). Mean left ventricular ejection fraction (LVEF) was 51 ± 16 %. Area under the curve (AUC) was found to be higher in HNN (80%) followed by clinical data prediction model (78%), and superior to image CNN (63%). There was no significant AUC difference between HNN and clinical model (p=0.072). However, the F1 score showed significant improvement in precision in HNN model (72%) followed by image CNN model (52%), overtaking clinical data prediction model (21%). There was significant F1 score difference between HNN and clinical model (p<0.001). Conclusion Death in patients with suspected or known CAD can be better predicted by a hybrid neural network that combines clinical and SP-CMR image data than by using clinical data alone.Full pipeline for hybrid neural networkAUC and F1 Score comparisons