Multi-modal fusion network with channel information guided module for prognosis prediction in patients with anti-N-methyl-D-aspartate receptor encephalitis

Abstract

As the most common type of autoimmune encephalitis, the pathological mechanism of anti-N-methyl-D-aspartate receptor (anti-NMDAR) encephalitis has been gradually clarified, but the optimal treatment has not yet been clarified. Accurate early prognostic assessment is of great significance for reversing the symptoms of anti-NMDAR encephalitis. Compared with the expensive costs of human evaluation, the prognostic evaluation method based on the deep learning diagnosis model has obvious advantages. In addition, full consideration of multi-modal information, such as multi-sequence magnetic resonance imaging (MRI) features and clinical variables, has a positive impact on the accurate prediction of patient prognosis. In this paper, a multimodal fusion network is proposed to predict the prognosis of patients with anti-NMDAR encephalitis. The proposed network contains three key substructures. First of all, the channel information guided module is designed to constrain the heterogeneity between the features of different modes through the attention mechanism to achieve more effective feature fusion. Second, we design a backbone network for feature extraction of brain MRI, which is based on a dual-branch residual structure and utilizes the channel information guided module to ensure that the information of features on different scales is complementary. Finally, a feature fusion network is proposed, which uses channel information guided module and dynamic normalization weighting to control the fusion of clinical variables and MRI features. Different from the existing multi-modal methods, our method avoids the huge model structure and uses an end-to-end structure to predict the prognosis of patients with multi-modal features. The above method was applied to the dataset proposed by a dual-center study on anti-NMDAR encephalitis in Southwest China, achieving excellent performance in terms of AUC (0.9799) and accuracy (0.9512). We further validate the model on an independent external validation dataset, and the results show that the model has good generalization.