Hierarchical and Global Modality Interaction for Brain Tumor Segmentation

Abstract

AbstractMulti-modality brain tumor segmentation is vital for the treatment of gliomas, which aims to predict the regions of the necrosis, edema and tumor core on multi-modality magnetic resonance images (MRIs). However, it is a challenging task due to the complex appearance and diversity shapes of tumors. Considering that multi modality of MRIs contain rich biological properties of the tumors, we propose a novel multi-modality tumor segmentation network for segmenting the brain tumor based on fusing the complementary information and global semantic dependency information upon the multi-modality imaging data. Specifically, we propose a hierarchical modality interaction block to build the internal relationship between complementary modality pair, and then enhance the complementary information between the them by using the channel and spatial co-attention. To capture the long-dependency relationship of cross-modality information, we propose a global modality interaction transformer block to build the global semantic interaction between the multi-modality local features. The global modality interaction Transformer block makes up for CNN’s poor perception of global semantic dependency information across modes. We evaluate our method on the validation set of multi-modality brain tumor segmentation challenge 2021 (BraTs2021). The proposed multi-modality brain tumor segmentation network achieves 0.8518, 0.8808 and 0.926 Dice score for the ET, CT and WT.KeywordsBrain tumor segmentationTransformerCross-modality information