Global feature-based multimodal semantic segmentation

Abstract

Incorporating complementary modality into RGB branch can significantly improve the effectiveness of semantic segmentation. However, fusion between the two modalities faces huge challenge due to the difference of their optical dimensions. Existed fusion methods can't keep a balance between performance and efficiency in aggregating detailed features. To address this problem, we propose a global feature-based network (GFBN) for semantic segmentation that establishes mapping function and extraction relationship among the multi-modalities. The GFBN contains three important modules, which are used for feature correction, fusion and edge enhancement. Firstly, the cross-attention rectification module (CARM) adaptively extracts mapping relationships and rectifies the RGB and complementary features. Secondly, the cross-field fusion module (CFM) integrates long-range rectified features of two branches to obtain an optimal fusion feature. Finally, the boundary guidance module (BGM) sharpens the boundary information of the fused features to effectively improve the segmentation accuracy of object boundaries. We make the experiments of GFBN on the challenging MCubeS and ZJU-RGB-Ps datasets. The results show that GFBN outperforms state-of-the-art methods by at least 0.64 % and 0.7 % on mean intersection over union (mIoU), respectively. It demonstrates the performance and efficiency of our proposed method. The code corresponding to our method can be found at the following link: https://github.com/Sci-Epiphany/GFBNext.