Feature fusion and context interaction for RGB-D indoor semantic segmentation

Abstract

Depth information in indoor scenes is crucial to understanding the spatial and occluding relationships of objects. Depth information contains rich geometric details and is unaffected by lighting, texture, and color, significantly enhancing traditional color images. RGB images and depth images have been widely applied in various image analysis. However, effectively leveraging the complementary information from both modalities remains a challenge. To address this issue, we propose a unified and effective feature fusion and context interaction network (FCINet). It includes three modules. Firstly, we construct a multi-modal feature fusion module (MFFM), which can achieve the aggregation of two modalities along spatial and channel dimensions. Secondly, we construct a global and local information interaction context module (GLCM) to encode rich semantic information. Finally, we construct a feature alignment and fusion module (FAFM), which integrates upsampled low-level features with high-level features to mitigate spatial information loss. Experimental results indicate that the model achieved favorable outcomes on both the NYU Depth v2 dataset and the more complex SUN RGB-D dataset.