CAFseg: A Semantic segmentation network with cross aggregation fusion strategy for RGB-thermal semantic segmentation

Abstract

Semantic segmentation utilises the RGB-Thermal (RGB-T) source images with the capacity of provide pixel-level prediction for surrounding scenes in harsh imaging conditions. However, existing feature fusion strategies for RGB-T semantic segmentation networks generally fail to obtain the desirable fused features, and segmentation edge errors tend to occur in the segmentation results due to the difference between the features of multi-modalities. To this end, a novel semantic segmentation network with cross aggregation fusion strategy named CAFseg is proposed for RGB-T semantic segmentation in this study. Three feature extraction branches are included in the proposed CAFseg to supply sufficient source modality features. The light-weighted feature extraction backbones which integrated residual blocks and MobileViT blocks are developed for each feature extraction branch to accomplish optimal balance between feature extraction capacity and computational efficiency. The cross aggregation fusion strategy which consists of cross aggregation fusion architecture and adaptive aggregation feature fusion modules is innovatively designed for the proposed network to obtain fused features with abundant complementary information. Moreover, three independent triangle-shaped decoders are implemented for their correspondence feature extraction branch to make them achieve outstanding segmentation results separately, and the final segmentation results are obtained by fusing the segmentation results from each decoder, hence, further correcting the segmentation edge errors. Extensive ablation studies and comparative experiments have been conducted on two publicly available RGB-T semantic segmentation datasets. Experimental results validated the effectiveness of each component of the proposed CAFseg and demonstrated the superior performance of CAFseg over other state-of-the-art RGB-T semantic segmentation networks for RGB-T semantic segmentation.