HAFFseg: RGB-Thermal semantic segmentation network with hybrid adaptive feature fusion strategy

Abstract

RGB-Thermal (RGB-T) semantic segmentation provides the pixel-level prediction of surrounding environments for autonomous vehicles and mobile robots in harsh conditions such as insufficient illumination and severe weather. Existing RGB-Thermal semantic segmentation networks with two modalities feature extraction branches and insufficient feature fusion strategy limit their segmentation performance. To this end, this study proposed a novel RGB-Thermal semantic segmentation network with hybrid adaptive feature fusion strategy (HAFFseg). The proposed HAFFseg contains three feature extraction branches to achieve outstanding feature extraction capacity. The frequency domain feature enhanced module is invented and inserted in the first feature extraction stage to enhance the low-level extracted features. A hybrid adaptive feature fusion strategy which consists of hybrid feature fusion architecture and feature enhanced adaptive fusion modules (FEAFM) to obtain excellent feature fusion performance is designed for the proposed network. The full-scale fusion connection decoder is developed to perform multi-stage feature integration and further boost segmentation accuracy. Extensive ablation studies and comparison experiments have been conducted on MF dataset and RoadScene-seg dataset. Experimental results proved the effectiveness of the proposed HAFFseg structure and demonstrated the superiority of HAFFseg over other state-of-the-art RGB-T semantic segmentation networks.