A RGB-Thermal based adaptive modality learning network for day–night wildfire identification

Abstract

Wildfires have long been a danger to the atmosphere and ecological environment. With the advancement of deep learning technology and sensor equipment, wildfire identification methods based on optical (RGB) or thermal infrared (TIR) images have made significant progress. However, single-modal identification technologies often experience serious performance degradation in challenging scenarios. Therefore, a multi-modal framework that combines both RGB and TIR is necessary for accurate wildfire identification. Despite this, current RGB-T based identification methods focus solely on developing fusion strategies to learn shared representations of the two modalities, while ignoring the effect of modality-specific features. To address this issue, we propose a novel wildfire identification framework that can adaptively learn modality-specific and shared features. Specifically, our model utilizes two parallel encoders to extract multi-scale RGB and TIR features, and integrates these modality-specific features into the fusion feature layer. Each modality-specific decoder and shared decoder corresponds to three independent label supervisions that oversee feature adaptive learning. To validate the effectiveness of our approach, we propose a new RGB-T paired wildfire semantic segmentation dataset that includes flames of different scales captured under varying lighting conditions from multiple UAV-based and ground-based cameras. Finally, we evaluate our proposed method alongside benchmarking single-modal methods and state-of-the-art multi-modal methods on two tasks: a common test setting under various lighting conditions, and a generalization test across three subsets. The experimental results demonstrate the superiority of our proposed method, with the average ＋6.41% (IoU) and ＋3.39% (F1-score) improvement over the second-best RGB-T semantic segmentation method.