Dual-Space Graph-Based Interaction Network for RGB-Thermal Semantic Segmentation in Electric Power Scene

Abstract

Real-time scene comprehension is the basis for automatic electric power inspection. However, existing RGB-based scene comprehension methods may achieve unsatisfied performance when dealing with complex scenarios, insufficient illumination or occluded appearances. To solve this problem, by cooperating visual and thermal images, the Dual-Space Graph-based Interaction Network (DSGBINet) is proposed to achieve all-day time semantic segmentation of power equipment in high-voltage power transmission line and electric transformer substation scenes. Specifically, modality-specific features are first extracted via two separate backbone networks with the same architecture. Multi-modality high-level features are first fused via long-range relationship in coordinate space. Then, multi-modality features from regular grids are further clustered and assigned to vertices in feature space. Cross-graph and inner-graph regional relations are utilized for reasoning and enhancement, which could exploit the mutual benefits and extract rich contextual information in a semantic view. Furthermore, to overcome the huge scale difference and the inherent characteristic of thermal images, the idea of multi-task learning is integrated into the decoding process. The edge detection and semantic segmentation are achieved collaboratively, which could segment the different power equipment more accurately and completely. The comparative and ablation experiments on the proposed two RGB-T semantic segmentation datasets evaluate the effectiveness and robustness of the proposed network compared with existing state-of-the-art methods. The extended experiments on the public datasets further demonstrate the superiority of the proposed method. Our dataset and code will be released at: <uri xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">https://github.com/hhujiang/DSGBINet</uri> .