On the use of GNN-based structural information to improve CNN-based semantic image segmentation

Abstract

Convolutional neural networks (CNNs) are widely used for semantic image segmentation across various fields (medicine, robotics), capturing local pixel dependencies for good results. Nevertheless, CNNs struggle to grasp global contextual representations, sometimes leading to structural inconsistencies. Recent approaches aim to broaden their scope using attention mechanisms or deep models, resulting in heavy-weight architectures. To boost CNN performance in semantic segmentation, we propose using a graph neural network (GNN) as a post-processing step. The GNN conducts node classification on appropriately coarsened graphs encoding class probabilities and structural information related to regions segmented by the CNN. The proposal, applicable to any CNN producing a segmentation map, is evaluated on several CNN architectures, using two public datasets (FASSEG and IBSR), with four graph convolution operators. Results reveal performance improvements, enhancing on average the Hausdorff distance by 24.3% on FASSEG and by 74.0% on IBSR. Furthermore, our approach demonstrates resilience to small training datasets.