Link-RGBD: Cross-Guided Feature Fusion Network for RGBD Semantic Segmentation

Abstract

Significant advances have been made in the development of deep convolutional networks for RGBD semantic segmentation. However, the full extraction and utilization of depth information to assist semantic segmentation remain challenging. In this study, we propose the Link-RGBD module to fuse RGB and depth features using an innovative interactive attention mechanism to augment the representations of the objects of interest. The attention map gotten from RGB and depth branch is regarded as a weight for value (v) in the other branch in order to achieve cross-fusion. This enhances the coordinates and feature information and augments the representations of the objects of interest. We evaluated the effectiveness of the proposed method and achieved competitive results on the challenging SUN RGB-D and NYUDv2 semantic segmentation benchmarks. Its pixel accuracy is 0.2% higher than that of DeepLabv3+, reaching 83.1% on the SUN RGB-D benchmark. Moreover, its mean IoU score is 0.3% higher than that of DeepLabv3+, reaching 49.5% on the NYUDv2 benchmark. We further constructed a robot test system to evaluate the performance of the proposed method in a real scenario. The results show that the proposed method can support the completion of the moving forward and grasping task.