Joining geometric and RGB features for RGB-D semantic segmentation

Abstract

Depth map is a regular format of geometric data structure. Some approaches attempted to harness point cloud from depth channel to extract 3D features and demonstrated the superiority over traditional 2.5D representation approaches. However, how to add 3D features to the pixel on RGB frames in order to incorporate geometric information is a challenging task. In this paper, we propose a simple and general framework combining geometric information of depth maps and RGB information of color maps for semantic segmentation task. Specifically, we first extract geometric feature from an associated point cloud which is harnessed from depth map, and then the RGB feature from color map. Due to the regular format of depth map, the geometric feature can be easily mapped to the corresponding pixel on RGB feature. After that, we get a combination of RGB and geometric features with our Element-Max-Min-Fuse function. Additionally, an efficient decoder module is designed to refine the segmentation results. We demonstrate the effectiveness of the proposed model on S3DIS dataset, the experimental results show that our method enhances the result of using RGB image or point cloud alone.