A Transformer-based multi-modal fusion network for 6D pose estimation

Abstract

6D pose estimation has garnered significant attention and research. RGB images and point clouds converted from RGB-D images provide complementary color and geometry information, making them the mainstream data sources for object 6D pose estimation. However, due to the fact that RGB image and point cloud belong to different dimensional spaces and have different distribution characteristics, the fusion of these two complementary data sources remains a key technical challenge for 6D pose estimation. In contrast to prior approaches that simply concatenate separately processed RGB images and point clouds, this work introduces a Transformer-based multi-modal fusion network to address this challenge. More precisely, We build a Transformer architecture based pixel-wise feature extraction to optimize feature extraction from RGB images and point clouds. Subsequently, we investigate various multi-modal feature fusion methods to process these features, enabling deeper fusion of complementary data. Additionally, during the experimental phase, we design a 6D pose estimation network based on depth prediction to assess the impact of point cloud accuracy on the multi-modal fusion module. Finally, the proposed method is verified on four datasets: LineMOD, Occlusion Linemod, MP6D and YCB-Video. Experimental results show that the proposed method outperforms similar methods on these datasets.