Semantic part segmentation method based 3D object pose estimation with RGB-D images for bin-picking

Abstract

• The Semantic PPF method that incorporates CNN based part instance segmentation with point cloud for the pose estimation is proposed. • A physically-simulated engine for dataset generation is proposed for semantic object-part segmentation, which reduces the cost of constructing datasets for real scenes. • The Part Mask RCNN is proposed to predict category, bounding box, object mask and object-part mask in the RGB image. • The robustness and effectiveness of the voting-based pose estimation algorithm are improved using semantic information of objects comparing to the original PPF method. • The value accumulator is proposed to reduce the discretization error in the voting stage of the PPF method. 3D object pose estimation for grasping and manipulation is a crucial task in robotic and industrial applications. Robustness and efficiency for robotic manipulation are desirable properties that are still very challenging in complex and cluttered scenes, because 3D objects have different appearances, illumination and occlusion when seen from different viewpoints. This article proposes a Semantic Point Pair Feature (PPF) method for 3D object pose estimation, which combines the semantic image segmentation using deep learning with the voting-based 3D object pose estimation. The Part Mask RCNN ispresented to obtain the semantic object-part segmentation related to the point cloud of object, which is combined with the PPF method for 3D object pose estimation. In order to reduce the cost of collecting datasets in cluttered scenes, a physically-simulated environment is constructed to generate labeled synthetic semantic datasets. Finally, two robotic bin-picking experiments are demonstrated and the Part Mask RCNN for scene segmentation is evaluated through the constructed 3D object datasets. The experimental results show that the proposed Semantic PPF methodimproves the robustness and efficiency of 3D object pose estimation in cluttered scenes with partial occlusions.