Instance segmentation of point cloud captured by RGB-D sensor based on deep learning

Abstract

ABSTRACT RGB-D sensors are gradually introduced into the robotic system to help the machine understand its surroundings. Among the point cloud processing methods, instance segmentation of point cloud is extremely important since the quality of segmentation will affect the performance of subsequent algorithms. In this paper, the 3D reconstruction process of RGB-D sensor is analyzed, and a framework PointSeg is proposed to handle the instance segmentation of point cloud captured by RGB-D sensor. The PointSeg realizes point cloud instance segmentation by applying the deep learning method YOLACT++ to instance segment the color image first and then matching the instance information with the point cloud. In addition, an experimental platform that is equipped with a Kinect v2 is built, and a dataset is set up and then the PointSeg is tested on the dataset. The result shows that the PointSeg achieves point cloud instance segmentation according to the instance information extracted from color images, which not only has good real-time performance but also has better instance segmentation accuracy compared with the method of conducting instance segmentation directly on point cloud data, and the introduction of data augmentation in the training phase can achieve good training effect even on a small training dataset.