PolishNet-2d and PolishNet-3d: Deep Learning-Based Workpiece Recognition

Fuqiang Liu,Zongyi Wang

doi:10.1109/access.2019.2940411

Abstract

In recent years, there have been many deep learning research projects based on two-dimensional object detection and three-dimensional point cloud recognition. However, relatively few of these combine the two, and the number of projects based on a three-dimensional workpiece recognition method for the industrial field is even fewer. In this paper, to perform the recognition task for polishing workpieces in manufacturing, we use RGB-D images as input, and propose our designed PolishNet-2d for two-dimensional workpiece detection and our designed PolishNet-3d for three-dimensional workpiece recognition. The two deep neural networks are employed together in series to first detect and then recognize polishing workpieces in an industrial environment. For this paper, a large number of experiments have been carried out on deep learning datasets of polishing workpieces. These datasets were created by us to contain diverse combinations of real and simulated workpieces in real and simulated industrial environments. The contributions of this paper are as follows: (1) A rotation parameter learning network is proposed and a two-dimensional workpiece detection neural network, named PolishNet-2d, which was constructed by integrating our designed algorithms with the backbone networks ResNet101 and region proposal network (RPN), is introduced; (2) A hierarchical feature extraction network is proposed and a three-dimensional workpiece recognition neural network, named PolishNet-3d, which was constructed by integrating our designed algorithms with the backbone network PointNet, is introduced; (3) PolishNet-2d and PolishNet-3d are employed in series, with the detection output of PolishNet-2d being used as the input for PolishNet-3d for its recognition tasks: the workpiece regions are detected in the RGB image; the workpiece point cloud is segmented in the corresponding regions in the depth image, and lastly the segmented point cloud is placed into PolishNet-3d to identify the workpiece; (4) For the experiments in this paper, datasets containing rich and diverse data types of polishing workpieces for industrial fields have been constructed; (5) Numerous experimental results on polishing workpiece datasets show that the conjunction of PolishNet-2d and PolishNet-3d can achieve exemplary recognition results on polishing workpiece datasets.

Highlights

Nowadays, in the manufacturing industrial field, it is still a frontier topic and a difficult problem to recognize three-dimensional workpieces stably, robustly and accurately
Combining the previous two-dimensional workpiece detection network, PolishNet-2d; the three-dimensional workpiece point cloud recognition network, PolishNet-3d; the random sample consensus (RANSAC) based spatial point cloud segmentation method; and a new gray contour image and depth contour image fusion algorithm, the workpiece images captured in the actual industrial scenes are processed to obtain the two-dimensional workpiece detection and the three-dimensional workpiece point cloud recognition
In the workpiece regions detected by PolishNet-2d, the point cloud data transformed from depth images are segmented by the RANSAC algorithm

Summary

Introduction

In the manufacturing industrial field, it is still a frontier topic and a difficult problem to recognize three-dimensional workpieces stably, robustly and accurately. C: ANALYSIS OF TRAINING AND TESTING RESULTS According to the results, Fig. 8 and Table 2, it can be seen that PolishNet-2d achieves higher accuracy and AUC values than Faster R-CNN, both in the case of where there is a single workpiece in the image and in the case of where there are multiple workpieces in the image.

Results

Conclusion