Prioritizing robotic grasping of stacked fruit clusters based on stalk location in RGB-D images

Abstract

Determining the priority order for grasping stacked fruit clusters such as those of grape and lychee is a challenging problem in robotic manipulation based on RGB-D images. Grasping fruit cluster stalks at the top of a stack can improve robotic grasping success and reduce fruit damage. Hence, stalk location is an important factor affecting grasping priority. Because of the disordered arrangement of fruit clusters, stalks are not neatly arranged in horizontal planes, and this makes it difficult to determine grasping priority by stalk vertical location. Therefore, this paper proposes a prioritization method for grasping stacked fruit clusters by extracting and classifying “stalk depth sets“ in RGB-D images. For extracting small stalks with multiple background elements in RGB-D images, a convolutional neural network (CNN) is constructed to reduce the variation in resolution across the feature map and to retain details and local features. To reduce background interference, a coarse-to-fine cascaded Faster R-CNN based on multi-scale feature maps is proposed. A transfer learning scheme is used to improve the network’s learning capability for stalk depth sets. Using grape cluster experiments with a robotic sorting system, the proposed method was verified and compared with the existing LeNet-5, AlexNet, VGG16, and Faster R-CNN methods. Compared with existing networks, the average precisions of cascaded Faster R-CNN for extracting stalk depth sets and constructed CNN trained by improved transfer learning for grasping prioritization increased by more than 8.01% and 3.79%, respectively. The experimental results demonstrate that the proposed method can accurately extract and classify stalk depth sets and improve grasping prioritization for stacked fruit clusters.