A Method of Grasping Detection for Kiwifruit Harvesting Robot Based on Deep Learning

Abstract

Kiwifruit harvesting with robotics can be troublesome due to the clustering feature. The gripper of the end effector will easily cause unstable fruit grasping, or the bending and separation action will interfere with the neighboring fruit because of an inappropriate grasping angle, which will further affect the success rate. Therefore, predicting the correct grasping angle for each fruit can guide the gripper to safely approach, grasp, bend and separate the fruit. To improve the grasping rate and harvesting success rate, this study proposed a grasping detection method for a kiwifruit harvesting robot based on the GG-CNN2. Based on the vertical downward growth characteristics of kiwifruit, the grasping configuration of the manipulator was defined. The clustered kiwifruit was mainly divided into single fruit, linear cluster, and other cluster, and the grasping dataset included depth images, color images, and grasping labels. The GG-CNN2 was improved based on focal loss to prevent the algorithm from generating the optimal grasping configuration in the background or at the edge of the fruit. The performance test of the grasping detection network and the verification test of robotic picking were carried out in orchards. The results showed that the number of parameters of GG-CNN2 was 66.7 k, the average image calculation speed was 58 ms, and the average grasping detection accuracy was 76.0%, which ensures the grasping detection can run in real time. The verification test results indicated that the manipulator combined with the position information provided by the target detection network YOLO v4 and the grasping angle provided by the grasping detection network GG-CNN2 could achieve a harvesting success rate of 88.7% and a fruit drop rate of 4.8%; the average picking time was 6.5 s. Compared with the method in which the target detection network only provides fruit position information, this method presented the advantages of harvesting rate and fruit drop rate when harvesting linear clusters, especially other cluster, and the picking time was slightly increased. Therefore, the grasping detection method proposed in this study is suitable for near-neighbor multi-kiwifruit picking, and it can improve the success rate of robotic harvesting.