Occluded apples orientation estimator based on deep learning model for robotic harvesting

Abstract

Apple harvesting robot is in rapid development in the recent years due to the shortage of manual labor. Robotics vision system has been one of the main focus in the development to achieve successful grasping. Information such as the apple orientation is valuable for developing more effective fruit detachment strategies. Occlusions in the orchard environment and the lack of distinct features in an apple has made it challenging to identify the apple orientation. Our study presents a novel approach that aims to estimate the orientation of fruits by projecting 2D information onto a 3D space. This method specifically targets the field of robotic harvesting and leverages multiple well-established and reliable fundamental methods currently available that includes keypoint detection neural network and circle detection based on extracted line segments of occluded apple. Based on the extracted information, a unit vector that represents orientation of the apple can be computed and transformed according to the relative position between the apple and the camera. The proposed method is evaluated using a publicly available 3D point cloud data of 11 apple trees from an orchard farm that was additionally labeled manually with apple orientations. To verify the evaluation results, the proposed method is tested by integrating it into an apple harvesting robot to perform inspection. Results shows that the median angular error in the orchard setting is 17.6° whilst in the lab setting is 14.6°. The experimental results show that the performance of the proposed method is comparable to existing research when under heavy occlusion and thus it is suitable for harvesting robot operating in the orchard farm.