2D pose estimation of multiple tomato fruit-bearing systems for robotic harvesting

Abstract

Fruit detection is a priority for successful robotic harvesting. It should be able to provide geometrical information regarding the target fruit to be harvested. Recently, branch detection has been frequently used to determine cutting points for harvesting fruits, and studies have mostly focused on pedicel (or peduncle) detection in tomatoes. Previous studies detected the target tomato first and then estimated the cutting points on pedicels under the assumption that the target could be harvested; however, the cutting point of the detected tomato could not be guaranteed to be estimated visually by occlusions and postures. It is efficient to grasp the poses of all tomato-pedicel pairs in a scene simultaneously for determining the target object to be easily harvested. Thus, we proposed the 2D pose estimation of multiple tomatoes with pedicel based on the bottom-up method; we first detected all the tomato pose-related components in an image and then estimated the poses of each object. The proposed approach used a human pose estimation method as the backbone, and the model was trained suite to the four keypoints defined in this study: tomato-center, calyx, abscission zone, and branch point. The results demonstrate that the percentage of detected keypoints (PDK@0.3) was 0.85 and 0.80 for keypoints and their linkages, respectively, and the distal pedicel, which is a important part as a cutting point for harvesting, indicated a performance higher than 0.94. In conclusion, the proposed method can provide a multiple tomato-pedicel pose that can perform with a relatively constant amount of computation, regardless of the number of objects in a scene and can contribute to determine harvesting priorities based on accessibility evaluation during robotic harvesting.