Mature pomegranate fruit detection and location combining improved F-PointNet with 3D point cloud clustering in orchard

Abstract

Fruit detection and localization is of great significance for horticulture work and robotic harvesting in orchards. Although the existing studies of fruit detection have achieved good results based on 2D image analysis, accurate fruit detection on trees is still challenging because of illumination changes, shielding of leaves and branches, overlapping of fruits and so on. To improve the accuracy of fruit detection and location, this paper proposes a novel ripe pomegranate fruit detection and location method based on improved F-PointNet and 3D clustering method, which is consisting of: (1) RGB-D feature fusion Mask R-CNN was used to realize fruit detection and segmentation; (2) PointNet combined with OPTICS algorithm based on manifold distance and PointFusion was used to segment point clouds in the frustum fruit region, and 3D box was placed in the region of interest; (3) The sphere fitting was performed to obtain the position and the size of a pomegranate. The comparative experiments have been carried out and analyzed, the RGB-D feature fusion Mask R-CNN has the best performance with the F1 score of 0.845 and the AP score of 0.952 respectively, and the improved F-PointNet has better performance than the classical F-PointNet. The measurement radius experiment results of 100 pomegranate samples randomly selected demonstrate that the RMSE is 0.235 cm, the R2 is 0.826, and the position error is less than 5 mm. These results validate that the proposed detection and location method can effectively detect and locate a single ripe pomegranate under unstructured orchard environment.