Predicting positions and orientations of individual kiwifruit flowers and clusters in natural environments

Abstract

The positional and orientational detection of kiwifruit flower clusters in the natural environment is essential for the development of automated mechanical pollination systems. To ensure the timeliness and accuracy of this process, we propose a deep-learning-based method that predicts kiwifruit flower positions and orientations. Based on a single flower, this method effectively predicts the position and orientation of the entire cluster of flowers. First, the SOLOv2 instance segmentation model is used to locate and segment the location of the target kiwifruit flowers. Subsequently, the MobileNetV2 key point detection model is employed to detect the key points of the centre of gravity of the petal contour and the stamen part of the kiwifruit flower, and draw the orientation prediction line of the corresponding clusters. Finally, flower clusters are determined by calculating the Intersection over Union (IoU) of kiwi flowers. By taking the average of the orientation prediction line of each kiwi flower in a cluster as the entire cluster’s orientation, the method effectively generates a prediction of the said orientation. To verify the effectiveness of the proposed method, we compared and analysed the effects of SOLOv2 with the YOLOv5s, YOLOv4, and Faster-RCNN models with respect to the segmentation effect. The comparison results show that the SOLOv2 model exhibits a fast segmentation speed while maintaining a high segmentation accuracy. Verification results indicate that the SOLOv2 model has an accuracy rate of 74.2% for the detection of multi-target flowers of clustered kiwifruit from a wide perspective angle. It is therefore feasible to apply the proposed method to detect the positions and orientations of kiwi flower clusters in real time. This study can provide a technical reference for the development of robotic arms for the purpose of kiwifruit pollination.