An image-based system for locating pruning points in apple trees using instance segmentation and RGB-D images

Abstract

Intelligent pruning is an effective way to improve the efficiency of fruit tree pruning and reduce production costs, where the positioning of fruit tree pruning points is the key. In this study, a pruning point localisation method based on deep learning and red-green-blue-depth (RGB-D) is proposed for dormant tall spindle apple trees, which can quickly and accurately identify the pruning points on the primary branches. Firstly, red-green-blue (RGB) images and depth images of apple trees were acquired by using the Realsense D435i depth camera, and the SOLOv2 instance segmentation model was used to segment the trunks, branches, and supports in RGB images. Secondly, the manual pruning rules were adapted to improve the pruning methods; then using OpenCV image processing method, the junction points of branches and trunk and potential pruning points were gained, and the world coordinates of the points were obtained according to the coordinate transformation to calculate the length of branch diameter and spacing. Finally, the pruning points were determined according to the pruning rules. The results show that SOLOv2 has better segmentation effect compared with Mask R–CNN and Cascade Mask R–CNN. The mean absolute error between the estimated and manually measured values of branch diameter and spacing are 1.10 mm and 16.06 mm, and the recognition accuracy of pruning points is 87.2%, with a recognition time of about 3.8 s for each image. It is shown that the method can measure branch diameter and spacing and quickly locate pruning points with high reliability and accuracy, and the study provides a basis for the development of apple tree pruning robots.