Apple Tree Trunk and Branch Segmentation for Automatic Trellis Training Using Convolutional Neural Network Based Semantic Segmentation

Abstract

Apple orchard in modern fruiting wall architectures (e.g. Vertical and V-trellis) help to attain high fruit yield and quality. These systems are also key to developing simpler tree canopies, which improves productivity of manual orchard operations while creating opportunities for automated field operations such as robotic harvesting and/or pruning. Training of fruit trees to these architectures is carried out manually, which is becoming challenging due to the increasing labor cost and uncertainty in the labor availability. With reduced cost and increasing speed and robustness of sensing and robotic technologies, automated tree training could be a viable alternative. One of the most important steps to automate tree training operation is to segment out the trunk and branches of the trees that are ready to be trained and then select the suitable branch for the training. In this work, a trunk and branch segmentation method was developed using Kinect V2 sensor and deep learning-based semantic segmentation. Kinect was used to acquire point cloud data of the tree canopies in a commercial orchard. Depth and RGB information extracted from the point cloud data were used to remove the background trees from the RGB image. Then trunk and branches of the tree that share the common appearance and features were segmented out using a convolutional neural network (SegNet) for the semantic segmentation. We achieved trunk and branch segmentation accuracy of 0.92 and 0.93 and the mean intersection-over-union (IoU) score of 0.59 and 0.44, respectively. The Boundary-F1 score, which directly relates the accuracy of the segmented region boundaries, was 0.93 and 0.88 for the trunk and branch, respectively. These assessments show the potential of the deep learning-based semantic segmentation for automated branch detection in the orchard environment, which provides a foundation for developing automated tree training systems.