Deep learning based segmentation for automated training of apple trees on trellis wires

Abstract

Trellised fruiting-wall training systems are becoming standard for modern apple orchards due to their high fruit yield and quality, and their suitability to robotic operations in pruning and harvesting. In a common practice of training young apple trees to a trellis-trained canopy system in PNW region of US, trees branches are manually selected and then tied to horizontal trellis wires in 6 or 7 tiers. As manual training of apple trees to these modern orchard architectures is becoming challenging due to less availability of skilled human labor with quickly increased labor cost, automated training using sensing and robotic techniques could be an alternative solution. Segmenting trunks, branches, and trellis wires is a critical step in automating a tree training operation. In this study, a deep learning-based semantic segmentation method was developed for automatically performing this segmentation task. A Kinect V2 sensor was used to obtain the RGB and point cloud data of target trees. Then both Simple- and Foreground-RGB images were used for training a convolutional neural network (CNN)-based segmentation network (SegNet) to segment the trunk, branch, and trellis wire. Trunks and branches, which share some common features, were segmented from each other with accuracies of 0.82 and 0.89 for Simple-RGB images and 0.91 and 0.92 for Foreground-RGB images, respectively. Similarly, trellis wires, which have distinct features from both the trunk and branches, were segmented with accuracies of 0.92 and 0.97 for the Simple- and Foreground-RGB images, respectively. Obtained results showed that the performance of the developed semantic segmentation technique was better with Foreground-RGB images compared to the same with Simple-RGB images. Accuracy in identifying the segmented region boundaries in Foreground-RGB images, represented by the Boundary-F1 score, was 0.93, 0.89, and 0.91 for the trunk, branch, and trellis wire, respectively. These results showed a promising potential for adopting deep learning-based semantic segmentation for automating apple tree training in orchard environment.