Thin wire segmentation and reconstruction based on a novel image overlap-partitioning and stitching algorithm in apple fruiting wall architecture for robotic picking

Abstract

The layout of orchards usually requires the use of wires, to provide sturdy support. Such is the case of apple trees in fruiting wall architecture, where wires are conducive for mechanical harvesting and especially robotic picking. However, wires may cause damage to the robotic gripper, especially when direct picking occluded apple fruits. Hence, the importance of identifying the wires of a fruit wall architecture. In this study, a pixel-level segmentation network, BlendMask, was adopted to segment wires. The wires are thin and normally behind the branches or leaves, making difficult for their identification. Therefore, a novel data processing algorithm called image overlap-partitioning and stitching (IOS) is proposed for BlendMask to segment the wires. A total of 82 RGB (Red, Green, and Blue) images registered to create a raw dataset. The dataset was augmented and then overlap-partitioned into 12,736 images with a resolution of 800 × 1024 pixels and corresponding annotation files based on input size of BlendMask to make an overlap-partitioned dataset. Then BlendMask was trained with the overlap-partitioned dataset and tested on the other images, where additional stitching for the overlap-partitioned images was needed in the image testing dataset. Results showed that BlendMask with IOS obtained Intersection over Union (IoU) and pixel accuracy of 43.86 % and 61.01 %, respectively. It achieved a better average precision of 38.75 % with IoU of 0.5 on the overlap-partitioned dataset, which was 38.42 % higher than the full image dataset. Moreover, a reconstruction method based on feature point extraction and fitting was proposed to estimate wire skeletons, which achieved a reconstruction accuracy of 90.70 %. These results showed a promising potential using segmentation and reconstruction methods for identifying wires and thus providing a basis for robotic picking in modern orchards.