AGHRNet: An attention ghost-HRNet for confirmation of catch‐and‐shake locations in jujube fruits vibration harvesting

Abstract

The development of an intelligent jujube fruit harvesting device is a critical step in achieving the whole mechanization process. Catch‐and‐shake harvesting, as an efficient and stable vibration harvesting method, has widely been used to save labor and improve harvesting efficiency in large-scale jujube orchards. However, existing catch‐and‐shake harvesters still rely heavily on the operator's naked eyes to determine the shaking position, which is subjective, highly inefficient and highly labor intensive. To address this problem, this study proposes a computer vision system including truck and branches segmentation, skeleton extraction and vibration position selection to automatically identify appropriate shaking positions. A lightweight attention Ghost-HRNet (AGHRNet) based on deep learning is designed to separate the truck and branches from the complex orchard background. In AGHRNet, an attention Ghost block is proposed to lighten the model and improve segmentation accuracy. Moreover, we also construct a hybrid loss function to solve the pixel-level category imbalance of the trunks, branches and background. To verify the effectiveness of the AGHRNet, ten state-of-the-art semantic segmentation methods, including the FCN8s, SegNet, U-Net, PSPNet, DeepLabv3+, BiSeNet, DDRNet, DANet, SegFormer and HRNet, are compared. Comparative experimental results show that compared with the other state-of-the-art methods, the proposed AGHRNet has higher segmentation accuracy (77.79 % mIoU, 89.46 % mPA) and a smaller model size (9.53 MB) on our datasets. Finally, mask medial axis extraction and the minimum inscribed circle operation are applied to obtain the appropriate vibration location. The computer vision system can provide technical references for catch‐and‐shake harvesting and developing jujube fruit-harvesting robots.