Picking point recognition for ripe tomatoes using semantic segmentation and morphological processing

Abstract

Identifying picking points is the key to the intelligent picking of ripe tomatoes and an important guarantee of efficient and lossless picking. A semantic segmentation model with the improved Swin Transformer V2 and a picking point recognition algorithm based on the connection of tomato fruit, calyx and stem are proposed for the problem of picking point recognition of ripe tomatoes in complex environments. Firstly, the tomato fruit, calyx and stem are accurately identified by an improved semantic segmentation model. On the one hand, the SeMask module is integrated into the traditional Swin Transformer V2 architecture for incorporating semantic information of the image into the encoder to improve the segmentation performance. On the other hand, UPerNet is used as the decoder of the model to improve the prediction results by integrating the feature selection module and feature alignment module into the decoder. Secondly, the calyx regions and stem regions of ripe tomatoes are extracted by a connected component labeling algorithm and morphological processing. Finally, the picking points located on the tomato lateral stems are extracted by the image refinement algorithm, deburring algorithm and position constraints of the tomato calyxes. The results of the ablation and comparison experiments show that the improved semantic segmentation model proposed in this paper has the best performance on Mean Intersection over Union (MIoU) and Mean Pixel Accuracy (MPA) with 82.5% and 89.79%, respectively. After accurately identifying tomato fruits, calyxes and stems, the picking point recognition algorithm proposed in this paper has 97.43% and 86.04% on precision and recall, respectively. This research provides an effective picking point recognition method for tomato harvesting robots.