U2ESPNet—A lightweight and high-accuracy convolutional neural network for real-time semantic segmentation of visible branches

Abstract

To alleviate the impact of shrinking labor on seasonal apple harvesting efforts, development of apple harvesting robots is in full swing. For harvesting robots, efficiency and productivity will increase with their intelligence, and the first step to improve the intelligence is how to recognize the branches of apple trees with high-accuracy and real-time under constrained hardware. In this paper, we combined the advantages of Efficient Spatial Pyramid (ESP) and U2Net in terms of efficiency and accuracy to design an extremely lightweight, high-accuracy and real-time convolutional neural network, U2ESPNet, for semantic segmentation of branches under constrained hardware. U2ESPNet is a two-level nested U-shaped network, i.e., both the top and bottom levels are U-shaped structures whose basic component modules are ESPs. Such a unique design has 3 strengths: (1) the ability to adequately capture more contextual information from different scales; (2) the efficient performance of ESPs in terms of computation, memory, and power; and (3) the capability to significantly increase architectural depth without unduly increasing computational cost. Under the same conditions, the segmentation performance of 8 different networks was evaluated, including U2ESPNet, BiseNetV2, ESPNetV2, etc., in our own constructed branch datasets. The experimental results showed that U2ESPNet achieved the best Intersection over Union (IoU) and F1-score in visible branches segmentation with 75.35% and 85.94%, respectively. Most importantly, the Total Params and Floating-Point Operations (FLOPs) of U2ESPNet were only 0.29M and 1.05G, respectively, which were much lower than the other 7 networks. For images with a resolution of 1280 × 720, U2ESPNet achieved a segmentation speed of 55.25 frames per second (FPS) with the GPU(RTX3090), meeting the real-time requirement. This study proved that our proposed U2ESPNet has outstanding performance and great potential for mobile harvesting robots with constrained hardware devices.