Canopy-attention-YOLOv4-based immature/mature apple fruit detection on dense-foliage tree architectures for early crop load estimation

Abstract

Accurate detection of both immature and mature apples in orchard environments is essential for early crop load management. A near real-time method is proposed in this study for detecting green (early-stage), green–red-mixed (mid-stage; red varieties), or red apples (harvest-stage; red varieties). Both the number of fruits and fruit size were estimated for the entire tree with a single image captured by a low-cost smartphone using two different imaging methods (oblique and panorama modes). An attention mechanism module called the convolutional block attention module (CBAM) was added to the generic YOLOv4 detector to improve the detection accuracy by only focusing on the target canopies. Furthermore, an adaptive layer and larger-scale feature map were included in the modified network structure, enabling it to adapt to various characteristics of fruits and canopies during the entire growing season, such as different fruit colors and sizes, dense-foliage conditions, and severe occlusions. To verify the effectiveness of the proposed method, we compared our improved model, canopy-attention-YOLOv4 (or CA-YOLOv4), with other commonly adopted models available in the literature, such as the original YOLOv4, Faster R-CNN, and single-shot multibox detector (SSD). Two commonly planted apple varieties, “Envy” and “Scifresh”, were used in the study. The results showed that the proposed CA-YOLOv4 detector performed the best among all the algorithms, with up to ∼3% improvement in terms of fruit counting over the original YOLOv4. With the “Envy” variety, fruit detection accuracies were 86.2%, 87.5%, and 92.6% for the early-, mid-, and harvest stages, respectively, whereas the same were 71.0%, 83.6%, and 86.3% for the “Scifresh” variety, which has denser canopy foliage. Both imaging methods proposed in this study only needed one/single shot targeting the entire fruiting tree, which can be highly efficient for real-world applications in crop load management. Finally, CA-YOLOv4 estimated fruit sizes were compared to manual measurements, achieving up to R2 values of 0.68 in fruit height and 0.66 in fruit width estimations.