A convolutional neural network (Seg-CNN) for recognition, classifi cation, and segmentation of apple fruit branches and stems in images

Abstract

This article reports the results of research studies conducted in 2023–2024 on transfer learning of Segmentation Convolutional Neural Networks (Seg-CNN) models for classification, recognition, and segmentation of branches with apple fruits and stems in images. State-of-the-art convolutional neural network architectures, i.e., YOLOv8(n,s,m,l,x)-seg, were used for a detailed segmentation of biological objects in images of varying complexity and scale at the pixel level. An image dataset collected in the fi eld using a GoPro HERO 11 camera was marked up for transfer model training. Data augmentation was performed, producing a total of 2500 images. Image markup was performed using the polygon annotation tool. As a result, polygonal contours around objects were created, outlines of branches, apple tree fruits, and stems were outlined, and segments of objects in the images were indicated. The objects were assigned the following classes: Apple branch, Apple fruit, and Apple stem. Binary classification metrics, such as Precision and Recall, as well as Mean Average Precision (mAP), were used to evaluate the performance of the trained models in recognizing branches with apple fruits and stems in images. The YOLOv8x-seg (mAP50 0.758) and YOLOv8l-seg (mAP50 0.74) models showed high performance in terms of all metrics in recognizing branches, apple fruit, and fruit stems in images, outperforming the YOLOv8n-seg (mAP50 0.7) model due to their more complex architecture. The YOLOv8n-seg model has a faster frame processing speed (11.39 frames/s), rendering it a preferred choice for computing systems with limited resources. The results obtained confirm the prospects of using machine learning algorithms and convolutional neural networks for segmentation and pixel-by-pixel classification of branches with apple fruits and stems on RGB images for monitoring the condition of plants and determining their geometric characteristics.