Harvesting Robot Research Articles

Human-Centered Robotic System for Agricultural Applications: Design, Development, and Field Evaluation

This paper introduce advancements in agricultural robotics in response to the increasing demand for automation in agriculture. Our research aims to develop humancentered agricultural robotic systems designed to enhance efficiency, sustainability, and user experience across diverse farming environments. We focus on essential applications where human labor and experience significantly impact performance, addressing four primary robotic systems, i.e., harvesting robots, intelligent spraying robots, autonomous driving robots for greenhouse operations, and multirobot systems, as a method to expand functionality and improve performance. Each system is designed to operate in unstructured agricultural environments, adapting to specific needs. The harvesting robots address the laborintensive demands of crop collection, while intelligent spraying robots improve precision in pesticide application. Autonomous driving robots ensure reliable navigation within controlled environments, and multirobot systems enhance operational efficiency through optimized collaboration. Through these contributions, this study offers insights into the future of agricultural robotics, emphasizing the transformative potential of integrated, experience-driven intelligent solutions that complement and support human labor in digital agriculture.

On-the-go table grape ripeness estimation via proximal snapshot hyperspectral imaging

The monitoring of grapes for ripeness estimation is a practice that enables fruit harvesting at the optimal time. Hyperspectral Imaging (HSI) represents a non-destructive and high-throughput alternative to traditional laboratory analyses. Current literature approaches perform hyperspectral measurements using line scan sensors or low-resolution static snapshot cameras, which hinder a fast per-bunch ripeness characterization. We propose a framework for on-the-go collection and processing of proximal snapshot hyperspectral images to estimate single bunch ripeness parameters. Focusing on table grapes (Vitis vinifera L. cv. Red Globe), we collected images under natural illumination with a hyperspectral camera (500–900 nm) mounted on a moving vehicle in an experimental block sited in Piacenza, Italy. We investigated images collected in August and September 2021 representing two ripening stages. The composition of the imaged grape bunches was determined through laboratory chemical analyses to predict Total Soluble Solids (TSS) and anthocyanin concentration. The images were pre-processed via multimodal image registration to correct the unalignment of bands due to the vehicle motion, and the single bunches were automatically identified on false RGB images through a Mask Region-Convolutional Neural Network (Mask R-CNN) instance segmentation network. The mean spectra of the bunches were used as input features of a Partial Least Squares Regression (PLSR) model to predict the chemical parameters at single bunch and whole vine scales. The regression model of TSS had an R2 (10-fold nested cross-validation) of 0.75 and 0.85 on a per-bunch and per-vine basis, respectively. The regression model of anthocyanin had an R2 of 0.68 and 0.49 on a per-bunch and per-vine basis, respectively. The results suggest the potential of using snapshot hyperspectral images for high-throughput analysis of a per-bunch grape ripeness estimation. The method described in this study could give valuable information to improve grape ripening monitoring and management of harvest operations and even allow for precise and automated robotic harvesting.

Barrier-free tomato fruit selection and location based on optimized semantic segmentation and obstacle perception algorithm

With the advancement of computer vision technology, vision-based target perception has emerged as a predominant approach for harvesting robots to identify and locate fruits. However, little attention has been paid to the fact that fruits may be obscured by stems or other objects. In order to improve the vision detection ability of fruit harvesting robot, a fruit target selection and location approach considering obstacle perception was proposed. To enrich the dataset for tomato harvesting, synthetic data were generated by rendering a 3D simulated model of the tomato greenhouse environment, and automatically producing corresponding pixel-level semantic segmentation labels. An attention-based spatial-relationship feature extraction module (SFM) with lower computational complexity was designed to enhance the ability of semantic segmentation network DeepLab v3+ in accurately segmenting linear-structured obstructions such as stems and wires. An adaptive K-means clustering method was developed to distinguish individual instances of fruits. Furthermore, a barrier-free fruit selection algorithm that integrates information of obstacles and fruit instances was proposed to identify the closest and largest non-occluded fruit as the optimal picking target. The improved semantic segmentation network exhibited enhanced performance, achieving an accuracy of 96.75%. Notably, the Intersection-over-Union (IoU) of wire and stem classes was improved by 5.0% and 2.3%, respectively. Our target selection method demonstrated accurate identification of obstacle types (96.15%) and effectively excluding fruits obstructed by strongly resistant objects (86.67%). Compared to the fruit detection method without visual obstacle avoidance (Yolo v5), our approach exhibited an 18.9% increase in selection precision and a 1.3% reduction in location error. The improved semantic segmentation algorithm significantly increased the segmentation accuracy of linear-structured obstacles, and the obstacle perception algorithm effectively avoided occluded fruits. The proposed method demonstrated an appreciable ability in precisely selecting and locating barrier-free fruits within non-structural environments, especially avoiding fruits obscured by stems or wires. This approach provides a more reliable and practical solution for fruit selection and localization for harvesting robots, while also being applicable to other fruits and vegetables such as sweet peppers and kiwis.

SGW-YOLOv8n: An Improved YOLOv8n-Based Model for Apple Detection and Segmentation in Complex Orchard Environments

This study addresses the problem of detecting occluded apples in complex unstructured environments in orchards and proposes an apple detection and segmentation model based on improved YOLOv8n-SGW-YOLOv8n. The model improves apple detection and segmentation by combining the SPD-Conv convolution module, the GAM global attention mechanism, and the Wise-IoU loss function, which enhances the accuracy and robustness. The SPD-Conv module preserves fine-grained features in the image by converting spatial information into channel information, which is particularly suitable for small target detection. The GAM global attention mechanism enhances the recognition of occluded targets by strengthening the feature representation of channel and spatial dimensions. The Wise-IoU loss function further optimises the regression accuracy of the target frame. Finally, the pre-prepared dataset is used for model training and validation. The results show that the SGW-YOLOv8n model significantly improves relative to the original YOLOv8n in target detection and instance segmentation tasks, especially in occlusion scenes. The model improves the detection mAP to 75.9% and the segmentation mAP to 75.7% and maintains a processing speed of 44.37 FPS, which can meet the real-time requirements, providing effective technical support for the detection and segmentation of fruits in complex unstructured environments for fruit harvesting robots.

PcMNet: An efficient lightweight apple detection algorithm in natural orchards

Apple detection plays a critical role in enabling the functionality of harvesting robots within natural orchard environments. To address challenges related to low detection accuracy, slow inference speed, and high parameter count, we present PcMNet, a lightweight detection model based on an improved YOLOv8 network. Initially, we employed Partial Convolution (Pconv) to construct a PR module, forming the Pconv-block, which subsequently replaced the original C2f feature extraction module within the YOLOv8n backbone. This replacement led to improvements in both detection accuracy and speed, while simultaneously reducing computational complexity (FLOPs), parameter count, and model size. Furthermore, the Cross-Scale Feature Fusion (CCFF) module was refined into Faster-Cross-Scale Feature Fusion (Faster-CCFF) with the integration of Pconv-block, significantly enhancing the model's feature extraction and fusion capabilities. Additionally, we introduced Mixed Local Channel Attention (MLCA) to further strengthen the model's capacity to capture essential features while effectively suppressing background noise. Experimental results demonstrate that PcMNet achieved a detection accuracy of 92.8 % and an mAP@0.5 of 95.5 %, representing improvements of 1.4 and 0.7 percentage points, respectively, over YOLOv8n. Moreover, PcMNet successfully reduced FLOPs, parameter count, and model size to 5.1 G, 1.4 M, and 3.2 MB, respectively. The per-image detection time was reduced to 2.3 ms, indicating reductions of 37.80 %, 53.33 %, 49.21 %, and 56.60 % in FLOPs, parameters, model size, and detection time compared to YOLOv8n. When deployed on edge computing devices with TensorRT acceleration, PcMNet achieved a detection rate of 92 FPS. Field validation experiments conducted in natural orchard environments confirmed PcMNet's superior ability to detect apples under challenging conditions, such as occlusions and varying lighting conditions. Its lightweight design and rapid detection capabilities provide a valuable reference for achieving real-time apple detection in automated and intelligent harvesting robots, thereby contributing to advancements in smart agriculture.

Automation and AI in Precision Agriculture: Innovations for Enhanced Crop Management and Sustainability

Precision agriculture is one of the ways to achieve food security and sustainability through better resource-use optimization and crop productivity dealing with the challenges posed by the growing population and addressing environmental concerns. The study offers an in-depth look at the most recent developments in artificial intelligence (AI) and automation in precision agriculture (PA), with a particular emphasis on important technologies such as drones, autonomous tractors, AI-driven irrigation systems, and predictive analytics for crop management. The accuracy of crop monitoring and health assessments has increased by 30–50 percent as a result of AI-powered solutions, which have improved resource-based decision-making. Systems for precision irrigation and fertilization have increased crop yields by 5–15 percent when using 25–40 percent less water and 30-40 percent less fertilizer, respectively. Robotic harvesters and sprayers are examples of automation technologies that have reduced labor expenses by 20–40 percent and increased operational efficiency by 35 percent. Additionally, AI-based prediction models have reduced pest damage by 20–25 percent and reached an accuracy of 85–90 percent for crop yield forecasts and pest control. Despite these developments, issues of scalability, affordability for small farms, and data privacy still exist, which can hinder technology adoption among farmers. The evaluation follows by outlining ideas for future research, such as 5G, blockchain, and AI integration with cloud and edge computing. These technologies could improve decision-making and transparency in precision agriculture by enabling real-time data transmission, secure data management, and enhanced traceability, thus addressing current limitations and fostering trust among stakeholders.

Cherry Tomato Detection for Harvesting Using Multimodal Perception and an Improved YOLOv7-Tiny Neural Network

Robotic fruit harvesting has great potential to revolutionize agriculture, but detecting cherry tomatoes in farming environments still faces challenges in accuracy and efficiency. To overcome the shortcomings of existing cherry tomato detection methods for harvesting, this study introduces a deep-learning-based cherry tomato detection scheme for robotic harvesting in greenhouses using multimodal RGB-D perception and an improved YOLOv7-tiny Cherry Tomato Detection (YOLOv7-tiny-CTD) network, which has been modified from the original YOLOv7-tiny by eliminating the “Objectness” output layer, introducing a new “Classness” method for the prediction box, and incorporating a new hybrid non-maximum suppression. Acquired RGB-D images undergo preprocessing such as color space transformation, point cloud normal vector angle computation, and multimodal regions of interest segmentation before being fed into the YOLOv7-tiny-CTD. The proposed method was tested using an AGV-based robot in a greenhouse cherry tomato farming facility. The results indicate that the multimodal perception and deep learning method improves detection precision and accuracy over existing methods while running in real time, and the robot achieved over 80% successful picking rates in two-trial mode in the greenhouse farm, showing promising potential for practical harvesting applications.

Improved Multi-Size, Multi-Target and 3D Position Detection Network for Flowering Chinese Cabbage Based on YOLOv8.

Accurately detecting the maturity and 3D position of flowering Chinese cabbage (Brassica rapa var. chinensis) in natural environments is vital for autonomous robot harvesting in unstructured farms. The challenge lies in dense planting, small flower buds, similar colors and occlusions. This study proposes a YOLOv8-Improved network integrated with the ByteTrack tracking algorithm to achieve multi-object detection and 3D positioning of flowering Chinese cabbage plants in fields. In this study, C2F-MLCA is created by adding a lightweight Mixed Local Channel Attention (MLCA) with spatial awareness capability to the C2F module of YOLOv8, which improves the extraction of spatial feature information in the backbone network. In addition, a P2 detection layer is added to the neck network, and BiFPN is used instead of PAN to enhance multi-scale feature fusion and small target detection. Wise-IoU in combination with Inner-IoU is adopted as a new loss function to optimize the network for different quality samples and different size bounding boxes. Lastly, ByteTrack is integrated for video tracking, and RGB-D camera depth data are used to estimate cabbage positions. The experimental results show that YOLOv8-Improve achieves a precision (P) of 86.5% and a recall (R) of 86.0% in detecting the maturity of flowering Chinese cabbage. Among them, mAP50 and mAP75 reach 91.8% and 61.6%, respectively, representing an improvement of 2.9% and 4.7% over the original network. Additionally, the number of parameters is reduced by 25.43%. In summary, the improved YOLOv8 algorithm demonstrates high robustness and real-time detection performance, thereby providing strong technical support for automated harvesting management.

In-field performance evaluation of robotic arm developed for harvesting cotton bolls

Robotic harvesting of cotton bolls combines the advantages of manual picking and mechanical harvesting, including selective picking and reduced labor dependency. Thus, we designed, developed, and tested a cotton-picking robotic arm equipped with a depth camera, a 4-DoF manipulator, and a vacuum-type end-effector. Convolutional neural networks were implemented for cotton boll recognition, while an artificial neural network-assisted particle swarm optimization-based model was utilized for solving inverse kinematics. Linear actuators, controlled by a microcontroller and relays, positioned the end-effector for cotton boll harvesting. Laboratory testing resulted in picking 44–49 cotton bolls per hour, taking approximately 73.0 s per boll. During field testing, the robotic arm picked 40–42 bolls per hour with an average time of 86.0 s per boll. Picking efficiency ranged from 76.19% to 85.71% in the lab and 66.32% to 72.04% in the field. The complex structure of cotton plants sometimes hindered the end-effector’s path, resulting in lower picking efficiency during field testing. As observed in present study, manual pickers achieved a significantly higher picking rate, harvesting 1.381 kg of cotton per hour, nearly 4.7 times more than the robotic arm. Manual pickers achieved 98.04% picking efficiency, significantly higher than the robotic arm’s 68.25%. This significant difference in efficiency is due to the agility, adaptability, and decision-making abilities of human pickers. The developed robotic arm demonstrates competitive efficiency in the domain of harvesting robots, contributing to advancements in robotic agriculture. Future research and development efforts in this field hold the potential to enhance the efficiency and productivity of robotic cotton harvesters.

Tomato Pedicel Physical Characterization for Fruit-Pedicel Separation Tomato Harvesting Robot

To solve the problem of the lack of physical properties of pedicels and the changing pattern for designing the end-effector of tomato harvesting robot and different harvesting modes, research was conducted on the physical properties of tomato pedicels and their change patterns. Using a Universal TA texture analyzer, tensile, three-point bending, and shearing tests were performed on tomato pedicels in the early firm-ripening stage. The tomato variety used was Syngenta Spectrum, cultivated seasonally with two crops per year. Spring crop tomatoes were used in this study. The experimental results provide a theoretical basis for designing tomato harvesting robots across three harvesting modes. Tensile tests measured the pull-off force and tensile strength of the abscission zone with varying diameters. These results are crucial for designing robots using a tensile harvesting mode. The location of the tomato pedicel significantly affects the shearing force. A one-way test was conducted on the shearing part. The results showed that the shearing force and energy required for the proximal pedicel are significantly greater than for the distal pedicel. To reduce the shearing force and energy needed by the end-effector’s shearing mechanism on distal pedicels, a response surface test was conducted. Three factors were examined: shearing speed, angle, and distal pedicel diameter. Design–Expert software optimized these factors to minimize shearing energy and force, leading to the best shearing parameters for different distal pedicel diameters. From the three-point bending tests, the average maximum bending breaking force, bending modulus, and bending strength of the tomato abscission zone were determined. These findings offer a theoretical basis for designing tomato harvesting robots with a bending-type harvesting mode.

Navigating the Landscape of Precision Horticulture: Sustainable Agriculture in the Digital Age

Automation, autonomy, and precision play vital roles in modern technology and procedures, especially in response to the rising global population and the need for improved fruit production technologies. The quality criteria for fruit yield are becoming increasingly complex, necessitating fundamental upgrades. Sensor-based technologies pave the way for more rational and efficient soil usage. Precision farming, an advanced agricultural approach, employs software technologies and principles to optimize various aspects of horticultural production. Its goal is to enhance crop yields and promote environmental sustainability by managing spatial and temporal variations. Precision horticulture, a cutting-edge farming method, focuses on increasing crop yields and quality while minimizing environmental impact. This approach leverages advanced technology and data-driven decision-making. By utilizing sensors, robots, drones, and other innovative tools for yield mapping, irrigation control, robotic harvesting, nutrient and pesticide application, soil sensing, and crop growth analysis, growers can monitor plant growth and health, identify potential issues, and make informed crop management decisions. Tailoring practices to specific plant needs can lead to higher yields, improved product quality, and reduced environmental impact. Overall, precision horticulture holds significant promise for promoting sustainable agricultural practices while meeting the increasing demand for food production.

Chinese Bayberry Detection in an Orchard Environment Based on an Improved YOLOv7-Tiny Model

The precise detection of Chinese bayberry locations using object detection technology is a crucial step to achieve unmanned harvesting of these berries. Because of the small size and easy occlusion of bayberry fruit, the existing detection algorithms have low recognition accuracy for such objects. In order to realize the fast and accurate recognition of bayberry in fruit trees, and then guide the robotic arm to carry out accurate fruit harvesting, this paper proposes a detection algorithm based on an improved YOLOv7-tiny model. The model introduces partial convolution (PConv), a SimAM attention mechanism and SIoU into YOLOv7-tiny, which enables the model to improve the feature extraction capability of the target without adding extra parameters. Experimental results on a self-built Chinese bayberry dataset demonstrate that the improved algorithm achieved a recall rate of 97.6% and a model size of only 9.0 MB. Meanwhile, the precision of the improved model is 88.1%, which is 26%, 2.7%, 4.7%, 6.5%, and 4.7% higher than that of Faster R-CNN, YOLOv3-tiny, YOLOv5-m, YOLOv6-n, and YOLOv7-tiny, respectively. In addition, the proposed model was tested under natural conditions with the five models mentioned above, and the results showed that the proposed model can more effectively reduce the rates of misdetections and omissions in bayberry recognition. Finally, the improved algorithm was deployed on a mobile harvesting robot for field harvesting experiments, and the practicability of the algorithm was further verified.

Detection of Camellia oleifera fruit maturity in orchards based on modified lightweight YOLO

Fruit maturity is the main factor affecting the quality and yield of Camellia oil. At present, accurate and efficient detection of the maturity level of Camellia oleifera fruits in orchards and enabling selective picking for harvesting robots is a crucial issue. Aiming at the challenge of low detection efficiency of Camellia oleifera fruit maturity and complex object detection models that are difficult to deploy to Camellia oleifera selective harvesting robots, a modified lightweight You Only Look Once model (YOLO-LM) based on YOLOv7-tiny is proposed. Specifically, three Criss-Cross Attention (CCA) modules are introduced to the backbone network to reduce the influence of leaves and branches occlusion as well as mutual occlusion between fruits. Then, the Adaptively Spatial Feature Fusion (ASFF) module is applied as the head to solve the limitation of inconsistency across different feature scales on the feature pyramid. Furthermore, the GSConv is introduced to replace the standard convolution of the Neck network to enable the model to simultaneously preserve model accuracy and reduce model complexity. Experiment results demonstrate that the precision, recall, mAP@0.5, parameters, FLOPs, and model size of YOLO-LM reached 93.96 %, 93.32 %, 93.18 %, 10.17 million, 19.46 G, and 19.82 MB, which outperformed YOLOv3, Faster R-CNN (VGG16), Faster R-CNN (ResNet50), YOLOv4, and YOLOv5m. Compared to YOLOv3-tiny, YOLOv4-tiny, YOLOv5s, YOLOv7-tiny, and YOLOv8s, the mAP@0.5 of YOLO-LM increased by 6.24 %, 6.11 %, 2.52 %, 2.45 %, and 0.57 %, respectively. Meanwhile, the parameters and model size of YOLO-LM are slightly higher than that of YOLOv3-tiny, YOLOv4-tiny, YOLOv5s and YOLOv7-tiny, and significantly lower than that of YOLOv8s. In addition, the detection heatmaps of YOLOv7-tiny and YOLO-LM demonstrate these optimization operations contribute to the enhanced performance of maturity detection for Camellia oleifera fruits within natural orchard environments. Overall, the YOLO-LM model can be used for the maturity detection of Camellia oleifera fruits in orchards, which is expected to provide theoretical reference for orchard yield estimation, growth monitoring, planting management optimization and the development of Camellia oleifera selective harvesting robots.

Automatic Method for Extracting Tree Branching Structures from a Single RGB Image

Creating automated methods for detecting branches in images is crucial for applications like harvesting robots and forest monitoring. However, the tree images encountered in real-world scenarios present significant challenges for branch detection techniques due to issues such as background interference, occlusion, and varying environmental lighting. While there has been notable progress in extracting tree trunks for specific species, research on identifying lateral branches remains limited. The primary challenges include establishing a unified mathematical representation for multi-level branch structures, conducting quantitative analyses, and the absence of suitable datasets to facilitate the development of effective models. This study addresses these challenges by creating a dataset encompassing various tree species, developing annotation tools for multi-level branch structure labeling, designing branch vector representations and quantitative metrics. Building on this foundation, the study introduces an automatic extraction model for multi-level branch structures that utilizes ResNet and a self-attention mechanism, along with a tailored loss function for branch extraction tasks. The study evaluated several model variants through both qualitative and quantitative experiments. Results from different tree images demonstrate that the final model can accurately identify the trunk structure and effectively extract detailed lateral branch structures, offering a valuable tool for applications in this area.

Design and development of a peduncle-holding end effector for robotic harvesting of mango

Design and development of a peduncle-holding end effector for robotic harvesting of mango

Image Analysis Artificial Intelligence Technologies for Plant Phenotyping: Current State of the Art

Modern agriculture is characterized by the use of smart technology and precision agriculture to monitor crops in real time. The technologies enhance total yields by identifying requirements based on environmental conditions. Plant phenotyping is used in solving problems of basic science and allows scientists to characterize crops and select the best genotypes for breeding, hence eliminating manual and laborious methods. Additionally, plant phenotyping is useful in solving problems such as identifying subtle differences or complex quantitative trait locus (QTL) mapping which are impossible to solve using conventional methods. This review article examines the latest developments in image analysis for plant phenotyping using AI, 2D, and 3D image reconstruction techniques by limiting literature from 2020. The article collects data from 84 current studies and showcases novel applications of plant phenotyping in image analysis using various technologies. AI algorithms are showcased in predicting issues expected during the growth cycles of lettuce plants, predicting yields of soybeans in different climates and growth conditions, and identifying high-yielding genotypes to improve yields. The use of high throughput analysis techniques also facilitates monitoring crop canopies for different genotypes, root phenotyping, and late-time harvesting of crops and weeds. The high throughput image analysis methods are also combined with AI to guide phenotyping applications, leading to higher accuracy than cases that consider either method. Finally, 3D reconstruction and a combination with AI are showcased to undertake different operations in applications involving automated robotic harvesting. Future research directions are showcased where the uptake of smartphone-based AI phenotyping and the use of time series and ML methods are recommended.

Research on the Jet Distance Enhancement Device for Blueberry Harvesting Robots Based on the Dual-Ring Model

In China, most blueberry varieties are characterized by tightly clustered fruits, which pose challenges for achieving precise and non-destructive automated harvesting. This complexity limits the design of robots for this task. Therefore, this paper proposes adding a jetting step during harvesting to separate fruit clusters and increase the operational space for mechanical claws. First, a combined approach of flow field analysis and pressure-sensitive experiments was employed to establish design criteria for the number, diameter, and inclination angle parameters of two types of nozzles: flat tip and round tip. Furthermore, fruit was introduced, and a fluid–structure coupling method was employed to calculate the deformation of fruit stems. Simultaneously, a mechanical analysis was conducted to quantify the relationship between jet characteristics and separation gaps. Simulation and pressure-sensitive experiments show that as the number of holes increases and their diameter decreases, the nozzle’s convergence becomes stronger. The greater the inclination angle of the circular nozzle holes, the more the gas diverges. The analysis of the output characteristics of the working section indicates that the 8-hole 40° round nozzle is the optimal solution. At an air compressor working pressure of 0.5 MPa, force analysis and simulation results both show that it can increase the picking space for the mechanical claw by about 5–7 mm without damaging the blueberries in the jet area. The final field experiments show that the mean distance for Type I (mature fruit) is 5.41 mm, for Type II (red fruit) is 6.42 mm, and for Type III (green fruit) is 5.43 mm. The short and curved stems of the green fruit are less effective, but the minimum distance of 4.71 mm is greater than the claw wall thickness, meeting the design requirements.

Enhanced precision in greenhouse tomato recognition and localization

The core challenge in realizing automatic tomato harvesting in greenhouse environments lies in the precise identification and localization of the fruits. This paper introduces a comprehensive approach based on an improved YOLOv5 detection algorithm and optimized binocular stereo vision technology. Firstly, by introducing the C3-Transformer Encoder (CTM) structure and Bidirectional Feature Pyramid Network (Bi-FPN), this study enhanced the model’s ability to recognize tomatoes, especially under complex backgrounds and occlusion conditions. After field testing, the mAP50 accuracy reached 97.1%, an increase of 1.2 percentage points, enhancing detection precision. In addition, the ZED binocular camera was used, and the census stereo matching algorithm was optimized, significantly reducing disparity errors, thereby improving the accuracy of depth information. This allows the model to accurately calculate the three-dimensional spatial position of tomatoes obscured by branches and leaves, greatly improving the efficiency of the harvesting robot. Through field debugging verification with the harvesting robot, the method proposed in this study has shown high accuracy and reliability in the recognition and localization of tomatoes in complex greenhouse environments.

Relative Position Detection of Clustered Tomatoes Based on BlendMask-BiFPN

In robotic harvesting, maneuvering around obstacles to position manipulators is challenging, especially in unstructured environments. This study proposes a method to detect the relative position of tomato bunches to the main stem position using the BlendMask-BiFPN algorithm. Initial comparative tests between full-stem and partial-stem labelling strategies revealed that the latter produced more complete peduncle masks, which guided our choice for subsequent experiments. Significant modifications to the BlendMask algorithm included the integration of a ResNet-101-BiFPN backbone, which improved the feature fusion network of the model. The revised model demonstrated high efficiency in pinpointing the relative positions of clustered tomatoes, achieving 91.3 % ARmask 50 and 84.8 % APmask 50 for the detection of tomato bunches. Comparisons with Mask RCNN, YOLACT, YOLACT++, and YOLOv8 showed that the BlendMask-BiFPN model outperforms these alternatives, suggesting its potential for more effective robotic harvesting in complex agricultural scenarios.

A Novel Fusion Perception Algorithm of Tree Branch/Trunk and Apple for Harvesting Robot Based on Improved YOLOv8s

Aiming to accurately identify apple targets and achieve segmentation and the extraction of branch and trunk areas of apple trees, providing visual guidance for a picking robot to actively adjust its posture to avoid branch trunks for obstacle avoidance fruit picking, the spindle-shaped fruit trees, which are widely planted in standard modern apple orchards, were focused on, and an algorithm for apple tree fruit detection and branch segmentation for picking robots was proposed based on an improved YOLOv8s model design. Firstly, image data of spindle-shaped fruit trees in modern apple orchards were collected, and annotations of object detection and pixel-level segmentation were conducted on the data. Training set data were then augmented to improve the generalization performance of the apple detection and branch segmentation algorithm. Secondly, the original YOLOv8s network architecture’s design was improved by embedding the SE module visual attention mechanism after the C2f module of the YOLOv8s Backbone network architecture. Finally, the dynamic snake convolution module was embedded into the Neck structure of the YOLOv8s network architecture to better extract feature information of different apple targets and tree branches. The experimental results showed that the proposed improved algorithm can effectively recognize apple targets in images and segment tree branches and trunks. For apple recognition, the precision was 99.6%, the recall was 96.8%, and the mAP value was 98.3%. The mAP value for branch and trunk segmentation was 81.6%. The proposed improved YOLOv8s algorithm design was compared with the original YOLOv8s, YOLOv8n, and YOLOv5s algorithms for the recognition of apple targets and segmentation of tree branches and trunks on test set images. The experimental results showed that compared with the other three algorithms, the proposed algorithm increased the mAP for apple recognition by 1.5%, 2.3%, and 6%, respectively. The mAP for tree branch and trunk segmentation was increased by 3.7%, 15.4%, and 24.4%, respectively. The proposed detection and segmentation algorithm for apple tree fruits, branches, and trunks is of great significance for ensuring the success rate of robot harvesting, which can provide technical support for the development of an intelligent apple harvesting robot.