Harvesting Robot Research Articles

Imitation Learning from a Single Demonstration Leveraging Vector Quantization for Robotic Harvesting

The ability of robots to tackle complex non-repetitive tasks will be key in bringing a new level of automation in agricultural applications still involving labor-intensive, menial, and physically demanding activities due to high cognitive requirements. Harvesting is one such example as it requires a combination of motions which can generally be broken down into a visual servoing and a manipulation phase, with the latter often being straightforward to pre-program. In this work, we focus on the task of fresh mushroom harvesting which is still conducted manually by human pickers due to its high complexity. A key challenge is to enable harvesting with low-cost hardware and mechanical systems, such as soft grippers which present additional challenges compared to their rigid counterparts. We devise an Imitation Learning model pipeline utilizing Vector Quantization to learn quantized embeddings directly from visual inputs. We test this approach in a realistic environment designed based on recordings of human experts harvesting real mushrooms. Our models can control a cartesian robot with a soft, pneumatically actuated gripper to successfully replicate the mushroom outrooting sequence. We achieve 100% success in picking mushrooms among distractors with less than 20 min of data collection comprising a single expert demonstration and auxiliary, non-expert, trajectories. The entire model pipeline requires less than 40 min of training on a single A4000 GPU and approx. 20 ms for inference on a standard laptop GPU.

Research on the visual location method for strawberry picking points under complex conditions based on composite models.

Strawberry, being an important economic crop, requires a large amount of human labor for harvesting operations. Efficient and non-destructive harvesting by strawberry harvesting robots requires the precise location of the picking points. Current algorithms for locating picking points encounter significant issues with location errors and minimal effective information in complex situations. To improve the accuracy of the location of picking points, this study proposes a visual location method based on composite models. This method employs object detection and instance segmentation models to detect fruits and segment peduncles sequentially, thereby enabling the identification of picking points and inclination on the peduncle. Different object detection algorithms and instance segmentation models were validated to explore the optimal model combination, and the Convolutional Block Attention Module (CBAM) was integrated into YOLOv8s-seg to construct YOLOv8s-seg-CBAM. Test results show that the composite model built with YOLOv8s and YOLOv8s-seg-CBAM achieved a peduncle detection accuracy of 86.2%, with an inference time of 30.6 ms per image. The picking point visual location method based on YOLOv8s and YOLOv8s-seg-CBAM composite models can better balance accuracy and efficiency and can provide more accurate guidance for automated harvesting. © 2024 Society of Chemical Industry.

Design and Experiment of Nighttime Greenhouse Tomato Harvesting Robot

In response to the issue of high tomato yield, low efficiency in harvesting tomatoes grown in greenhouses, and low recognition accuracy of nighttime harvesting robots, a design was developed and a robotic system was created specifically for nighttime greenhouse tomato harvesting. The robot employs a vision system and YOLOv5+HSV fusion algorithm to recognize and locate tomatoes. It then transmits this information to the robotic arm. By coordinating the visual system, the robotic arm, the end effector, and the lifting mechanism, the robot accurately picks ripe tomatoes. The robot was subjected to simulated field tests for visual recognition and harvesting, both during daytime and nighttime conditions. The results showed that the success rate of nighttime harvesting was slightly lower than during the daytime but remained at a relatively high level. The daytime harvesting success rate and the average time to pick a single fruit were 87.78% and 15.99 seconds, respectively. The nighttime harvesting success rate and the average time to pick a single fruit were 87.55% and 17.26 seconds, respectively. This approach effectively improves the recognition accuracy and harvesting speed of the harvesting robot, reducing damage to tomatoes during harvesting, and addresses the issues of supplementary lighting and image noise reduction for nighttime harvesting robots.

An Improved YOLOv8 Model for Lotus Seedpod Instance Segmentation in the Lotus Pond Environment

Lotus seedpod maturity detection and segmentation in pond environments play a significant role in yield prediction and picking pose estimation for lotus seedpods. However, it is a great challenge to accurately detect and segment lotus seedpods due to insignificant phenotypic differences between the adjacent maturity, changing illumination, overlap, and occlusion of lotus seedpods. The existing research pays attention to lotus seedpod detection while ignoring maturity detection and segmentation problems. Therefore, a semantic segmentation dataset of lotus seedpods was created, where a copy-and-paste data augmentation tool was employed to eliminate the class-imbalanced problem and improve model generalization ability. Afterwards, an improved YOLOv8-seg model was proposed to detect and segment the maturity of lotus seedpods. In the model, the convolutional block attention module (CBAM) was embedded in the neck network to extract distinguished features of different maturity stages with negligible computation cost. Wise-Intersection over Union (WIoU) regression loss function was adopted to refine the regression inference bias and improve the bounding box prediction accuracy. The experimental results showed that the proposed YOLOv8-seg model provides an effective method for “ripe” and “overripe” lotus seedpod detection and instance segmentation, where the mean average precision of segmentation mask (mAPmask) reaches 97.4% and 98.6%, respectively. In addition, the improved YOLOv8-seg exhibits high robustness and adaptability to complex illumination in a challenging environment. Comparative experiments were conducted using the proposed YOLOv8-seg and other state-of-the-art instance segmentation methods. The results showed that the improved model is superior to the Mask R-CNN and YOLACT models, with recall, precision, mAPbox and mAPmask being 96.5%, 94.3%, 97.8%, and 98%, respectively. The average running time and weight size of the proposed model are 25.9 ms and 7.4 M, respectively. The proposed model obtained the highest mAP for lotus seedpod maturity detection and segmentation while maintaining an appropriate model size and speed. Furthermore, based on the obtained segmentation model, 3D visualization of the lotus pond scene is performed, and cloud point of lotus seedpods is generated, which provides a theoretical foundation for robot harvesting in the lotus pond.

Assessing a multi-camera system to enhance fruit visibility for robotic harvesting in a V-trellised apple orchard

Accurate detection and localization of fruits within the canopy are crucial for various tasks, such as perception for robotic harvesters, vision-based yield estimation, and early disease detection. However, complex canopy structures render fruit detection challenging due to obstructions from leaves, branches, and other fruits. This study assesses the efficacy of utilizing multiple cameras at varying elevations and azimuthal angles to enhance fruit visibility. We used a V-trellised apple orchard as a case study, where four cameras were employed; the first was oriented normally to the fruit canopy, while three others maintained a fixed position relative to the first but with variable orientation. Adjusting elevation and azimuthal angles revealed that, in the region of interest of the canopy, a single camera whose optical axis was perpendicular to the canopy could detect up to 88.3% of the fruits detected by four cameras. Adding one more camera – for a total of two cameras – increased the detection rate up to 97.5% of the four-camera detection rate. Also, the experimental results showed that three cameras could detect between 96.0% to 99.7% of fruits compared to four cameras under the proposed camera configuration. Hence, the utility of adding a third or fourth camera would need to be considered carefully, given the added cost and complexity. The results of this study are a step toward exploring and optimizing multiple-camera perception systems for orchard operations, in particular, robotic harvesters.

FF3D: A Rapid and Accurate 3D Fruit Detector for Robotic Harvesting.

This study presents the Fast Fruit 3D Detector (FF3D), a novel framework that contains a 3D neural network for fruit detection and an anisotropic Gaussian-based next-best view estimator. The proposed one-stage 3D detector, which utilizes an end-to-end 3D detection network, shows superior accuracy and robustness compared to traditional 2D methods. The core of the FF3D is a 3D object detection network based on a 3D convolutional neural network (3D CNN) followed by an anisotropic Gaussian-based next-best view estimation module. The innovative architecture combines point cloud feature extraction and object detection tasks, achieving accurate real-time fruit localization. The model is trained on a large-scale 3D fruit dataset and contains data collected from an apple orchard. Additionally, the proposed next-best view estimator improves accuracy and lowers the collision risk for grasping. Thorough assessments on the test set and in a simulated environment validate the efficacy of our FF3D. The experimental results show an AP of 76.3%, an AR of 92.3%, and an average Euclidean distance error of less than 6.2 mm, highlighting the framework's potential to overcome challenges in orchard environments.

Maturity discrimination of tobacco leaves for tobacco harvesting robots based on a Multi-Scale branch attention neural network

Traditional machine vision methods are susceptible to environmental influences, low recognition accuracy, and poor real-time performance in identifying tobacco leaf maturity. Currently, the maturity of tobacco leaves heavily relies on manual observation, and existing research on tobacco maturity discrimination focuses on standard light sources and standard background, which difficult to be applied to tobacco harvesting robots in natural environment. To tackle this problem, a maturity discrimination method based on the branch attention mechanism is proposed. Firstly, the region of interest in the extracted tobacco leaf images is obtained as a foundation for subsequent image processing. Then, a deep learning network model called the Multi-Scale Branch Attention Network (MSBANet) is designed for identification of tobacco leaf maturity. The effectiveness of MSBANet stems from its newly designed modules for multi-scale information extraction and branch attention. These modules help overcome the impact of lighting conditions and enable the extraction of more comprehensive and informative feature representations, capturing the relationship between phenotype and maturity. On the self-constructed dataset of tobacco leaf maturity (color images are 576 × 768 pixels), MSBANet-L achieves 89.6 % accuracy with 7.1GFLOPs and MSBANet-S achieves 88.2 % accuracy with 1.7 GFLOPs. When compared with widely used BP and MobileNet, as well as state-of-the-art structures such as YOLOv7-tiny-SiLU and FasterNet-T0, MSBANet demonstrates excellent performance. This method preliminarily explores the maturity discrimination of tobacco leaves in actual environments and shows a potential for development of tobacco harvesting robots.

A selective harvesting robot for cherry tomatoes: Design, development, field evaluation analysis

AbstractWith the aging population and increasing labor costs, traditional manual harvesting methods have become less economically efficient. Consequently, research into fully automated harvesting using selective harvesting robots for cherry tomatoes has become a hot topic. However, most of the current research is focused on individual harvesting of large tomatoes, and there is less research on the development of complete systems for harvesting cherry tomatoes in clusters. The purpose of this study is to develop a harvesting robot system capable of picking tomato clusters by cutting their fruit‐bearing pedicels and to evaluate the robot prototype in real greenhouse environments. First, to enhance the grasping stability, a novel end‐effector was designed. This end‐effector utilizes a cam mechanism to achieve asynchronous actions of cutting and grasping with only one power source. Subsequently, a visual perception system was developed to locate the cutting points of the pedicels. This system is divided into two parts: rough positioning of the fruits in the far‐range view and accurate positioning of the cutting points of the pedicels in the close‐range view. Furthermore, it possesses the capability to adaptively infer the approaching pose of the end‐effector based on point cloud features extracted from fruit‐bearing pedicels and stems. Finally, a prototype of the tomato‐harvesting robot was assembled for field trials. The test results demonstrate that in tomato clusters with unobstructed pedicels, the localization success rates for the cutting points were 88.5% and 83.7% in the two greenhouses, respectively, while the harvesting success rates reached 57.7% and 55.4%, respectively. The average cycle time to harvest a tomato cluster was 24 s. The experimental results prove the potential for commercial application of the developed tomato‐harvesting robot and through the analysis of failure cases, discuss directions for future work.

3D Camera and Single-Point Laser Sensor Integration for Apple Localization in Spindle-Type Orchard Systems.

Accurate localization of apples is the key factor that determines a successful harvesting cycle in the automation of apple harvesting for unmanned operations. In this regard, accurate depth sensing or positional information of apples is required for harvesting apples based on robotic systems, which is challenging in outdoor environments because of uneven light variations when using 3D cameras for the localization of apples. Therefore, this research attempted to overcome the effect of light variations for the 3D cameras during outdoor apple harvesting operations. Thus, integrated single-point laser sensors for the localization of apples using a state-of-the-art model, the EfficientDet object detection algorithm with an mAP@0.5 of 0.775 were used in this study. In the experiments, a RealSense D455f RGB-D camera was integrated with a single-point laser ranging sensor utilized to obtain precise apple localization coordinates for implementation in a harvesting robot. The single-point laser range sensor was attached to two servo motors capable of moving the center position of the detected apples based on the detection ID generated by the DeepSORT (online real-time tracking) algorithm. The experiments were conducted under indoor and outdoor conditions in a spindle-type apple orchard artificial architecture by mounting the combined sensor system behind a four-wheel tractor. The localization coordinates were compared between the RGB-D camera depth values and the combined sensor system under different light conditions. The results show that the root-mean-square error (RMSE) values of the RGB-D camera depth and integrated sensor mechanism varied from 3.91 to 8.36 cm and from 1.62 to 2.13 cm under 476~600 lx to 1023~1100 × 100 lx light conditions, respectively. The integrated sensor system can be used for an apple harvesting robotic manipulator with a positional accuracy of ±2 cm, except for some apples that were occluded due to leaves and branches. Further research will be carried out using changes in the position of the integrated system for recognition of the affected apples for harvesting operations.

Excluding sunscald from long-term storage of ‘Granny Smith’ apples

Sunscald is a postharvest apple and pear peel disorder primarily caused by excessive light exposure during the growing season. Symptoms are not apparent at-harvest but develop during storage as superficial brown or gray discolored patches on the exposed portion of the apple. A novel protocol using hyperspectral imaging (HSI) was developed to identify apples at- harvest that are more likely to develop the disorder and reduce sunscald in the cold chain by eliminating at-risk apples from long-term storage. PLS-DA and LOGISTIC models and an index summarizing the chlorophyll to carotenoid ratio were tested for this purpose using two populations selected for different sun exposure levels and four populations harvested from whole trees. The accuracy of models varied seasonally. However, detection of sunburn and estimation of sunscald risk based on index values was a more consistent approach. Summarizing peel area according to this index was the basis for categorizing whole apples as rejected (sunburned), high (sunscald) risk, or low-risk. Categorization segregated apples nearly entirely into these categories with only an insignificant frequency of misclassification. The frequency of high-risk apples without sunburn symptoms in the populations harvested from whole trees fell below 20 %, although actual sunscald incidence within this category ranged from 55 % to 81 % among populations harvested from whole trees, highlighting that other factors beyond elevated sun exposure contribute to sunscald incidence. The classification model for low and high-risk categories had an accuracy of 83–93 % in four independent populations, with 99.9 % sensitivity to classify non-sunscalded fruit into the low-risk category. Sorting by sunburn presence and sunscald risk allows for apples that have sunburn symptoms to be excluded and those that are marketable but most likely to develop sunscald to be pushed to market rather than storage. With the appropriate source/sensor array, this approach is readily adaptable to sorting lines or even robotic harvesters. Challenges include adapting this approach for other apple and pear cultivars and extending capabilities for additional uses, including sorting fruit into additional categories representative of relative sunlight exposure.

Fusion of fruit image processing and deep learning: a study on identification of citrus ripeness based on R-LBP algorithm and YOLO-CIT model.

Citrus fruits are extensively cultivated fruits with high nutritional value. The identification of distinct ripeness stages in citrus fruits plays a crucial role in guiding the planning of harvesting paths for citrus-picking robots and facilitating yield estimations in orchards. However, challenges arise in the identification of citrus fruit ripeness due to the similarity in color between green unripe citrus fruits and tree leaves, leading to an omission in identification. Additionally, the resemblance between partially ripe, orange-green interspersed fruits and fully ripe fruits poses a risk of misidentification, further complicating the identification of citrus fruit ripeness. This study proposed the YOLO-CIT (You Only Look Once-Citrus) model and integrated an innovative R-LBP (Roughness-Local Binary Pattern) method to accurately identify citrus fruits at distinct ripeness stages. The R-LBP algorithm, an extension of the LBP algorithm, enhances the texture features of citrus fruits at distinct ripeness stages by calculating the coefficient of variation in grayscale values of pixels within a certain range in different directions around the target pixel. The C3 model embedded by the CBAM (Convolutional Block Attention Module) replaced the original backbone network of the YOLOv5s model to form the backbone of the YOLO-CIT model. Instead of traditional convolution, Ghostconv is utilized by the neck network of the YOLO-CIT model. The fruit segment of citrus in the original citrus images processed by the R-LBP algorithm is combined with the background segment of the citrus images after grayscale processing to construct synthetic images, which are subsequently added to the training dataset. The experiment showed that the R-LBP algorithm is capable of amplifying the texture features among citrus fruits at distinct ripeness stages. The YOLO-CIT model combined with the R-LBP algorithm has a Precision of 88.13%, a Recall of 93.16%, an F1 score of 90.89, a mAP@0.5 of 85.88%, and 6.1ms of average detection speed for citrus fruit ripeness identification in complex environments. The model demonstrates the capability to accurately and swiftly identify citrus fruits at distinct ripeness stages in real-world environments, effectively guiding the determination of picking targets and path planning for harvesting robots.

Weed Detection and Classification with Computer Vision Using a Limited Image Dataset

In agriculture, as precision farming increasingly employs robots to monitor crops, the use of weeding and harvesting robots is expanding the need for computer vision. Currently, most researchers and companies address these computer vision tasks with CNN-based deep learning. This technology requires large datasets of plant and weed images labeled by experts, as well as substantial computational resources. However, traditional feature-based approaches to computer vision can extract meaningful parameters and achieve comparably good classification results with only a tenth of the dataset size. This study presents these methods and seeks to determine the minimum number of training images required to achieve reliable classification. We tested the classification results with 5, 10, 20, 40, 80, and 160 images per weed type in a four-class classification system. We extracted shape features, distance transformation features, color histograms, and texture features. Each type of feature was tested individually and in various combinations to determine the best results. Using six types of classifiers, we achieved a 94.56% recall rate with 160 images per weed. Better results were obtained with more training images and a greater variety of features.

The selection of appropriate rootstock and training system towards sustainable production of stone fruits

In stone fruit species, such as peach, cherry, apricot and plum labour requirement is important ranging from 45% to 71% of the total cost of production. In this sense, combining size controlling rootstocks as Rootpac® series, Isthara® and others in peach and plum, or Gisela® and Corette® series in cherry, with intensive bidimensional training systems based on central axis or bi axis resulted in earlier and higher yields, compared with the traditional open vase associated with vigorous rootstocks. The association of these intensive orchards with bidimensional canopies and good accessibility to canopy by labour and machines, is essential to reduce the cost of production. These canopies allow an efficient use of labour and mechanization. Regarding peach, under Ebro Valley (Spain) conditions, the reduction on total cost of production ranged from 15% to 20% and the harvest rate increased by 41%, enhancing also fruit quality due to better light distribution within the tree canopy. In cherry the increase of harvest rate increased by 72% when UFO was compared with the traditional open center. Regarding what in Washington State (US). With European and Japanese varieties, intensification by using Rootpac®20 rootstock and super high density plantings (SHD), allowed to get more than 40 t ha-1 in the third year in Ebro Valley (Spain). With ‘Agen’ European plum, using the same rootstock, under Chilean conditions training trees in a small hedge (SHD) allowed the full mechanization of the harvest by using over-the-row harvester or pedestrian harvest for the needs of fresh market. In this case, manual harvest efficiency increased by 49% compared with the traditional open vase. In all species, the use of size controlling rootstocks offers different options to growers when selecting the training systems concerning skilled labour requirements to train the trees in the first years. Thus, an increase was recorded when the number of axes increased from central axis to bi-, tri- and multi-axes. Furthermore, all these planar canopies were better adapted to the application of precision technologies based on artificial vision (AV) for monitoring pest and diseases, neural network and AV for fruit counting, yield forecasting and robotic harvesting.

Field Performance of a Dual Arm Robotic System for Efficient Tomato Harvesting

The robot device that is being addressed in this research has two arms: one for picking the fruit and the other for chopping it. The arms find and locate pods with the help of a complex vision system that employs cameras. In this human-robot workflow, the operator chooses the tomatoes they want picked, and then the robot does the actual picking. The robot management and communication system use the EtherCAT bus to create a link with the graphical user interface (GUI), enabling human administration and control. The objective of this project is to create and assess a robotic system for harvesting tomatoes, equipped with dual arms. This system incorporates a mobile model equipped with two robotic arms and an end effector to enhance the efficiency of tomato harvesting. The system uses a GUI to enhance interaction between the robot and the human operator. Additionally, it employs a vision model to streamline the process of fruit detection. Findings from this study demonstrate that HMI may significantly improve the accuracy of tomato harvesting robots. Finally, there were some difficulties in developing 3D models because this study included outdoor experiments.

Autonomous navigation system for greenhouse tomato picking robots based on laser SLAM

Aiming at the specific operation path cruising problem of greenhouse tomato picking robots, this paper designs a laser SLAM (Simultaneous Localization And Mapping)-based autonomous navigation system for greenhouse tomato picking robots.The autonomous navigation system utilizes LiDAR to detect the greenhouse environment, creates an environment map with the Gmapping algorithm employing particle filtering, locates the robot with the AMCL (Adaptive Monte Carlo Localization) algorithm, devises the robot's path for picking tasks with the A* algorithm, and dynamically navigates around obstacles during operations with the DWA (Dynamic Window Approach) algorithm. The greenhouse experiment revealed that the robot's average deviance for both directions (X and Y) is 2.87 and 0.80 centimeters, respectively, when traveling a 320 centimeter path. On the other hand, its average variation across the directions of X and Y is 1.48 versus 1.08 centimeters when it passes through a 40 centimeter zone. The robot exhibits averaged variations of 1.73 and 2.24 millimeters in the X and Y axes, each, when navigating a 40-centimeter-radius turn. Furthermore, the robot exhibits average variations of 1.12 centimeters in the X and 0.78 centimeters in the Y directions when it follows a certain "S"-shaped course. Based on the fulfillment of prescribed path cruising, the system has a high level of navigation and positioning accuracy, better meeting the requirements for greenhouse tomato harvesting robot direction and placement.

Lightweight-Improved YOLOv5s Model for Grape Fruit and Stem Recognition

Mechanized harvesting is the key technology to solving the high cost and low efficiency of manual harvesting, and the key to realizing mechanized harvesting lies in the accurate and fast identification and localization of targets. In this paper, a lightweight YOLOv5s model is improved for efficiently identifying grape fruits and stems. On the one hand, it improves the CSP module in YOLOv5s using the Ghost module, reducing model parameters through ghost feature maps and cost-effective linear operations. On the other hand, it replaces traditional convolutions with deep convolutions to further reduce the model’s computational load. The model is trained on datasets under different environments (normal light, low light, strong light, noise) to enhance the model’s generalization and robustness. The model is applied to the recognition of grape fruits and stems, and the experimental results show that the overall accuracy, recall rate, mAP, and F1 score of the model are 96.8%, 97.7%, 98.6%, and 97.2% respectively. The average detection time on a GPU is 4.5 ms, with a frame rate of 221 FPS, and the weight size generated during training is 5.8 MB. Compared to the original YOLOv5s, YOLOv5m, YOLOv5l, and YOLOv5x models under the specific orchard environment of a grape greenhouse, the proposed model improves accuracy by 1%, decreases the recall rate by 0.2%, increases the F1 score by 0.4%, and maintains the same mAP. In terms of weight size, it is reduced by 61.1% compared to the original model, and is only 1.8% and 5.5% of the Faster-RCNN and SSD models, respectively. The FPS is increased by 43.5% compared to the original model, and is 11.05 times and 8.84 times that of the Faster-RCNN and SSD models, respectively. On a CPU, the average detection time is 23.9 ms, with a frame rate of 41.9 FPS, representing a 31% improvement over the original model. The test results demonstrate that the lightweight-improved YOLOv5s model proposed in the study, while maintaining accuracy, significantly reduces the model size, enhances recognition speed, and can provide fast and accurate identification and localization for robotic harvesting.

Improved Real-Time Models for Object Detection and Instance Segmentation for Agaricus bisporus Segmentation and Localization System Using RGB-D Panoramic Stitching Images

The segmentation and localization of Agaricus bisporus is a precondition for its automatic harvesting. A. bisporus growth clusters can present challenges for precise localization and segmentation because of adhesion and overlapping. A low-cost image stitching system is presented in this research, utilizing a quick stitching method with disparity correction to produce high-precision panoramic dual-modal fusion images. An enhanced technique called Real-Time Models for Object Detection and Instance Segmentation (RTMDet-Ins) is suggested. This approach utilizes SimAM Attention Module’s (SimAM) global attention mechanism and the lightweight feature fusion module Space-to-depth Progressive Asymmetric Feature Pyramid Network (SPD-PAFPN) to improve the detection capabilities for hidden A. bisporus. It efficiently deals with challenges related to intricate segmentation and inaccurate localization in complex obstacles and adhesion scenarios. The technology has been verified by 96 data sets collected on a self-designed fully automatic harvesting robot platform. Statistical analysis shows that the worldwide stitching error is below 2 mm in the area of 1200 mm × 400 mm. The segmentation method demonstrates an overall precision of 98.64%. The planar mean positioning error is merely 0.31%. The method promoted in this research demonstrates improved segmentation and localization accuracy in a challenging harvesting setting, enabling efficient autonomous harvesting of A. bisporus.

SDC-DeepLabv3+: Lightweight and Precise Localization Algorithm for Safflower-Harvesting Robots.

Harvesting robots had difficulty extracting filament phenotypes for small, numerous filaments, heavy cross-obscuration, and similar phenotypic characteristics with organs. Robots experience difficulty in localizing under near-colored backgrounds and fuzzy contour features. It cannot accurately harvest filaments for robots. Therefore, a method for detecting and locating filament picking points based on an improved DeepLabv3+ algorithm is proposed in this study. A lightweight network structure, ShuffletNetV2, was used to replace the backbone network Xception of the traditional DeepLabv3+. Convolutional branches for 3 different sampling rates were added to extract information on the safflower features under the receptive field. Convolutional block attention was incorporated into feature extraction at the coding and decoding layers to solve the interference problem of the near-color background in the feature-fusion process. Then, using the region of interest of the safflower branch obtained by the improved DeepLabv3+, an algorithm for filament picking-point localization was designed based on barycenter projection. The tests demonstrated that this method was capable of accurately localizing the filament. The mean pixel accuracy and mean intersection over union of the improved DeepLabv3+ were 95.84% and 96.87%, respectively. The detection rate and weights file size required were superior to those of other algorithms. In the localization test, the depth-measurement distance between the depth camera and target safflower filament was 450 to 510 mm, which minimized the visual-localization error. The average localization and picking success rates were 92.50% and 90.83%, respectively. The results show that the proposed localization method offers a viable approach for accurate harvesting localization.

Tomato pose estimation using the association of tomato body and sepal

In facility horticulture smart farms, harvesting robotic systems to automate harvesting tasks are challenging due to the complex environment, irregular growth and fruits pose. To harvest a fruits, the vision information required by a harvesting robot is accurate position and pose. Especially, fruit pose information is essential for planning paths to avoid damaging stems, leaves, branches, and obstacles, preventing damage to the fruit and harvesting robots, and planning efficient harvest sequences. This paper presents a method that uses information from tomato sepals to estimate the orientation of tomatoes, which are a commonly grown crop in horticulture. First, we train a YOLOv8 model to detect and segment bodies and sepals of tomatoes. For robust training of the model, the training data is constructed using effective data augmentation, which synthesizes segmented foreground objects and inserting them into the background. Then, for finding association between the body and sepal of a tomato, we apply IoU score based matching and the Hungarian algorithm. Consequently, we obtain point clouds for both parts using RGB-D data. Finally, we compute the center point of each object by using spherical fitting and the statistics of the point cloud, respectively. Then, we estimate the pose of the tomato as a vector between two center points of the body and sepal. To accurately and practically evaluate the proposed method, we generated ground truth data using calibration patterns in tomato greenhouse. As the experimental results show, The segmentation results show that AP50,sepal is 94.7, AP50,tomato is 96.3, and mAP is 61.5. The result of the pose estimation for the pose validation dataset is a mean error angle of 6.79 ± 3.18 and angle errors of less than 10 degrees account for 87.2%. The total algorithm proposed in this paper requires about 0.038s for each tomato greenhouse image. As a result, our proposed pose estimation can be practically utilized in robot systems for tomato harvesting.

A novel jujube tree trunk and branch salient object detection method for catch-and-shake robotic visual perception

Visual perception has become a prerequisite for automated jujube harvesting robot operations under complex orchard conditions. Catch-and-Shake harvesting, as the most efficient and common harvesting method, has widely been applied on various manually operated harvesters to complete large-area jujube fruit harvesting. However, the main factors restricting the development of existing harvesters are labor shortage, high labor cost, and low operating efficiency. To address the issues, we designed a catch-and-shake harvesting robot for jujube tree trunks and branches visual perception that can provide a barrier-free catch-and-shake operation area and guide the manipulator to reach the area to complete the harvesting operation. Meanwhile, a visual perception system including tree trunks and branches detection, skeleton extraction, catch-and-shake area confirmation was presented to guide robot intelligent operations. In the visual perception system, a novel salientobjectdetectionmodel called feature intersection and fusion Transformer (FIT-Transformer) network was proposed to split branches and background to provide reference for determining safe catch-and-shake areas. Moreover, we designed a diverse feature aggregation (DFA) and an attention feature fusion module (AFFM) to strengthen feature learning capabilities and obtain robust perception models. Comparative experimental results showed that our proposed FIT-Transformer model outperformed 12 state-of-the-art (SOTA) algorithms including C2FNet, RAS, BASNet, U2Net, SCRNet, PiCANet, EDRNet, EGNet, ICONR, VST, TransSOD and ABiU_Net. Specifically, the segmentation accuracy of jujube tree trunks and branches using our method showed the satisfactory result on five evaluation indexes under natural environment (the EM, SM, WF, FM and MAE reached 0.9713, 0.8991, 0.8854, 0.8905, and 0.0302, respectively). Field experiments also proved that our method could meet the requirements of operational accuracy and real-time operations.