Fruit-picking Robots Research Articles

A Review of Perception Technologies for Berry Fruit-Picking Robots: Advantages, Disadvantages, Challenges, and Prospects

Berries are nutritious and valuable, but their thin skin, soft flesh, and fragility make harvesting and picking challenging. Manual and traditional mechanical harvesting methods are commonly used, but they are costly in labor and can damage the fruit. To overcome these challenges, it may be worth exploring alternative harvesting methods. Using berry fruit-picking robots with perception technology is a viable option to improve the efficiency of berry harvesting. This review presents an overview of the mechanisms of berry fruit-picking robots, encompassing their underlying principles, the mechanics of picking and grasping, and an examination of their structural design. The importance of perception technology during the picking process is highlighted. Then, several perception techniques commonly used by berry fruit-picking robots are described, including visual perception, tactile perception, distance measurement, and switching sensors. The methods of these four perceptual techniques used by berry-picking robots are described, and their advantages and disadvantages are analyzed. In addition, the technical characteristics of perception technologies in practical applications are analyzed and summarized, and several advanced applications of berry fruit-picking robots are presented. Finally, the challenges that perception technologies need to overcome and the prospects for overcoming these challenges are discussed.

3D positioning of Camellia oleifera fruit-grabbing points for robotic harvesting

Camellia oleifera is an oilseed crop with high economic value. The short optimum harvest period and high labour costs of C. oleifera harvesting have prompted research on intelligent robotic harvesting. This study focused on the determination of grabbing points for the robotic harvesting of C. oleifera fruits, providing a basis for the decision making of the fruit-picking robot. A relatively simple 2D convolutional neural network (CNN) and stereoscopic vision replaced the complex 3D CNN to realise the 3D positioning of the fruit. Apple datasets were used for the pretraining of the model and knowledge transfer, which shared a certain degree of similarity to C. oleifera fruit. In addition, a fully automatic coordinate conversion method has been proposed to transform the fruit position information in the image into its 3D position in the robot coordinate system. Results showed that the You Only Look Once (YOLO)v8x model trained using 1012 annotated samples achieved the highest performance for fruit detection, with mAP50 of 0.96 on the testing dataset. With knowledge transfer based on the apple datasets, YOLOv8x using few-shot learning realised a testing mAP50 of 0.95, reducing manual annotation. Moreover, the error in the 3D coordinate calculation was lower than 2.1 cm on the three axes. The proposed method provides the 3D coordinates of the grabbing point for the target fruit in the robot coordinate system, which can be transferred directly to the robot control system to execute fruit-picking actions. This dataset was published online to reproduce the results of this study.

Design of Adaptive Grippers for Fruit-Picking Robots Considering Contact Behavior

Adaptability to unstructured objects and the avoidance of target damage are critical challenges for flexible grippers in fruit-picking robots. Most existing flexible grippers have many problems in terms of control complexity, stability and cost. This paper proposes a flexible finger design method that considers contact behavior. The new approach incorporates topological design of contact targets and introduces contact stress constraints to directly obtain a flexible finger structure with low contact stress and good adaptability. The study explores the effects of design parameters, including virtual spring stiffness, volume fraction, design domain size, and discretization, on the outcomes of the flexible finger topology optimization. Two flexible finger structures were selected for comparative analysis. The experimental results verified the effectiveness of the design method and the maximum contact stress was reduced by about 70%. An adaptive two-finger gripper was developed. This design allows the gripper to achieve damage-free grasping without additional sensors and control systems. The adaptive and contact performances of the grippers with different driving modes were analyzed. Practical grasping tests were also performed, including evaluation of adaptive performance, stability, and maximum grasping weight. The results indicate that gripper 2 with flexible finger 2 excelled in contact stress and adaptive wrapping, making it well-suited for grasping unstructured and fragile objects. This paper provides valuable insights for the design and application of flexible grippers for picking robots, offering a promising solution to enhance adaptability while minimizing target damage.

Application of Image Restoration and Object Detection Techniques in Fruit-Picking Robots

This paper focuses on the problem that existing apple picking robots are unable to accurately recognize targets in complex and unstructured orchard environments. The target detection algorithm of the robot is optimized based on YOLOv5 and ResNet neural network model. First, considering the low resolution of the robot's own binocular camera, this paper adopts the RealSR model to process the image with super-resolution. Then, the YOLOv5 model is chosen as the optimized target detection algorithm, which can obtain the number, location, ripeness and quality information of apples in the image through training. Secondly, by comparing the VGG and ResNet models, which are widely used in the field of image classification, the ResNet model is finally chosen as the main body for recognizing and classifying different fruits, and its correct rate can reach more than 97%.

Strawberry Detection and Ripeness Classification Using YOLOv8+ Model and Image Processing Method

As strawberries are a widely grown cash crop, the development of strawberry fruit-picking robots for an intelligent harvesting system should match the rapid development of strawberry cultivation technology. Ripeness identification is a key step to realizing selective harvesting by strawberry fruit-picking robots. Therefore, this study proposes combining deep learning and image processing for target detection and classification of ripe strawberries. First, the YOLOv8+ model is proposed for identifying ripe and unripe strawberries and extracting ripe strawberry targets in images. The ECA attention mechanism is added to the backbone network of YOLOv8+ to improve the performance of the model, and Focal-EIOU loss is used in loss function to solve the problem of imbalance between easy- and difficult-to-classify samples. Second, the centerline of the ripe strawberries is extracted, and the red pixels in the centerline of the ripe strawberries are counted according to the H-channel of their hue, saturation, and value (HSV). The percentage of red pixels in the centerline is calculated as a new parameter to quantify ripeness, and the ripe strawberries are classified as either fully ripe strawberries or not fully ripe strawberries. The results show that the improved YOLOv8+ model can accurately and comprehensively identify whether the strawberries are ripe or not, and the mAP50 curve steadily increases and converges to a relatively high value, with an accuracy of 97.81%, a recall of 96.36%, and an F1 score of 97.07. The accuracy of the image processing method for classifying ripe strawberries was 91.91%, FPR was 5.03%, and FNR was 14.28%. This study demonstrates the program’s ability to quickly and accurately identify strawberries at different stages of ripeness in a facility environment, which can provide guidance for selective picking by subsequent fruit-picking robots.

Dynamic visual servo control methods for continuous operation of a fruit harvesting robot working throughout an orchard

Fruit-picking robots are crucial for achieving efficient orchard harvesting. To genuinely meet the commercial production needs of farmers, the new generation of fruit-picking robots must be capable of demonstrating complete and continuous observation, movement, and picking behaviors throughout complex orchards, akin to real human employees. This poses systematic challenges, as many prior researches have focused solely on a part of the continuous operation of the entire orchard, such as fruit positioning, navigation, path planning, or grasping. These isolated basic functions are important but insufficient for fulfilling operational requirements on a macro scale and continuous situation. Developing an efficient control method for each basic module and constructing their internal coordination is vital for transitioning a harvesting robot from a functional prototype to a practical machine. In this context, this study tackles the visual servo control problem for efficient locomotion, picking, and their seamless integration. A set of vision algorithms for locomotion destination estimation, real-time self-positioning, and dynamic harvesting is proposed. Additionally, a solid coordination mechanism for continuous locomotion and picking behavior is established. Each method offers distinct advantages, such as improved accuracy, adaptability to varying conditions, and enhanced picking efficiency, enabling the robot to operate autonomously and continuously. Comprehensive field experiments validated the soundness of the methods. The primary contribution of this study lies in addressing the challenge of continuous operation in an entire orchard as a systematic problem and providing new insights into control methods for the future development of highly autonomous, practical, and user-oriented fruit harvesting systems.

Image Recognition for Fruit-Picking Robots

This summary presents the solutions to the five questions posed in the competition regarding apple image recognition. The objective is to establish a robust apple image recognition model that accurately counts the number of apples, estimates their positions, determines their maturity state, calculates their masses, and recognizes them based on provided image datasets. The specific answers to each question are as follows: Using the image dataset of harvest-ready apples provided in Attachment 1, an image feature extraction method is employed to establish a mathematical model for counting the number of apples in each image. The distribution of all apples in Attachment 1 is then depicted in a histogram. The precise number of apples in each image is determined, enabling a comprehensive understanding of the dataset. Utilizing the image dataset of harvest-ready apples provided in Attachment 1, the positions of the apples in each image are identified. The left bottom corner of the image serves as the coordinate origin, and a two-dimensional scatter diagram is created, illustrating the geometric coordinates of all apples in Attachment 1. This analysis facilitates the visualization of the spatial distribution of apples within the dataset. Based on the image dataset of harvest-ready apples provided in Attachment 1, a mathematical model is developed to assess the maturity state of apples in each image. By applying this model, a histogram is generated, illustrating the distribution of maturity levels across all apples in Attachment 1. This analysis provides insights into the overall maturity status of the dataset.

Enhanced rendering-based approach for improved quality of instance segmentation in detecting green gram (Vigna Rediata) pods

The emergence of Artificial Intelligence, deep learning, and current computer vision algorithms are the main contributors to innovations in the agricultural domain. The most recent detection algorithms capable of giving real-time detections at the edge nodes tackle most agricultural problems, such as disease, pest or insect detections, and maturity level detection of crops (fruits and vegetables). Modern harvesters and fruit-picking robots rely heavily on the detection capability of the algorithms used. Various detection algorithms have been proposed and used in literature, having good performance in terms of mean average precision. Still, the current agricultural systems require not only high mean average accuracy but also algorithms should have high inference speeds. The research proposes a Detectron2-based framework with PointRend (Point-based Rendering), capable of providing enhanced, high-quality pixel-level instance segmentation in identifying and detecting green gram pods or Mung Bean (Vigna Radiata) in natural field conditions rendering crisp and smooth boundaries for accurately locating the green gram pods. The results indicate that the proposed framework outperforms the famous Mask R-CNN model to obtain higher mean average precision and improved quality of detections.

Real-Time Recognition and Localization of Apples for Robotic Picking Based on Structural Light and Deep Learning

The apple is a delicious fruit with high nutritional value that is widely grown around the world. Apples are traditionally picked by hand, which is very inefficient. The development of advanced fruit-picking robots has great potential to replace manual labor. A major prerequisite for a robot to successfully pick fruits the accurate identification and positioning of the target fruit. The active laser vision systems based on structured algorithms can achieve higher recognition rates by quickly capturing the three-dimensional information of objects. This study proposes to combine the laser active vision system with the YOLOv5 neural network model to recognize and locate apples on trees. The method obtained accurate two-dimensional pixel coordinates, which, when combined with the active laser vision system, can be converted into three-dimensional world coordinates for apple recognition and positioning. On this basis, we built a picking robot platform equipped with this visual recognition system, and carried out a robot picking experiment. The experimental findings showcase the efficacy of the neural network recognition algorithm proposed in this study, which achieves a precision rate of 94%, an average precision mAP% of 92.86%, and a spatial localization accuracy of approximately 4 mm for the visual system. The implementation of this control method in simulated harvesting operations shows the promise of more precise and successful fruit positioning. In summary, the integration of the YOLOv5 neural network model with an active laser vision system presents a novel and effective approach for the accurate identification and positioning of apples. The achieved precision and spatial accuracy indicate the potential for enhanced fruit-harvesting operations, marking a significant step towards the automation of fruit-picking processes.

Workspace decomposition based path planning for fruit-picking robot in complex greenhouse environment

The working environment of fruit-picking robots is very complicated. Generally, there are a large number of obstacles such as branches, immature crops, etc. Besides, the agricultural scene is not as static. For sampling-based methods such as Probabilistic RoadMap (PRM) and Rapidly-exploring Random Trees (RRT), they suffer from inferior path quality in obstacle condensed environments. This paper proposes a local search path planning method in the feasible region of fruit-picking robots based on discrete workspace guidance and configuration space exploration correction. It is used to plan the motion of fruit-picking robots for avoiding obstacles and grasping in dynamic and complex agricultural environments. First, the workspace is discretized, which makes the location information of obstacles computable. Then, a connected region path is found to guide the robot to search in the local configuration space. This greatly improves the continuity of the generated paths in the configuration space. The local search is further used to correct the weights of discrete regions of the workspace to find better connected paths with higher planning speed. The experimental results show that the quality of paths planned by proposed method is generally better than that of RRT and RRT-CONNECT, and the planning speed is faster than that of Task-Space RRT (TSRRT). Proposed algorithm enables real-world picking at 10.5 piece/s and a recovery rate of 80.0 %. This method significantly improves the path quality in dynamic and complex agricultural scenarios.

Fruit Picking Robot Arm Training Solution Based on Reinforcement Learning in Digital Twin

In the era of Industry 4.0, digital agriculture is developing very rapidly and has achieved considerable results. Nowadays, digital agriculture-based research is more focused on the use of robotic fruit picking technology, and the main research direction of such topics is algorithms for computer vision. However, when computer vision algorithms successfully locate the target object, it is still necessary to use robotic arm movement to reach the object at the physical level, but such path planning has received minimal attention. Based on this research deficiency, we propose to use Unity software as a digital twin platform to plan the robotic arm path and use ML-Agent plug-in as a reinforcement learning means to train the robotic arm path, to improve the accuracy of the robotic arm to reach the fruit, and happily the effect of this method is much improved than the traditional method.

Detection and Positioning of Camellia oleifera Fruit Based on LBP Image Texture Matching and Binocular Stereo Vision

To achieve the rapid recognition and accurate picking of Camellia oleifera fruits, a binocular vision system composed of two industrial cameras was used to collect images of Camellia oleifera fruits in natural environments. The YOLOv7 convolutional neural network model was used for iterative training, and the optimal weight model was selected to recognize the images and obtain the anchor frame region of the Camellia oleifera fruits. The local binary pattern (LBP) maps of the anchor frame region were extracted and matched by using the normalized correlation coefficient template matching algorithm to obtain the positions of the center point in the left and right images. The recognition experimental results showed that the accuracy rate, recall rate, mAP and F1 of the model were 97.3%, 97.6%, 97.7% and 97.4%. The recognition rate of the Camellia oleifera fruit with slight shading was 93.13%, and the recognition rate with severe shading was 75.21%. The recognition rate of the Camellia oleifera fruit was 90.64% under sunlight condition, and the recognition rate was 91.34% under shading condition. The orchard experiment results showed that, in the depth range of 400–600 mm, the maximum error value of the binocular stereo vision system in the depth direction was 4.279 mm, and the standard deviation was 1.142 mm. The detection and three-dimensional positioning accuracy of the binocular stereo vision system for Camellia oleifera fruits could basically meet the working requirements of the Camellia oleifera fruit-picking robot.

Vision-based fruit recognition via multi-scale attention CNN

Fruit quality assessment, grading and sorting are of vital importance to fruit processing, and all these involve fruit recognition. Vision-based fruit recognition can recognize fruit automatically and further support more applications such as fruit-picking robots, self-checkout service, and dietary guidance. Recent works resort to Convolutional Neural Networks (CNNs) for vision-based fruit recognition because of their powerful expressive capacity. However, most works simply utilize existing CNNs and ignore characteristics of fruit images, resulting in sub-optimal recognition performance. To solve this problem, we adopt a Multi-Scale Attention Network (MSANet), which explores attention from different layers of CNNs and aggregates various visual attentional features from different levels into final comprehensive representations. Extensive evaluations on four fruit benchmark datasets demonstrate that the method achieves state-of-the-art recognition performance. In addition, we report various experimental results from different deep networks as baselines. Codes and models will be released upon publication. Models and code are available at http://123.57.42.89/codes/msanet.html.

RDE-YOLOv7: An Improved Model Based on YOLOv7 for Better Performance in Detecting Dragon Fruits

There is a great demand for dragon fruit in China and Southeast Asia. Manual picking of dragon fruit requires a lot of labor. It is imperative to study the dragon fruit-picking robot. The visual guidance system is an important part of a picking robot. To realize the automatic picking of dragon fruit, this paper proposes a detection method of dragon fruit based on RDE-YOLOv7 to identify and locate dragon fruit more accurately. RepGhost and decoupled head are introduced into YOLOv7 to better extract features and better predict results. In addition, multiple ECA blocks are introduced into various locations of the network to extract effective information from a large amount of information. The experimental results show that the RDE-YOLOv7 improves the precision, recall, and mean average precision by 5.0%, 2.1%, and 1.6%. The RDE-YOLOv7 also has high accuracy for fruit detection under different lighting conditions and different blur degrees. Using the RDE-YOLOv7, we build a dragon fruit picking system and conduct positioning and picking experiments. The spatial positioning error of the system is only 2.51 mm, 2.43 mm, and 1.84 mm. The picking experiments indicate that the RDE-YOLOv7 can accurately detect dragon fruits, theoretically supporting the development of dragon fruit-picking robots.

A Review of Target Recognition Technology for Fruit Picking Robots: From Digital Image Processing to Deep Learning

Machine vision technology has dramatically improved the efficiency, speed, and quality of fruit-picking robots in complex environments. Target recognition technology for fruit is an integral part of the recognition systems of picking robots. The traditional digital image processing technology is a recognition method based on hand-designed features, which makes it difficult to achieve better recognition as it results in dealing with the complex and changing orchard environment. Numerous pieces of literature have shown that extracting special features by training data with deep learning has significant advantages for fruit recognition in complex environments. In addition, to realize fully automated picking, reconstructing fruits in three dimensions is a necessary measure. In this paper, we systematically summarize the research work on target recognition techniques for picking robots in recent years, analyze the technical characteristics of different approaches, and conclude their development history. Finally, the challenges and future development trends of target recognition technology for picking robots are pointed out.

A study on Shine-Muscat grape detection at maturity based on deep learning

The efficient detection of grapes is a crucial technology for fruit-picking robots. To better identify grapes from branch shading that is similar to the fruit color and improve the detection accuracy of green grapes due to cluster adhesion, this study proposes a Shine-Muscat Grape Detection Model (S-MGDM) based on improved YOLOv3 for the ripening stage. DenseNet is fused in the backbone feature extraction network to extract richer underlying grape information; depth-separable convolution, CBAM, and SPPNet are added in the multi-scale detection module to increase the perceptual field of grape targets and reduce the model computation; meanwhile, PANet is combined with FPN to promote inter-network information flow and iteratively extract grape features. In addition, the CIOU regression loss function is used and the prior frame size is modified by the k-means algorithm to improve the accuracy of detection. The improved detection model achieves an AP value of 96.73% and an F1 value of 91% on the test set, which are 3.87% and 3% higher than the original network model, respectively; the average detection speed under GPU reaches 26.95 frames/s, which is 6.49 frames/s higher than the original model. The comparison results with several mainstream detection algorithms such as SSD and YOLO series show that the method has excellent detection accuracy and good real-time performance, which is an important reference value for the problem of accurate identification of Shine-Muscat grapes at maturity.

Three-dimensional continuous picking path planning based on ant colony optimization algorithm.

Fruit-picking robots are one of the important means to promote agricultural modernization and improve agricultural efficiency. With the development of artificial intelligence technology, people are demanding higher picking efficiency from fruit-picking robots. And a good fruit-picking path determines the efficiency of fruit-picking. Currently, most picking path planning is a point-to-point approach, which means that the path needs to be re-planned after each completed path planning. If the picking path planning method of the fruit-picking robot is changed from a point-to-point approach to a continuous picking method, it will significantly improve its picking efficiency. The optimal sequential ant colony optimization algorithm(OSACO) is proposed for the path planning problem of continuous fruit-picking. The algorithm adopts a new pheromone update method. It introduces a reward and punishment mechanism and a pheromone volatility factor adaptive adjustment mechanism to ensure the global search capability of the algorithm, while solving the premature and local convergence problems in the solution process. And the multi-variable bit adaptive genetic algorithm is used to optimize its initial parameters so that the parameter selection does not depend on empirical and the combination of parameters can be intelligently adjusted according to different scales, thus bringing out the best performance of the ant colony algorithm. The results show that OSACO algorithms have better global search capability, higher quality of convergence to the optimal solution, shorter generated path lengths, and greater robustness than other variants of the ant colony algorithm.

Image Recognition for Fruit-Picking Robots

Image Recognition for Fruit-Picking Robots

A Study on Long-Close Distance Coordination Control Strategy for Litchi Picking

For the automated robotic picking of bunch-type fruit, the strategy is to roughly determine the location of the bunches, plan the picking route from a remote location, and then locate the picking point precisely at a more appropriate, closer location. The latter can reduce the amount of information to be processed and obtain more precise and detailed features, thus improving the accuracy of the vision system. In this study, a long-close distance coordination control strategy for a litchi picking robot was proposed based on an Intel Realsense D435i camera combined with a point cloud map collected by the camera. The YOLOv5 object detection network and DBSCAN point cloud clustering method were used to determine the location of bunch fruits at a long distance to then deduce the sequence of picking. After reaching the close-distance position, the Mask RCNN instance segmentation method was used to segment the more distinctive bifurcate stems in the field of view. By processing segmentation masks, a dual reference model of “Point + Line” was proposed, which guided picking by the robotic arm. Compared with existing studies, this strategy took into account the advantages and disadvantages of depth cameras. By experimenting with the complete process, the density-clustering approach in long distance was able to classify different bunches at a closer distance, while a success rate of 88.46% was achieved during fruit-bearing branch locating. This was an exploratory work that provided a theoretical and technical reference for future research on fruit-picking robots.

A Visual Recognition and Path Planning Method for Intelligent Fruit-Picking Robots

With the rapid development of economy and the increasing improvement of agricultural production level, people’s demand for fruits is also increasing year by year. China is the largest fruit production and consumption country in the world. According to relevant statistics reported for China, by the end of 2019, the total amount of various fruits sold had reached about 270 million tons, with apples accounting for 48% of the global output and pears accounting for 69% of the national total output. However, China’s fruit picking is still dominated by manual picking process, which takes a lot of manpower and time to complete, therefore resulting in low fruit picking efficiency. Some farmers are unable to complete fruit picking in a short time, resulting in a large number of fruits unable to be listed, resulting in huge losses. To solve this problem, this paper focuses on the visual recognition and path planning for intelligent fruit-picking robot. Using robot to complete fruit picking is the best way at present. This paper establishes a picking robot recognition and positioning system based on stereo vision, which is used to identify and locate the fruits planted in the orchard area. The coordinate error of the target point of the intelligent fruit-picking robot coordinate system is less than 10 mm, which has high accuracy. Then, the path of intelligent fruit-picking robot is planned based on visual feedback algorithm and biological stimulation neural network. Our empirical evaluations suggest that the proposed robot walks in the planting park in the shape of “zigzag” and realizes full-coverage path planning after 6 turns. The results show the efficiency of the intelligent method.