Robotic Harvesting System Research Articles

Risk-averse Optimization for Improving Harvesting Efficiency of Autonomous Systems through Human Collaboration

Autonomous systems operating in unstructured and complex agricultural environments are susceptible to errors leading to uncertain losses in the efficiency of the system. In robotic harvesting, these losses would translate into lower harvesting efficiency. To this end, it is desirable to improve harvesting efficiency of robotic systems through human collaboration. The added labor costs associated with human involvement could be a concern since the robotic harvesting systems are expected to reduce harvesting costs. Therefore, the objective of this work is to develop optimal human-robot collaboration policies that minimize the risk of economic losses by identifying the components of the system that need to be serviced by a human supervisor all while guaranteeing a desired level of financial return. The developed risk-averse optimization solution is verified in a simulated grove environment for Florida Valencia citrus.

Deepblueberry: Quantification of Blueberries in the Wild Using Instance Segmentation

An accurate and reliable image-based quantification system for blueberries may be useful for the automation of harvest management. It may also serve as the basis for controlling robotic harvesting systems. Quantification of blueberries from images is a challenging task due to occlusions, differences in size, illumination conditions and the irregular amount of blueberries that can be present in an image. This paper proposes the quantification per image and per batch of blueberries in the wild, using high definition images captured using a mobile device. In order to quantify the number of berries per image, a network based on Mask R-CNN for object detection and instance segmentation was proposed. Several backbones such as ResNet101, ResNet50 and MobileNetV1 were tested. The performance of the algorithm was evaluated using the Intersection over Union Error (IoU) and the competitive mean Average Precision (mAP) per images and per batch. The best detection result was obtained with the ResNet50 backbone achieving a mIoU score of 0.595 and mAP scores of 0.759 and 0.724 respectively (for IoU thresholds 0.5 and 0.7). For instance segmentation, the best results obtained were 0.726 for the mIoU metric and 0.909 and 0.774 for the mAP metric using thresholds of 0.5 and 0.7 respectively.

Design and test of robotic harvesting system for cherry tomato

Harvesting of fresh-eating cherry tomato was highly costly on labor and time. In order to achieve mechanical harvesting for the fresh-eating tomato, a new harvesting robot was designed, which consisted of a stereo visual unit, an end-effector, manipulator, a fruit collector, and a railed vehicle. The robot configuration and workflow design focused on the special cultivating condition. Three key parts were introduced in detail: a railroad vehicle capably moving on both ground and rail was adopted as the robot’s carrier, a visual servo unit was used to identify and locate the mature fruits bunch, and the end-effector to hold and separate the fruit bunch was designed based on the stalk’s mechanical features. The field test of the new developed robot was conducted and the results were analyzed. The successful harvest rate of the robot was 83%, however, each successful harvest averagely needed 1.4 times attempt, and a single successful harvesting cycle cost 8 s excluding the time cost on moving. Keywords: cherry tomato, harvesting robot, visual servo, configuration design, field test DOI: 10.25165/j.ijabe.20181101.2853 Citation: Feng Q C, Zou W, Fan P F, Zhang C F, Wang X. Design and test of robotic harvesting system for cherry tomato. Int J Agric & Biol Eng, 2018; 11(1): 96–100.

Design and test of robotic harvesting system for cherry tomato

Design and test of robotic harvesting system for cherry tomato

Design, integration, and field evaluation of a robotic apple harvester

AbstractEvery apple destined for the fresh market is picked by the human hand. Despite extensive research over the past four decades, there are no mechanical apple harvesters for the fresh market commercially available, which is a significant concern because of increasing uncertainty about the availability of manual labor and rising production costs. The highly unstructured orchard environment has been a major challenge to the development of commercially viable robotic harvesting systems. This paper reports the design and field evaluation of a robotic apple harvester. The approach adopted was to use a low‐cost system to assess required sensing, planning, and manipulation functionality in a modern orchard system with a planar canopy. The system was tested in a commercial apple orchard in Washington State. Workspace modifications and performance criteria are thoroughly defined and reported to help evaluate the approach and guide future enhancements. The machine vision system was accurate and had an average localization time of 1.5 s per fruit. The seven degree of freedom harvesting system successfully picked 127 of the 150 fruit attempted for an overall success rate of 84% with an average picking time of 6.0 s per fruit. Future work will include integration of additional sensing and obstacle detection for improved system robustness.

Detection of red and bicoloured apples on tree with an RGB-D camera

Recognising and accurately locating fruits on a tree is a critical challenge in developing fruit-by-fruit robotic harvesting. Many researchers have investigated the potential of red, green, blue (RGB) colour imaging for this purpose, but have had limited success due to the occlusion of the target fruits by foliage, branches or other fruits as well as due to the non-uniform and unstructured nature of an orchard environment. Recently, novel, cost-effective camera systems have become available which provide both colour (RGB) and three dimensional (3D) shape information. As these have shown potential for 3D perception for robots operating in unstructured environments, the potential of such an RGB-D camera for the detection and localisation of red and bicoloured apples on tree was investigated in this study. Images were acquired with this camera system in fruit orchards under a light shield blocking direct sunlight, and an algorithm to detect and localise red and bicoloured apples based on colour and shape features was developed. When the algorithm was applied to the data acquired in these orchards, 100% of the fully visible apples and 82% of the partially occluded apples were detected correctly. The location estimation error was below 10 mm in all the coordinate axes of the Cartesian space. This high detection and location accuracy and short processing time (below 1 s for simultaneous detection of 20 apples), makes the developed algorithm suitable for implementation in a robotic harvesting system, and for yield estimation and orchard monitoring.

Bruise Responses of Apple-to-Apple Impact

Abstract: Currently, no commercial success has been achieved for mechanized or automated harvesting of fresh market apples. Automated or robotic harvesting systems have primarily been limited by the speed, accuracy, robustness and cost of the machine. Bulk harvesting has a potential to address these challenges but the technique suffers from a high level of harvest-induced fruit damage. To gain an understanding of bruising potential during fruit-to-fruit collision that occurs during bulk harvesting, apple-to-apple impact tests were carried out in a laboratory using a specially design pendulum-type apple bruising test device. This device could attach a half-apple on its pendulum impactor with its middle zone as the impacting surface to apply an impact force on different surfaces of a whole apple placed on a stationary platform of the device to quantitatively study the effects of different impacts. Three impact patterns of middle-to-top, middle-to-middle and middle-to-bottom zone between two apple specimens were tested using “Granny Smith” apples. Six to seven different levels of impact forces were applied for each test pattern. Obtained results revealed that if the impact force were less than 16 N it would result in no bruising on apple specimen in all test conditions. This particular variety of apple could tolerate a moderate impact force of ~28 N and still maintain an “Extra Fancy” quality over 98% for all tested fruit. This study also showed that, if the impact force could be controlled on a level less than 53 N, about 94% or more apples could still meet a “fresh-market” quality threshold.

Development of a Robotic Harvesting System for Tomato Clusters with Low-Node-Order Pinching and High-Density Planting (Part 2)

Development of a Robotic Harvesting System for Tomato Clusters with Low-Node-Order Pinching and High-Density Planting (Part 2)

Development of a Robotic Harvesting System for Tomato Clusters with Low-Node-Order Pinching and High-Density Planting (Part 1)

Development of a Robotic Harvesting System for Tomato Clusters with Low-Node-Order Pinching and High-Density Planting (Part 1)

Review on fruit harvesting method for potential use of automatic fruit harvesting systems

In horticultural industry, conventional harvesting is done by ‘handpicking’ methods to remove hundreds of fruits such as citrus fruits in random spatial locations on the individual fruit trees. It is well known that harvesting fruits in a large scale is still inefficient and not cost effective. To solve this challenging task, mechanical harvesting systems have been investigated and practiced to enhance profitability and efficiency of horticultural businesses. However they often damage fruits in the harvesting process. Development of efficient fruit removal methods are required to maintain the fruits quality. This paper reviews fruit harvesting systems from purely mechanical based systems in which operator involvement is still required, to automatic robotic harvesting systems which require minimal or no human intervention in their operation. The researches on machine vision system methodologies used in the automatic detection, inspection and the location of fruits for harvesting are also included. The review is focused on the citrus fruits due to the fact that the research on citrus fruit harvesting mechanism is a bit more advanced than others. Major issues are addressed in the camera sensor and filter designs and image segmentation methods used to identify the fruits within the image. From this review, the major research issues are addressed as future research directions.

Technical Note: Ripe Tomato Detection for Robotic Vision Harvesting Systems in Greenhouses

Effective recognition and localization of ripe tomatoes from a complex background is the key issue for robotic harvesting systems in greenhouses. In this study, a detection approach for ripe tomatoes is proposed based on their color and shape features. Images containing ripe tomatoes are first segmented by K-means clustering using the L*a*b* color space. To recognize a single ripe tomato, mathematical morphology is used to denoise the image and to handle the situations of image overlapping and sheltering. To improve the accuracy of tomato detection, the shape features are combined with the color features. Finally, the center of the detected single ripe tomato is calculated. Experimental results demonstrate the effectiveness of the proposed method. The successful detection rate is approximately 94%. Even with the overlapping and complex cluster background, the proposed algorithm still shows a very strong robustness.

A Color Model for Recognition of Apples by a Robotic Harvesting System.

A Color Model for Recognition of Apples by a Robotic Harvesting System.

Ｖ字型整枝されたナスを対象としたロボット収穫システム（第２報）―ナスの収穫実験―

Ｖ字型整枝されたナスを対象としたロボット収穫システム（第２報）―ナスの収穫実験―

AE—Automation and Emerging Technologies: A Segmentation Algorithm for the Automatic Recognition of Fuji Apples at Harvest

An algorithm for the automatic recognition of Fuji apples on the tree was developed for a robotic harvesting system. The machine vision system was composed of a colour charge coupled device camera to capture apple images, and a personal computer to process images for recognising and locating the fruit. The fruit image was enhanced using the red colour difference because results showed that the fruit had the highest red colour difference among the objects in the image. The intensity histogram of the enhanced image had a bimodal distribution for the fruit portion and the background portion. The maximum grey level variance of the red colour difference between the fruit and the background determined the optimal threshold. The optimal threshold had the minimum probability between the peaks of the two distributions. Results of the segmentation using the optimal threshold showed a success rate of over 88%. However, the error rate was over 18% for images with the back lighting condition.

Robotic Harvesting System for Eggplants

The harvesting operation for eggplants is complicated and accounts for a little less than 40% of the total number of working hours. For automating the harvesting operation, an intelligent robot that can emulate the judgment of human labor is necessary. This study was conducted with a view to developing a robotic harvesting system that performs recognition, approach, and picking tasks. In order to accomplish these tasks, 3 essential components were developed. First, a machine vision algorithm combining a color segment operation and a vertical dividing operation was developed. The algorithm could detect the fruit even under different light conditions. Next, a visual feedback fuzzy control model to actuate a manipulator was designed. The control model enabled the manipulator end to approach the fruit from a distance of 300 mm. Furthermore, an end-effector composed of fruit-grasping mechanism, a size-judging mechanism, and a peduncle-cutting mechanism was developed. It produced enough force for grasping the fruit and cutting the tough peduncle. Finally, the 3 essential components were functionally combined, and a basic harvesting experiment was conducted in the laboratory to evaluate the performance of the system. The system showed a successful harvesting rate of 62.5%, although the end-effector cut the peduncle at a slightly higher position from the fruit base. The execution time for harvesting of an eggplant was 64.1 s.

Development of a Harvesting End-Effector for Eggplants.

A robotic harvesting system, which emulates the fruit recognition, approach, and picking tasks, has been developed for the purpose of automatic harvesting for eggplants (Solanum melongena L.). In this study the harvesting end-effectors for eggplants (prototype-1 and prototype-2) were fabricated. The end-effectors were composed of a fruit-size-judging mechanism, a fruit-grasping mechanism, and a peduncle-cutting mechanism to pick the fruit selectively on the basis of fruit length after the manipulator end closely approaches. An experimental result on picking posture using prototype-1 proved that lifting the fruit at an angle of approximately 30° was effective for separating the fruit from leaves. A basic harvesting experiment using prototype-2, which was improved on the fruit-size-judging and fruit-grasping mechanism, was also conducted. It was found that the success rate of harvesting was 62. 5% and the harvesting time per fruit was about 64 seconds. The main cause of unsuccessful harvesting was that the accuracy of detection of the fruit base was not sufficient. This study showed the great feasibility of automating the basic harvesting motion for eggplants.

Visual Feedback Guidance of Manipulator for Eggplant Harvesting using Fuzzy Logic.

This study was conducted as a first step in the development of the robotic harvesting system for eggplants. The robotic harvesting system with hand-eye structure was fabricated and the visual feedback fuzzy control model to operate the manipulator was developed. The control model was examined in terms of manipulator guidance to the fruit, estimation of the fruit maximal diameter for selective harvesting and estimation of the fruit angle for the picking movement. From the results of the approach experiment, the control model enabled the manipulator end to reach a fruit 300 mm away with 6 to 10 feedback signals. The manipulator trajectory varied in vertical direction with fruit location and the control model actuated the manipulator end slightly upward than the target region. The control model could estimate the fruit maximal diameter with an accuracy of 29.0±17.9%. However, it still has a low standard of performance as a visual sensor for selective harvesting. Moreover, it could estimate the fruit angle in row direction with an error of 3.6±2.60. The visual feedback guidance ot the manipulator was confirmed to be an effective method for a robotic eggplant harvesting.

An algorithm for locating mushrooms in a growing bed

Robotic harvesting systems for commercial mushroom production will require image guidance. This paper describes an algorithm which will identify the size and position of individual mushrooms in the growing bed. The algorithm uses knowledge of the shape of the mushroom to track around the outline in the grey-level image.

Culturing and robotic harvesting of bone marrow, lymph nodes, peripheral blood, fibroblasts, and solid tumors with in situ techniques

We have modified our in situ method for culturing amniocytes and have successfully used this new procedure to culture cells from bone marrow, lymph nodes, peripheral blood, fibroblasts, and solid tumors. We have been so satisfied with this method that we have used it in our routine practice for the past 2.5 years. A convenient feature of the in situ method is that the cultures can be processed with a commercially available robotic harvesting system (modified Tecan Robotic Sample Processor Model 505). This system also works well for in situ cultures of amniotic fluids. Robotic harvesting of in situ cultures provides a consistent and accurate procedure that saves about 10–15 minutes of technologist time per case.