Fruit Harvesting System Research Articles

Dynamic visual servo control methods for continuous operation of a fruit harvesting robot working throughout an orchard

Fruit-picking robots are crucial for achieving efficient orchard harvesting. To genuinely meet the commercial production needs of farmers, the new generation of fruit-picking robots must be capable of demonstrating complete and continuous observation, movement, and picking behaviors throughout complex orchards, akin to real human employees. This poses systematic challenges, as many prior researches have focused solely on a part of the continuous operation of the entire orchard, such as fruit positioning, navigation, path planning, or grasping. These isolated basic functions are important but insufficient for fulfilling operational requirements on a macro scale and continuous situation. Developing an efficient control method for each basic module and constructing their internal coordination is vital for transitioning a harvesting robot from a functional prototype to a practical machine. In this context, this study tackles the visual servo control problem for efficient locomotion, picking, and their seamless integration. A set of vision algorithms for locomotion destination estimation, real-time self-positioning, and dynamic harvesting is proposed. Additionally, a solid coordination mechanism for continuous locomotion and picking behavior is established. Each method offers distinct advantages, such as improved accuracy, adaptability to varying conditions, and enhanced picking efficiency, enabling the robot to operate autonomously and continuously. Comprehensive field experiments validated the soundness of the methods. The primary contribution of this study lies in addressing the challenge of continuous operation in an entire orchard as a systematic problem and providing new insights into control methods for the future development of highly autonomous, practical, and user-oriented fruit harvesting systems.

Application of Machine Vision Techniques in Low-Cost Devices to Improve Efficiency in Precision Farming

In the context of recent technological advancements driven by distributed work and open-source resources, computer vision stands out as an innovative force, transforming how machines interact with and comprehend the visual world around us. This work conceives, designs, implements, and operates a computer vision and artificial intelligence method for object detection with integrated depth estimation. With applications ranging from autonomous fruit-harvesting systems to phenotyping tasks, the proposed Depth Object Detector (DOD) is trained and evaluated using the Microsoft Common Objects in Context dataset and the MinneApple dataset for object and fruit detection, respectively. The DOD is benchmarked against current state-of-the-art models. The results demonstrate the proposed method’s efficiency for operation on embedded systems, with a favorable balance between accuracy and speed, making it well suited for real-time applications on edge devices in the context of the Internet of things.

Multi-view gripper internal sensing for the regression of strawberry ripeness using a mini-convolutional neural network for robotic harvesting

The capability of robotic fruit-harvesting systems to accurately assess the ripeness of fruits is crucial for fulfilling the diverse standards of the market and the preferences of consumers. Existing studies involving fruit ripeness estimation mainly focus on detecting of ripe strawberries as one class or classifying of ripeness into several stages, such as overripe, ripe, and unripe. Current harvesting robots also lack the ability to determine the ripeness of the back of fruit with respect to the robots. This paper proposes a lightweight convolutional neural network (CNN) regression model for ripeness quantification based on the internal image sensing system of the gripper with full-view coverage of the fruit for strawberry-harvesting robots. A gripper internal sensing system was developed using two RGB cameras that could provide full-view fruit coverage for a more accurate estimation of fruit ripeness. Four base CNN networks capable of feature learning were used for feature extraction, followed by the utilization of newly added dense layers that produced a regressed value to represent the strawberry ripeness. However, the base networks were cumbersome and relatively slow due to their complex structures. To simplify this, a new MiniNet with fewer convolutional layers was proposed to reduce the model size and inference time. All models were trained via two loss functions, mean square error and Huber loss. The results showed that the models trained via Huber loss performed better. An Xception model trained on Huber loss showed the best performance with a mean absolute error of 4.0% and an average inference time of 42.5 ms. Of all the models, the new MiniNet was the most lightweight and fastest model while maintaining high performances (an mean absolute error of 4.8% and an inference time of 6.5 ms for Huber loss trained model). The proposed method may be also applicable to other fruit-harvesting systems.

Tomato Fruit Detection Using Modified Yolov5m Model with Convolutional Neural Networks.

The farming industry is facing the major challenge of intensive and inefficient harvesting labors. Thus, an efficient and automated fruit harvesting system is required. In this study, three object classification models based on Yolov5m integrated with BoTNet, ShuffleNet, and GhostNet convolutional neural networks (CNNs), respectively, are proposed for the automatic detection of tomato fruit. The various models were trained using 1508 normalized images containing three classes of cherry tomatoes, namely ripe, immature, and damaged. The detection accuracy for the three classes was found to be 94%, 95%, and 96%, respectively, for the modified Yolov5m + BoTNet model. The model thus appeared to provide a promising basis for the further development of automated harvesting systems for tomato fruit.

Assessment of Smart Mechatronics Applications in Agriculture: A Review

Smart mechatronics systems in agriculture can be traced back to the mid-1980s, when research into automated fruit harvesting systems began in Japan, Europe, and the United States. Impressive advances have been made since then in developing systems for use in modern agriculture. The aim of this study was to review smart mechatronics applications introduced in agriculture to date, and the different areas of the sector in which they are being employed. Various literature search approaches were used to obtain an overview of the current state-of-the-art, benefits, and drawbacks of smart mechatronics systems. Smart mechatronics modules and various networks applied in the processing of agricultural products were examined. Finally, relationships in the data retrieved were tested using a one-way analysis of variance on keywords and sources. The review revealed limited use of sophisticated mechatronics in the agricultural industry in practice at a time of falling production rates and a dramatic decline in the reliability of the global food supply. Smart mechatronics systems could be used in different agricultural enterprises to overcome these issues.

Recent patents on intelligent automated fruit harvesting robots for sweet pepper and apple

Robotic harvesting offers suitable solutions for optimizing scheduling, selection enabling, increasing operation efficiency and finally reduce the labor costs. These attributes allow the users of robot harvesters to maximize production efficiency and profits. This article reviews two automated fruit harvesting systems used for sweet pepper and apple as an example to demonstrate the effectiveness of recent patents on intelligent automatic harvesting robots in horticulture.

Integration of stereo vision system calibration and kinematic calibration for an autonomous kiwifruit harvesting system

Stereo vision system and manipulator are two major components of an autonomous fruit harvesting system. In order to raise the fruit-harvesting rate, stereo vision system calibration and kinematic calibration are two significant processes to improve the positional accuracy of the system. This article reviews the mathematics of these two calibration processes and presents an integrated approach for acquiring calibration data and calibrating both components of an autonomous kiwifruit harvesting system. The calibrated harvesting system yields good positional accuracy in the laboratory tests, especially in harvesting individual kiwifruit. However, the performance is not in line with the outcomes in the orchard field tests due to the cluster growing style of kiwifruit. In the orchard test, the calibrations reduce the fruit drop rate but it does not impressively raise the fruit harvesting rate. Most of the fruit in the clusters remain in the canopy due to the invisibility of the stereo vision system. After analyzing the existing stereo vision system, a future visual sensing system research direction for an autonomous fruit harvesting system is justified.

Design and Evaluation of End Effectors for a Vacuum-Based Robotic Apple Harvester

HighlightsA critical torque of 0.257 N-m is required to detach 95% of apples, when the preferred twist picking mode is used.New silicone-based end effectors performed much better than the original, non-conformable end effector.The straight end effector had the best overall picking performance with 87% picking success rate.The new vacuum-based harvesting robot looks promising for automated harvesting of apples.Abstract. The end effector plays a critical role in fruit picking by a robotic fruit harvesting system. A newly developed vacuum-based robotic apple harvesting system, using the twist-and-pull fruit picking method, has shown promise in picking fruit from clusters and navigating through the tree canopies. The robot’s original thin foam end effector failed to achieve acceptable picking performance because it was unable to conform to fruit of different sizes and thus could not generate sufficient suction forces needed to detach fruit. This research was therefore aimed at developing new end effectors to greatly enhance the robot’s fruit picking performance. Field manual pulling and twisting experiments for three varieties of apples were conducted, and the critical pulling or suction force and twisting torque needed to detach 95% of apples were determined to be 28.3 N and 0.257 N-m (equivalent to a pulling force of 21.0 N for the current robot’s configuration), respectively. Three new silicone-based end effectors of different geometries (denoted as “Straight”, “Bellow” and “Curved”) were designed and fabricated, and they were evaluated in lab and field experiments. Results showed that the three new end effectors performed significantly better than the original, unconformable end effector based on multiple performance metrics, including vacuum pressure, overall picking success rate, picking rate by the rotation mechanism, and fruit attachment orientation. The “Straight” end effector performed consistently better than the other two new end effectors; it had 87% overall picking success rate, 65% success rate by the rotation mechanism alone, and 89% success rate when the middle section or cheek of fruit attached to the end effector. With further improvements, the “Straight” end effector should meet the apple picking performance requirement for the new vacuum-based robotic harvesting system. Keywords: Apple, End Effector, Fruit, Harvest, Robotics, Vacuum.

Development of a Visual Servo System for Robotic Fruit Harvesting

One of the challenges in the future of food production, amidst increasing population and decreasing resources, is developing a sustainable food production system. It is anticipated that robotics will play a significant role in maintaining the food production system, specifically in labor-intensive operations. Therefore, the main goal of this project is to develop a robotic fruit harvesting system, initially focused on the harvesting of apples. The robotic harvesting system is composed of a six-degrees-of-freedom (DOF) robotic manipulator, a two-fingered gripper, a color camera, a depth sensor, and a personal computer. This paper details the development and performance of a visual servo system that can be used for fruit harvesting. Initial test evaluations were conducted in an indoor laboratory using plastic fruit and artificial trees. Subsequently, the system was tested outdoors in a commercial fruit orchard. Evaluation parameters included fruit detection performance, response time of the visual servo, and physical time to harvest a fruit. Results of the evaluation showed that the developed visual servo system has the potential to guide the robot for fruit harvesting.

Symmetry-based 3D shape completion for fruit localisation for harvesting robots

Fruit localisation is a crucial step in developing a robotic fruit-harvesting system. This paper aims to improve the localisation accuracy of fruits in 3D space. In the machine vision system of a harvesting robot, in a single view the visible area of a target is often incomplete and therefore, cannot be directly used to accurately determine the target location. A 3D shape completion method is proposed that can be used on the partially visible images of strawberries obtained from a single view. This method proposed a given number of symmetric plane candidates based on the assumption that the targets are symmetrical, which is normally true for fruits such as such apples, citrus fruits and strawberries. Corresponding rating rules were proposed to select the optimal symmetry to be used for the shape completion. The algorithm was then tested on reconstructed point clouds and implemented on a strawberry harvester equipped with a Red Green Blue-Depth (RGB-D) camera. The evaluation on reconstructed strawberry data showed that the intersection over union (IoU) and centre deviation between the results obtained by this method and ground truth were 0.77 and 6.9 mm, respectively, whilst those of the unprocessed partial data were 0.56 and 14.1 mm. The evaluation results of the strawberry data captured with the RGB-D camera showed that the IoU and centre deviation between the results obtained by this method and ground truth were 0.61 and 5.7 mm, respectively, whilst those of the unprocessed partial data were 0.47 and 8.9 mm.

Macaw palm supply chain: Evaluation of a semi-mechanized fruit harvesting system

The macaw palm (Acrocomia aculeata) is native to Brazil and is a promising crop for oil extraction for biodiesel production. Production methods are still lacking, as well as the in-depth knowledge required for domestication and improved fruit harvesting. The use of mechanical vibrations, which is well established in crops such as coffee, may help to develop harvesting methods. Therefore, this study evaluated the operational performance of a semi-mechanized macaw palm fruit harvester prototype. The prototype was evaluated in field tests conducted in Araponga, Minas Gerais, with plants originating from accessions in Minas Gerais and Mato Grosso do Sul. The evaluations were based on data involving operational capacity, operational efficiency and damage to fruits. The prototype presented an average operational capacity of over 90 %, thus validating the use of mechanical vibrations for macaw palm fruit harvesting.

Recent Advances in Intelligent Automated Fruit Harvesting Robots

Robotic harvesting offers a solution to reducing labor costs, optimizing harvest scheduling, enabling selective harvesting, and increasing operation efficiency. These attributes allow the users of robotic harvesters to maximize production efficiency and profits. This article reviews automated fruit harvesting systems of sweet pepper, tomato, apple and kiwifruit as an example to demonstrate the recent advances in intelligent automatic harvesting robots in horticulture.

Improvements to and large‐scale evaluation of a robotic kiwifruit harvester

AbstractThe growing popularity of kiwifruit orchards in New Zealand is increasing the already high demand for seasonal labourers. A novel robotic kiwifruit harvester has been designed and built to help meet this demand [H. A. Williams et al. Biosystems Eng. 181 (2019), pp. 140–156]. Early evaluations of the platform have shown good results with the system capable of harvesting 51.0% of 1,456 kiwifruit in four bays with an average cycle‐time of 5.5 s/fruit. However, the harvester's high cycle‐time, and high fruit loss rate at 23.4%, prevent it from being commercially viable. This paper presents two new developments for the harvesting system, a new vision system and two new gripper variations designed to overcome the harvester's previous limitations. Furthermore, we have designed and conducted the largest real‐world evaluation of a robotic fruit harvesting system published to date with over 12,000 kiwifruit involved. The results of this trial demonstrated that our kiwifruit harvester is one of the most effective selective fruit harvesters in the world capable of successfully harvesting 86.0% of reachable fruit, and 55.8% of all kiwifruit with a cycle‐time of 2.78 s/fruit.

Colour-agnostic shape-based 3D fruit detection for crop harvesting robots

Most agricultural robots, fruit harvesting systems in particular, use computer vision to detect their fruit targets. Exploiting the uniqueness of fruit colour amidst the foliage, almost all of these computer vision systems rely on colour features to identify the fruit in the image. However, often the colour of fruit cannot be discriminated from its background, especially under unstable illumination conditions, thus rendering the detection and segmentation of the target highly sensitive or unfeasible in colour space. While multispectral signals, especially those outside the visible spectrum, may alleviate this difficulty, simpler, cheaper, and more accessible solutions are desired. Here exploiting both RGB and range data to analyse shape-related features of objects both in the image plane and 3D space is proposed. In particular, 3D surface normal features, 3D plane-reflective symmetry, and image plane highlights from elliptic surface points are combined to provide shape-based detection of fruits in 3D space regardless of their colour. Results are shown using a particularly challenging sweet pepper dataset with a significant degree of occlusions.

A Semi-Automated Harvesting Prototype for Shaking Fruit Tree Limbs

Mechanizing the fruit removal operation during fresh-market apple harvesting is essential for labor and cost savings for fruit growers. This study introduces a semi-automated mechanical fruit harvesting system that uses a fruit removal technique based a unique limb shaking mechanism called a dual motor actuator (DMA). The harvesting system was tested on ‘Gala‘ apples grown in a formally trained fruiting wall orchard. A manipulator arm, positioned by a human, held a fruit catchframe and an end-effector. The system was mounted on a mobile orchard platform. The DMA was developed as an infinitely variable pattern end-effector that applies rhythmic motions to a fruiting limb to remove fruit. The novelty of the DMA design is the use of two eccentrics mounted to electric motors that are pinned together to form a triangle with an adjustable base. Pattern, rhythm, and actuation time were varied to characterize the removal efficiency and fruit damage percentages. Removal efficiency averaged only 35% across the fruiting limb but increased to 88% within the actuation zone of the fruiting section, showing the potential of a semi-selective harvesting technique for localized fruit removal on branches trained to trellis wires. A circular pattern using a 200 cycles min-1 rhythm achieved the lowest overall damage (10%) of all combinations of the patterns and rhythms tested. Bruise percentage was also lowest (4%) using the same pattern and rhythm. Rhythm and actuation time were significant factors that influenced the fruit removal condition. No fruit was removed outside of the targeted actuation zone when using a single limb grasper. This localized limb actuation method shows great potential for removing apples from a limb while maintaining fresh-market quality fruit.

Evaluation of different mechanical fruit harvesting systems and oil quality in very large size olive trees

In 2006 and 2009, trials were carried out in the Apulia region in Southern Italy to evalu-ate the possibility of mechanizing olive harvesting in groves of old and very large trees. The trees belonged to the cultivars ‘Cellina di Nardò’ and ‘Ogliarola Salentina’. They were 60-100 years old and 7-9 m tall with a canopy volume of 140-360 m3. In the first half of November 2006, with a mechanical beater mounted on a tractor plus hand-held pneumatic combs, the harvesting yield was close to 90% of the total olives present in the canopy, and the harvesting working productivity was around 60 kg of harvested olives h-1 worker-1. With a self-propelled shaker attached to the main branches the harvesting yield was about 73% in ‘Cellina di Nardò’, and 40% in ‘Ogliarola Salentina’, while the harvesting working productivities were around 103 and 85 kg of harvested olives h-1 worker-1, respectively. In the second half of No-vember 2009, in ‘Cellina di Nardò’, with a mechanical beater mounted on a tractor plus nets on the ground or a catching frame (reversed umbrella) mounted on another tractor, the harvesting yield was about 97%. The working productivity was about 98 kg of harvested olives h-1 worker-1 with the mechanical beater plus nets and around 133 kg of harvested olives h-1 worker-1 when the mechanical beater was combined with a reversed umbrella. The oil obtained from the mechanically harvested olives was always of high quality. A basic economic evaluation of the harvesting costs is also reported.

A Proposal for Automatic Fruit Harvesting by Combining a Low Cost Stereovision Camera and a Robotic Arm

This paper proposes the development of an automatic fruit harvesting system by combining a low cost stereovision camera and a robotic arm placed in the gripper tool. The stereovision camera is used to estimate the size, distance and position of the fruits whereas the robotic arm is used to mechanically pickup the fruits. The low cost stereovision system has been tested in laboratory conditions with a reference small object, an apple and a pear at 10 different intermediate distances from the camera. The average distance error was from 4% to 5%, and the average diameter error was up to 30% in the case of a small object and in a range from 2% to 6% in the case of a pear and an apple. The stereovision system has been attached to the gripper tool in order to obtain relative distance, orientation and size of the fruit. The harvesting stage requires the initial fruit location, the computation of the inverse kinematics of the robotic arm in order to place the gripper tool in front of the fruit, and a final pickup approach by iteratively adjusting the vertical and horizontal position of the gripper tool in a closed visual loop. The complete system has been tested in controlled laboratory conditions with uniform illumination applied to the fruits. As a future work, this system will be tested and improved in conventional outdoor farming conditions.

Fruit Detection using Improved Multiple Features based Algorithm

icient locating the fruit on the tree is one of the major requirements for the fruit harvesting system. This paper presents the fruit detection using improved multiple features based algorithm. To detect the fruit, an image processing algorithm is trained for efficient feature extraction. The algorithm is designed with the aim of calculating different weights for features like intensity, color, orientation and edge of the input test image. The weights of different features represent the approximate locations of the fruit within an image. The Detection Efficiency is achieved up to 90% for different fruit image on tree, captured at different positions. The input images are the section of tree image. The proposed approach can be applied for targeting fruits for robotic fruit harvesting.

Review on fruit harvesting method for potential use of automatic fruit harvesting systems

In horticultural industry, conventional harvesting is done by ‘handpicking’ methods to remove hundreds of fruits such as citrus fruits in random spatial locations on the individual fruit trees. It is well known that harvesting fruits in a large scale is still inefficient and not cost effective. To solve this challenging task, mechanical harvesting systems have been investigated and practiced to enhance profitability and efficiency of horticultural businesses. However they often damage fruits in the harvesting process. Development of efficient fruit removal methods are required to maintain the fruits quality. This paper reviews fruit harvesting systems from purely mechanical based systems in which operator involvement is still required, to automatic robotic harvesting systems which require minimal or no human intervention in their operation. The researches on machine vision system methodologies used in the automatic detection, inspection and the location of fruits for harvesting are also included. The review is focused on the citrus fruits due to the fact that the research on citrus fruit harvesting mechanism is a bit more advanced than others. Major issues are addressed in the camera sensor and filter designs and image segmentation methods used to identify the fruits within the image. From this review, the major research issues are addressed as future research directions.

Development and performance evaluation of a manual fruit harvester

The conventional methods used for harvesting fruits are crude, laborious and time consuming. The fruit harvesting systems employing vibratory mechanism and mechanical harvesting prevalent in the advanced countries cannot be adopted in Indian conditions. With a view of simplifying the fruit harvesting operation, a manual fruit harvester was developed and tested. Three types of blade viz. curved blade, V-shaped blade and scissor type blade were fabricated and attached with the harvester. The field capacity and good fruit percentage by the three blades were determined for five important fruit crops. The combination of V-shaped harvester for pomegranate fruit gave maximum field capacity. The good fruit percentage was found to be more in lime. Guava was subjected to maximum damage due to its soft and smooth skin.