Fruit Detection System Research Articles

On-tree fruit detection system using Darknet-19 based SSD network

On-tree fruit detection system using Darknet-19 based SSD network

North Sulawesi Single Local Fruit Detection Using Efficient Attention Module Based on Deep Learning Architecture

A Local fruit detection system is an agricultural vision field that can be implemented to increase the profit of a commodity. Besides that, North Sulawesi has a variety of local fruits which are widely used by people in their area and have a high selling value. The sorting system is an essential process of agricultural robots to sequentially separate fruit one by one. This automation process requires an accurate vision system to detect and separate fruit precisely and precisely. In addition, the implementation of a practical application demands a method to be able to work in real-time on low-cost devices. This work aims to design a local single fruit detection system for Sulawesi North by applying deep learning architecture to produce high performance. The architecture is designed to consist of an effective backbone for rapidly separating the distinctive features, an efficient attention module to improve feature extraction performance, and a classifier module employed to estimate the probabilities of each local fruit category. As a result, the designed model produces an accuracy value of 99,27% and 99,57% on the Fruits-360 and the local datasets, respectively. It outperforms other light architectures. In addition, deep learning models are designed to produce higher efficiency values than other competitors and can operate quickly at 100,488 Frames per Second.

TomatoDet: Anchor-free detector for tomato detection.

The accurate and robust detection of fruits in the greenhouse is a critical step of automatic robot harvesting. However, the complicated environmental conditions such as uneven illumination, leaves or branches occlusion, and overlap between fruits make it difficult to develop a robust fruit detection system and hinders the step of commercial application of harvesting robots. In this study, we propose an improved anchor-free detector called TomatoDet to deal with the above challenges. First, an attention mechanism is incorporated into the CenterNet backbone to improve the feature expression ability. Then, a circle representation is introduced to optimize the detector to make it more suitable for our specific detection task. This new representation can not only reduce the degree of freedom for shape fitting, but also simplifies the regression process from detected keypoints. The experimental results showed that the proposed TomatoDet outperformed other state-of-the-art detectors in respect of tomato detection. The F1 score and average precision of TomatoDet reaches 95.03 and 98.16%. In addition, the proposed detector performs robustly under the condition of illumination variation and occlusion, which shows great promise in tomato detection in the greenhouse.

Benchmark of Deep Learning and a Proposed HSV Colour Space Models for the Detection and Classification of Greenhouse Tomato

The harvesting operation is a recurring task in the production of any crop, thus making it an excellent candidate for automation. In protected horticulture, one of the crops with high added value is tomatoes. However, its robotic harvesting is still far from maturity. That said, the development of an accurate fruit detection system is a crucial step towards achieving fully automated robotic harvesting. Deep Learning (DL) and detection frameworks like Single Shot MultiBox Detector (SSD) or You Only Look Once (YOLO) are more robust and accurate alternatives with better response to highly complex scenarios. The use of DL can be easily used to detect tomatoes, but when their classification is intended, the task becomes harsh, demanding a huge amount of data. Therefore, this paper proposes the use of DL models (SSD MobileNet v2 and YOLOv4) to efficiently detect the tomatoes and compare those systems with a proposed histogram-based HSV colour space model to classify each tomato and determine its ripening stage, through two image datasets acquired. Regarding detection, both models obtained promising results, with the YOLOv4 model standing out with an F1-Score of 85.81%. For classification task the YOLOv4 was again the best model with an Macro F1-Score of 74.16%. The HSV colour space model outperformed the SSD MobileNet v2 model, obtaining results similar to the YOLOv4 model, with a Balanced Accuracy of 68.10%.

Design of Citrus Fruit Detection System Based on Mobile Platform and Edge Computer Device.

Citrus fruit detection can provide technical support for fine management and yield determination of citrus orchards. Accurate detection of citrus fruits in mountain orchards is challenging because of leaf occlusion and citrus fruit mutual occlusion of different fruits. This paper presents a citrus detection task that combines UAV data collection, AI embedded device, and target detection algorithm. The system used a small unmanned aerial vehicle equipped with a camera to take full-scale pictures of citrus trees; at the same time, we extended the state-of-the-art model target detection algorithm, added the attention mechanism and adaptive fusion feature method, improved the model’s performance; to facilitate the deployment of the model, we used the pruning method to reduce the amount of model calculation and parameters. The improved target detection algorithm is ported to the edge computing end to detect the data collected by the unmanned aerial vehicle. The experiment was performed on the self-made citrus dataset, the detection accuracy was 93.32%, and the processing speed at the edge computing device was 180 ms/frame. This method is suitable for citrus detection tasks in the mountainous orchard environment, and it can help fruit growers to estimate their yield.

Research on Melon Fruit Selection Based on Rank with YOLOv4 Algorithm

Melon is one of the most popular fruits that is exceptionally favoured in Indonesia because it can be consumed directly as fresh fruit or processed as juice or salad. To meet the national market demand, several technologies are used to increase production, one of which is fruit selection. Plants need to be pruned based on fruit size so that fruit quality is maintained. One of the new approaches to detect plant fruits is using deep convolutional neural networks. The goal is to build a melon fruit detection system based on fruit size ranking for selection reliability. Recent work in deep neural networks has developed an excellent object detector, namely the one-stage You Only Look Once (YOLO) algorithm. We used the YOLOv4 model, the fourth generation of YOLO with speed acceleration and detection accuracy better than the previous versions. In addition, eight model schemes were tested with three different hyper-parameters: batch size, iterations, and learning rate. It was found that Scheme G using batch size 64, iterations 2000, and learning rate 0.001 obtained the highest score for both F1-score and mAP with values of 84.47% and 87.68%, respectively. It can be said that the F1-score value is directly proportional to the mAP value.

Automatic Fruits Classification System Based on Deep Neural Network

Fruit classification is playing a vital role in robot-based farming. The plucking of fruits and packing is done using robots nowadays. This could only be possible using efficiently trained robots base on machine learning. Different techniques have been developed for fruit classification, but still, there are many gaps, i.e., efficiency and accuracy. In this research work, we are targeting classification accuracy. This paper presented an Automatic Fruit Detection tool with good precision and recalled using deep learning neural networks. It will help in farming, cultivation, and produce sound effects in robotic farming. The aim is to build an accurate, fast and reliable fruit detection system, a vital element of an autonomous agricultural robotic platform; it is a crucial element for fruit yield estimation and automated harvesting. We used the ResNet-50 in the context of transfer learning. Different training choices were defined, i.e., 10% to 80%. Experimental results show that we compete for the prior approaches even on only 10% training. The proposed approach achieves state-of-the-art results compared to prior work with the F1 Score, which considers both precision and recall performances improving from 0.838 to 0.894 and 0.995 of accuracy. In addition to improved accuracy, this approach is also much quicker as compared to recent approaches.

Fruit detection, segmentation and 3D visualisation of environments in apple orchards

Development of an accurate and reliable fruit detection system is a challenging task. There are many complex conditions in orchard environments, such as changing illumination, appearance variation, and occlusion. Robotic vision is required to understand the working environments from the sensory data and guide the robotic arm to detach the fruits. In our previous work, a deep neural network DaSNet-v1 was developed to perform detection and segmentation on fruits and branches in orchard environments. However, semantic segmentation returns the mask for each class instead of each object. Segmentation on each fruit is important as it can provide abundant information of each object, especially for those overlapped fruits. This work presents an improved deep neural network DaSNet-v2, which can perform detection and instance segmentation on fruits, and semantic segmentation on branches. DaSNet-v2 is tested and validated by experimental results obtained from field-testing in an apple orchard. From the experiment results, DaSNet-v2 with resnet-101 achieves 0.868, 0.88 and 0.873 on recall and precision of detection, and accuracy of instance segmentation on fruits, and 0.794 on the accuracy of branches segmentation, respectively. DaSNet-v2 with light-weight backbone resnet-18 achieves 0.85, 0.87 and 0.866 on recall and precision of detection, and accuracy of instance segmentation on fruits, and 0.775 on the accuracy of branches segmentation, respectively. The average running time and weight size of light-weight DaSNet-v2 are 55 ms and 8.1 M, respectively. Experimental results show DaSNet-v2 can robustly and efficiently perform the vision sensing for robotic harvesting in apple orchards.

L*a*b*Fruits: A Rapid and Robust Outdoor Fruit Detection System Combining Bio-Inspired Features with One-Stage Deep Learning Networks.

Automation of agricultural processes requires systems that can accurately detect and classify produce in real industrial environments that include variation in fruit appearance due to illumination, occlusion, seasons, weather conditions, etc. In this paper we combine a visual processing approach inspired by colour-opponent theory in humans with recent advancements in one-stage deep learning networks to accurately, rapidly and robustly detect ripe soft fruits (strawberries) in real industrial settings and using standard (RGB) camera input. The resultant system was tested on an existent data-set captured in controlled conditions as well our new real-world data-set captured on a real strawberry farm over two months. We utilise score, the harmonic mean of precision and recall, to show our system matches the state-of-the-art detection accuracy (: 0.793 vs. 0.799) in controlled conditions; has greater generalisation and robustness to variation of spatial parameters (camera viewpoint) in the real-world data-set (: 0.744); and at a fraction of the computational cost allowing classification at almost 30fps. We propose that the L*a*b*Fruits system addresses some of the most pressing limitations of current fruit detection systems and is well-suited to application in areas such as yield forecasting and harvesting. Beyond the target application in agriculture this work also provides a proof-of-principle whereby increased performance is achieved through analysis of the domain data, capturing features at the input level rather than simply increasing model complexity.

Fruit detection and 3D location using instance segmentation neural networks and structure-from-motion photogrammetry

The development of remote fruit detection systems able to identify and 3D locate fruits provides opportunities to improve the efficiency of agriculture management. Most of the current fruit detection systems are based on 2D image analysis. Although the use of 3D sensors is emerging, precise 3D fruit location is still a pending issue. This work presents a new methodology for fruit detection and 3D location consisting of: (1) 2D fruit detection and segmentation using Mask R-CNN instance segmentation neural network; (2) 3D point cloud generation of detected apples using structure-from-motion (SfM) photogrammetry; (3) projection of 2D image detections onto 3D space; (4) false positives removal using a trained support vector machine. This methodology was tested on 11 Fuji apple trees containing a total of 1455 apples. Results showed that, by combining instance segmentation with SfM the system performance increased from an F1-score of 0.816 (2D fruit detection) to 0.881 (3D fruit detection and location) with respect to the total amount of fruits. The main advantages of this methodology are the reduced number of false positives and the higher detection rate, while the main disadvantage is the high processing time required for SfM, which makes it presently unsuitable for real-time work. From these results, it can be concluded that the combination of instance segmentation and SfM provides high performance fruit detection with high 3D data precision. The dataset has been made publicly available and an interactive visualization of fruit detection results is accessible at http://www.grap.udl.cat/documents/photogrammetry_fruit_detection.html.

Improved Kiwifruit Detection Using Pre-Trained VGG16 With RGB and NIR Information Fusion

This study presents a novel method to apply the RGB-D (Red Green Blue-Depth) sensors and fuse aligned RGB and NIR images with deep convolutional neural networks (CNN) for fruit detection. It aims to build a more accurate, faster, and more reliable fruit detection system, which is a vital element for fruit yield estimation and automated harvesting. Recent work in deep neural networks has led to the development of a state-of-the-art object detector termed Faster Region-based CNN (Faster R-CNN). A common Faster R-CNN network VGG16 was adopted through transfer learning, for the task of kiwifruit detection using imagery obtained from two modalities: RGB (red, green, blue) and Near-Infrared (NIR) images. Kinect v2 was used to take a bottom view of the kiwifruit canopy's NIR and RGB images. The NIR (1 channel) and RGB images (3 channels) were aligned and arranged side by side into a 6-channel image. The input layer of the VGG16 was modified to receive the 6-channel image. Two different fusion methods were used to extract features: Image-Fusion (fusion of the RGB and NIR images on input layer) and Feature-Fusion (fusion of feature maps of two VGG16 networks where the RGB and NIR images were input respectively). The improved networks were trained end-to-end using back-propagation and stochastic gradient descent techniques and compared to original VGG16 networks with RGB and NIR image input only. Results showed that the average precision (APs) of the original VGG16 with RGB and NIR image input only were 88.4% and 89.2% respectively, the 6-channel VGG16 using the Feature-Fusion method reached 90.5%, while that using the Image-Fusion method reached the highest AP of 90.7% and the fastest detection speed of 0.134 s/image. The results indicated that the proposed kiwifruit detection approach shows a potential for better fruit detection.

Faster R-CNN for multi-class fruit detection using a robotic vision system

An accurate and real-time image based multi-class fruit detection system is important for facilitating higher level smart farm tasks such as yield mapping and robotic harvesting. Robotic harvesting can reduce the costs of labour and increase fruit quality. This paper proposes a deep learning framework for multi-class fruits detection based on improved Faster R-CNN. The proposed framework includes fruits image library creation, data argumentation, improved Faster RCNN model generation, and performance evaluation. This work is a pioneer to create a multi-labeled and knowledge-based outdoor orchard image library using 4000 images in the real world. Also, improvement of the convolutional and pooling layers is achieved to have a more accurate and faster detection. The test results show the proposed algorithm has achieved higher detecting accuracy and lower processing time than the traditional detectors, which has excellent potential to build an autonomous and real-time harvesting or yield mapping/estimation system.

DeepFruits: A Fruit Detection System Using Deep Neural Networks.

This paper presents a novel approach to fruit detection using deep convolutional neural networks. The aim is to build an accurate, fast and reliable fruit detection system, which is a vital element of an autonomous agricultural robotic platform; it is a key element for fruit yield estimation and automated harvesting. Recent work in deep neural networks has led to the development of a state-of-the-art object detector termed Faster Region-based CNN (Faster R-CNN). We adapt this model, through transfer learning, for the task of fruit detection using imagery obtained from two modalities: colour (RGB) and Near-Infrared (NIR). Early and late fusion methods are explored for combining the multi-modal (RGB and NIR) information. This leads to a novel multi-modal Faster R-CNN model, which achieves state-of-the-art results compared to prior work with the F1 score, which takes into account both precision and recall performances improving from to for the detection of sweet pepper. In addition to improved accuracy, this approach is also much quicker to deploy for new fruits, as it requires bounding box annotation rather than pixel-level annotation (annotating bounding boxes is approximately an order of magnitude quicker to perform). The model is retrained to perform the detection of seven fruits, with the entire process taking four hours to annotate and train the new model per fruit.

An Embedded Real-Time Red Peach Detection System Based on an OV7670 Camera, ARM Cortex-M4 Processor and 3D Look-Up Tables

This work proposes the development of an embedded real-time fruit detection system for future automatic fruit harvesting. The proposed embedded system is based on an ARM Cortex-M4 (STM32F407VGT6) processor and an Omnivision OV7670 color camera. The future goal of this embedded vision system will be to control a robotized arm to automatically select and pick some fruit directly from the tree. The complete embedded system has been designed to be placed directly in the gripper tool of the future robotized harvesting arm. The embedded system will be able to perform real-time fruit detection and tracking by using a three-dimensional look-up-table (LUT) defined in the RGB color space and optimized for fruit picking. Additionally, two different methodologies for creating optimized 3D LUTs based on existing linear color models and fruit histograms were implemented in this work and compared for the case of red peaches. The resulting system is able to acquire general and zoomed orchard images and to update the relative tracking information of a red peach in the tree ten times per second.

4230 温州ミカン果皮に含まれる蛍光物質の特定と蛍光画像を利用した腐敗果実検出システム(S68 農業ロボット,S68 農業ロボット)

4230 温州ミカン果皮に含まれる蛍光物質の特定と蛍光画像を利用した腐敗果実検出システム(S68 農業ロボット,S68 農業ロボット)

Relationship between reflectance spectra of host plant surfaces and visual detection of host fruit by Rhagoletis pomonella flies

ABSTRACT. Within host trees, male and female Rhagoletis pomonella (Walsh) flies locate visually individual fruit of apple and hawthorn, which are sites of mating and oviposition. By measuring the diffuse reflectance spectra of both fruit and foliage and by using artificially pigmented natural fruit and artificial fruit mimics, we show that fruit hue is not as important in R. pomonella fruit detection as is intensity contrast of dark fruit against a bright background of light transmitted through foliage or skylight. In discussing the fruit detection system of R. pomonella, we compare it to that of vertebrate fruit consumers and seed dispersers.