Fruit Localization and Environment Perception for Strawberry Harvesting Robots

Yuanyue Ge,Ya Xiong,Gabriel Lins Tenorio,Pal Johan From

doi:10.1109/access.2019.2946369

Abstract

This work presents a machine vision system for the localization of strawberries and environment perception in a strawberry-harvesting robot for use in table-top strawberry production. A deep convolutional neural network for segmentation is utilized to detect the strawberries. Segmented strawberries are localized through coordinate transformation, density base point clustering and the proposed location approximation method. To avoid collisions between the gripper and fixed obstacles, the safe manipulation region is limited to the space in front of the table and underneath the strap. Therefore, a safe region classification algorithm, based on Hough Transform algorithm, is proposed to segment the strap masks into a belt region in order to identify the pickable strawberries located underneath the strap. Similarly, a safe region classification algorithm is proposed for the table, to calculate its points in 3D and fit the points onto a 3D plane based on the 3D point cloud, so that pickable strawberries in front of the table can be identified. Experimental tests showed that the algorithm could accurately classify ripe and unripe strawberries and could identify whether the strawberries are within the safe region for harvesting. Furthermore, harvester robot's optimized localization method could accurately locate the strawberry targets with a picking accuracy rate of 74.1% in modified situations.

Highlights

IntroductionDeep Convolutional Neural Networks (CNN) have greatly improved the performance of image processing, since the emergence of AlexNet, proposed by Krizhevsky et al [4] and the numerous other detection CNN subsequently developed, some of which have been utilized for the detection of crops and fruits
Machine vision is an essential element in agricultural robots
Deep Convolutional Neural Networks (CNN) have greatly improved the performance of image processing, since the emergence of AlexNet, proposed by Krizhevsky et al [4] and the numerous other detection CNN subsequently developed, some of which have been utilized for the detection of crops and fruits

Summary

Introduction

Deep Convolutional Neural Networks (CNN) have greatly improved the performance of image processing, since the emergence of AlexNet, proposed by Krizhevsky et al [4] and the numerous other detection CNN subsequently developed, some of which have been utilized for the detection of crops and fruits. Examples of such networks include You Only Look Once (YOLO), proposed by Redmon et al [5], Single Shot Detector (SSD), proposed by Liu et al [6] and the Region-based Convolutional Neural Network (Faster R-CNN), proposed by Girshick [7]. Bargoti and Underwood [9] adopted the same network to detect apples and mangoes, further improving its detection performance through data augmentation

Objectives

Methods

Findings

Conclusion