Window Zooming–Based Localization Algorithm of Fruit and Vegetable for Harvesting Robot

Chenglin Wang,Yunchao Tang,Tianhong Luo,Lijun Zhao,Xiangjun Zou

doi:10.1109/access.2019.2925812

Abstract

Localization of fruit and vegetable is of great significance to fruit and vegetable harvesting robots and even harvesting industries. However, uncontrollable factors, such as varying illumination, random occlusion, and various surface color and texture, constrain the localization of fruit and vegetable using the vision imaging technology under unconstructed environment. Our previous studies have developed various methods (illumination normalization, features-based classification, etc.) to localize a certain kind of fruit or vegetable using the binocular stereo vision. However, the localization of the multiple fruit and vegetable still faces challenges due to the uncontrollable factors. In order to address this issue, this study proposed an intelligent localization method of targets in fruit and vegetable images acquired by the two charge-coupled device (CCD) color cameras under unstructured environment. The method utilized the Faster region-based convolutional neural network (R-CNN) model to recognize the fruit and vegetable. Based on the recognition results, a window zooming method was proposed for the matching of the recognized target. Finally, the localization of the target was completed by calculating the three-dimensional coordinates of the matched target using the triangular measurement principle. The experimental results showed that the proposed method could be robust against the influences of varying illumination and occlusion, and the average accurate recognition rate was 96.33% under six different conditions. About 93.44% of 1036 pairs of tested targets from unoccluded and partially occluded conditions were successfully matched. Localization errors had no significant difference and they were less than 7.5 mm when the measuring distance was between 300 and 1600 mm under varying illumination and partially occluded conditions.

Highlights

In recent years, numerous kinds of fruit and vegetable harvesting robots have been developed by researchers with an aim of rapid, automatic and effective implementing harvesting mission [1]–[7]
The vision imaging system is affected by the growth environment of fruit and vegetable such as intensity changes of illumination, random occlusion of surface and so on, which seriously affects the accuracy of localization of fruit and vegetable
In order to eliminate the influences caused by unstructured environments and improve the detection rate of fruits, we proposed a method for the recognition of multiple fruit and vegetable by using a Faster region-based convolutional neural network (R-convolutional neural network (CNN))

Summary

Introduction

Numerous kinds of fruit and vegetable harvesting robots have been developed by researchers with an aim of rapid, automatic and effective implementing harvesting mission [1]–[7]. Localization of fruit and vegetable including recognition and matching is the critical step of the automatic-based harvesting [8]. The associate editor coordinating the review of this manuscript and approving it for publication was Nan Liu. of fruit and vegetable under the natural environment can be captured by using the vision imaging system of the harvesting robot. The robot can harvest fruit and vegetable by following the guide of the information. The vision imaging system is affected by the growth environment of fruit and vegetable such as intensity changes of illumination, random occlusion of surface and so on, which seriously affects the accuracy of localization of fruit and vegetable. Many approaches based on the vision imaging technology have been proposed using

Objectives

Methods

Results

Discussion

Conclusion