Size Estimation of Mango Using Mask-RCNN Object Detection and Stereo Camera for Agricultural Robotics

Abstract

—In agricultural robotics, to perform automatic harvesting task, it is important that the size of thetarget can be reasonably approximated.Stereo cameras can be used as a sensor to detect the targeted object and approximate its size. In this paper we present two methods for estimating the size of mangoes: pinhole model and ellipsoid model methods, using data obtained from a stereo camera.Object detection scheme using Mask-RCNN deep learning neural network on an RGB image and point cloud data from stereo camera are employed. The pinhole model method assumes simple triangular projection of the object onto the image plane.It makes use of the bounding boxes predicted by the neural network and the 3-D point cloud data. The ellipsoid model method, on the other hand, uses three-dimensional point cloud data obtained by processing the images from the stereo camera.It assumes that mango shape can be approximated as an ellipsoid and uses the predicted masked region and its corresponding point cloud data to fit an ellipsoid whose center and axes are extracted to estimate the position and the size of the mango, respectively. In the scope of this study where the full shape of mango can be seen in the image view, results from experiments suggest that the simpler pinhole model can effectively estimate the size of the mango with greater accuracy than the ellipsoid model. Keywords—Object Detection; Agricultural Robotics; Artificial Intelligent