Multi-view gripper internal sensing for the regression of strawberry ripeness using a mini-convolutional neural network for robotic harvesting

Abstract

The capability of robotic fruit-harvesting systems to accurately assess the ripeness of fruits is crucial for fulfilling the diverse standards of the market and the preferences of consumers. Existing studies involving fruit ripeness estimation mainly focus on detecting of ripe strawberries as one class or classifying of ripeness into several stages, such as overripe, ripe, and unripe. Current harvesting robots also lack the ability to determine the ripeness of the back of fruit with respect to the robots. This paper proposes a lightweight convolutional neural network (CNN) regression model for ripeness quantification based on the internal image sensing system of the gripper with full-view coverage of the fruit for strawberry-harvesting robots. A gripper internal sensing system was developed using two RGB cameras that could provide full-view fruit coverage for a more accurate estimation of fruit ripeness. Four base CNN networks capable of feature learning were used for feature extraction, followed by the utilization of newly added dense layers that produced a regressed value to represent the strawberry ripeness. However, the base networks were cumbersome and relatively slow due to their complex structures. To simplify this, a new MiniNet with fewer convolutional layers was proposed to reduce the model size and inference time. All models were trained via two loss functions, mean square error and Huber loss. The results showed that the models trained via Huber loss performed better. An Xception model trained on Huber loss showed the best performance with a mean absolute error of 4.0% and an average inference time of 42.5 ms. Of all the models, the new MiniNet was the most lightweight and fastest model while maintaining high performances (an mean absolute error of 4.8% and an inference time of 6.5 ms for Huber loss trained model). The proposed method may be also applicable to other fruit-harvesting systems.