Occlusion-aware fruit segmentation in complex natural environments under shape prior

Abstract

Widespread occlusion in complex natural environments severely restricts the accurate segmentation and localization of approximately spherical fruits by existing automatic picking devices, resulting in the confinement of efficient automatic picking to a controlled laboratory environment. While occlusion can be mitigated to some extent by optimizing annotations and refining the algorithm, the absence of partial fruit information due to occlusion may still result in misplacement or incorrect localization of the picking point. In this study, by introducing approximately spherical fruit shape priors, the widespread partial occlusion, slice occlusion, and self-occlusion found in complex natural environments can be effectively addressed. First, with the help of a boundary interest point–based fruit region contour segmentation method, the proposed method can achieve more accurate fruit segmentation in a complex environment. Self-occlusion can be mitigated with better separation and accurate localization for under-segmented overlapping fruit regions, which is achieved in this study through a local minima point–based fruit region contour segmentation method. This research demonstrates that occlusion-aware fruit segmentation can be achieved simply by introducing the approximately spherical fruit shape priors that are easily available in real production. Additional experiments are also performed to illustrate that the shape priors can be flexibly adjusted to ensure the generalizability of the proposed method to different species of fruits. The proposed method, which utilizes only plant phenotypic information and refrains from reliance on additional training data or equipment, can effectively solve the ubiquitous occlusion problem in complex natural environments and is important for future large-scale automatic fruit picking in smart agriculture.