Computer vision-based localisation of picking points for automatic litchi harvesting applications towards natural scenarios

Abstract

Locating picking points plays an important role in robotic litchi harvesting in orchards. However, the localisation accuracy can easily be affected by unstructured growing environments with variable illumination conditions and the unpredictable growing shapes of weight-bearing stems carrying litchi clusters. A computer-vision-based method is proposed to locate acceptable picking points for litchi clusters. To improve the illumination distributions of weakly illuminated images while leaving those of well-illuminated images unchanged, an iterative retinex algorithm was developed. Then, the litchi regions are segmented by combining red/green chromatic mapping and Otsu thresholding in the RGB colour space. By incorporating both local spatial and local hue-level intensity relationships, the hue component in the HSV colour space is re-constructed to filter the noise, and the stem regions are segmented in accordance with the distributed intervals of hue-level intensity. Finally, based on the connection and position relationship among the segmented litchi and stem regions, the structural distribution of the corners generated via Harris corner detection is analysed to locate the picking points. More than 97% of the pixels within litchi regions are correctly segmented after illumination compensation, and an improvement of 7% in the number of correctly segmented pixels within stem regions is achieved with the re-construction of the hue-level intensity. The overall performance of the proposed method was evaluated on test images with high illumination diversity, and accuracy greater than 83% in locating litchi picking points was achieved, showing better performance than three similar frameworks that do not incorporate our proposed procedures.