PlantNet: A dual-function point cloud segmentation network for multiple plant species

Abstract

The accurate plant organ segmentation is crucial and challenging to the quantification of plant architecture and selection of plant ideotype. The popularity of point cloud data and deep learning methods make plant organ segmentation a feasible and cutting-edge research. However, current plant organ segmentation methods are specially designed for only one species or variety, and they rarely perform semantic segmentation (stems and leaves) and instance segmentation (individual leaf) simultaneously. This study innovates a dual-function deep learning neural network (PlantNet) to realize semantic segmentation and instance segmentation of two dicotyledons and one monocotyledon from point clouds. The innovations of the PlantNet include a 3D Edge-Preserving Sampling (3DEPS) strategy for preprocessing input points, a Local Feature Extraction Operation (LFEO) module based on dynamic graph convolutions, and a semantic-instance Feature Fusion Module (FFM). The semantic segmentation results of tobacco, tomato, and sorghum in average Precision, Recall, F1-score, and IoU reached 92.49%, 92.04%, 92.13%, and 85.86%, respectively; and the instance segmentation results in the mean precision (mPrec), the mean recall (mRec), the mean coverage (mCov), and the mean weighted coverage (mWCov) reached 83.30%, 74.08%, 78.62%, and 84.38%, respectively. The PlantNet outperformed state-of-the-art deep learning networks including PointNet, PointNet++, SGPN, and ASIS, which achieved an average improvement of 5.56%, 3.58%, 4.78%, and 6.74% in Precision, Recall, F1-score, IoU on semantic segmentation, and an average improvement of 22.18%, 16.37%, 14.13%, and 13.35% in mPrec, mRec, mCov, and mWCov on instance segmentation. In addition, the effectiveness of 3DEPS, sub-modules, and the new loss function were verified separately by the ablation analysis, in which the removal of any of them can result in a segmentation performance decline of up to 2.0% on average quantitative measures. This study may contribute to the development of plant phenotype extraction, ideotype selection, and intelligent agriculture.