PST: Plant segmentation transformer for 3D point clouds of rapeseed plants at the podding stage

Abstract

Segmentation of plant point clouds to obtain high-precise morphological traits is essential for plant phenotyping. Although the fast development of deep learning has boosted much research on segmentation of plant point clouds, previous studies mainly focus on the hard voxelization-based or down-sampling-based methods, which are limited to segmenting simple plant organs. Segmentation of complex plant point clouds with a high spatial resolution still remains challenging. In this study, we proposed a deep learning network plant segmentation transformer (PST) to achieve the semantic segmentation of rapeseed plants point clouds acquired by handheld laser scanning (HLS) with the high spatial resolution, which can characterize the tiny siliques as the main traits targeted. PST is composed of: (i) a dynamic voxel feature encoder (DVFE) to aggregate the point features with the raw spatial resolution; (ii) the dual window sets attention blocks to capture the contextual information; and (iii) a dense feature propagation module to obtain the final dense point feature map. We then integrated PST with an instance segmentation head in the point grouping network (PointGroup) and developed PST-PointGroup (PG) to achieve the instance segmentation of the siliques. The results proved that PST and PST-PG achieved superior performance in semantic and instance segmentation tasks. For the semantic segmentation, the mean IoU, mean Precision, mean Recall, mean F1-score, and overall accuracy of PST were 93.96%, 97.29%, 96.52%, 96.88%, and 97.07%, achieving an improvement of 7.62, 3.28, 4.8, 4.25, and 3.88 percentage points compared to the second-best state-of-the-art network position adaptive convolution (PAConv). For instance segmentation, PST-PG reached 89.51%, 89.85%, 88.83% and 82.53% in mCov, mWCov, mPerc90, and mRec90, achieving an improvement of 2.93, 2.21, 1.99, and 5.9 percentage points compared to the original instance segmentation network PointGroup. This study extends the phenotyping of rapeseed plants in an end-to-end way and proves that the deep-learning-based point cloud segmentation method has a great potential for resolving dense plant point clouds with complex morphological traits.