RoseSegNet: An attention-based deep learning architecture for organ segmentation of plants

Abstract

An important component for the advancement of plant breeding, genetics, and genomics research is the rapid and accurate measurement of phenotypic traits of large plant populations. The phenotypic data that are of interest can be at multiple levels of plant organization including organ-level geometric characteristics as well as the spatial organization of the organs. 3D computer vision enabling 3D geometry acquisition and processing promises to supply fast, automated phenotypic data collection. One important component of the processing pipeline is the segmentation of the plant into its structural components, such as leaves, stems, and flowers. In this paper, a novel 3D point-based deep learning network, namely RoseSegnet, is proposed for segmentation of point clouds of rosebush plants to their organs. The network is equipped with two attention-based modules, one for extracting contextual features at the encoder phase, another for feature propagation at the decoder phase. The network processes regions of points in a hierarchical manner, where at each level, point features are aggregated using attention-based operators. The aggregation is performed by incorporating point relations both within and between the receptive fields, defined by the hierarchical organization of points. RoseSegNet outperforms the widely-used architecture PointNet++ by 4% in terms of MIoU on the publicly available ROSE-X data set. Also, it is demonstrated that introducing local surface features together with the spatial coordinates of each 3D point at the input level boosts the segmentation performance of both networks by 9% in terms of MIoU . • A new 3D point-based deep learning architecture for organ segmentation of plants. • Attention-based modules for aggregation and propagation of features. • Exploiting point interactions within and between regions at multiple scales. • Incorporation of local surface features to the deep learning framework.