Three-dimensional quantification of apple phenotypic traits based on deep learning instance segmentation

Abstract

Estimating three-dimensional (3D) fruit-level phenotypic traits of apple trees can potentially improve apple orchard breeders' management strategy. However, the phenotypic traits of apple fruits (including the quantity, 3D distribution, and volume of apples) constitute important parameters that influence yield but are difficult to quantify manually. Therefore, it is necessary to effectively and efficiently quantify apple phenotyping to monitor apple yield and support a better management system. This study developed a novel method for extracting individual 3D apple traits and 3D mapping for three apple training systems. The 3D point cloud of apples was reconstructed from multi-view images collected via a multi-camera system-based unmanned-aerial vehicle. Individual apples in the 3D point cloud were extracted via a 3D instance segmentation algorithm that included generalized sparse convolutional neural networks, a weighted discriminative loss function, and a varying density-based 3D clustering method. The developed apple trait extraction algorithm can help compute the position and volume of an individual apple. The R2 values with the average weighted-mean-absolute-percentage error (WMAPE) of apple counting and apple volume estimation were 0.84–0.99 (VMAPE = 3.41–12.75%) and 0.84–0.90 (WMAPE = 4.74–7.03%), respectively. The 3D spatial and volumetric distribution of apples were obtained and analyzed. This study developed an effective method combining 3D photography and 3D instance segmentation that can accurately estimate individual apple phenotypic traits from different types of apple training systems in orchards and can also be utilized for the analysis of other fruit traits.