• A lightweight fruit phenotypic processing network is proposed to solve the complex calculation of the fruit features recognition in real scenes. • By combining 3D-SPP and global feature feedback, a novel fruit segmentation network is designed to improve the segmentation integrity. • Based on preprocessing and 3D STN, LFPNet is proved to be generalization in directly processing the disordered fruit scenes without ideal labeled. • Our LFPNet has only 1.17 M parameters and improves 2.15% on mIoU, 3.175% on OA, 3.45% on mAcc. 3D point cloud reconstruction, as the key technology to obtain high-throughput fruit phenotypic data, has solved the problems caused by complex environments, high fruit similarity, and the lack of public datasets suitable for fruit characterization. However, in the process of identifying and segmenting fruit data from point cloud, the existing network architectures lead to problems such as classification error, incomplete segmentation and low efficiency. In this paper, we introduce LFPNet, a novel and efficient lightweight neural network that directly consumes fruit point clouds in the real scene. Our network mainly has the following three advantages: 1) The introduction of voxel-filter based down-sampling preprocessing can help to avoid classification error caused by invalid noise interference. 2) A 3D STN is designed to solve the lack of spatial invariance in convolutional neural network (CNN) when calculating and analyzing fruit point clouds. 3) By introducing spatial pyramid pooling and combining local and global features, a fruit segmentation network is built to improve the integrity of segmentation in fruit scenes. Experimental results show that our LFPNet performs as well as or better than most of its peers in terms of classification accuracy and segmentation integrity.