Three-dimensional reconstruction of cotton plant with internal canopy occluded structure recovery

Abstract

The inner leaves of crop canopies are obscured by outer branches and leaves, leading to information loss and observation difficulty of the occluded canopy structure using modern crop monitoring techniques. It has restricted the development of phenotypic analysis and precision agriculture. In this paper, we propose a neural network approach to reconstruct the occluded structure of crop canopies with an RGB-D sensor. Taking the cotton plant as the object of study, we propose a novel Cascade Leaf Segmentation and Completion Network (CLSCN) to reconstruct the occluded leaf images and propose a Fragmental Leaf Point–cloud Reconstruction Algorithm (FLPRA) to complete the missing point clouds. By combining the Instance Segmentation Network (ISN), Generative Adversarial Network (GAN) and Point-cloud Reconstruction Algorithm (PRA), the three-dimensional models of cotton plants with both completed internal and external structures of the canopy are smoothly reconstructed. Firstly, we collect a large number of leaf images and point clouds of cotton plants using an RGB-D sensor with the top view and construct a manually labeled cotton leaf dataset for training and evaluation. Secondly, a network named CLSCN is cascading constructed with an Instance Segmentation Network (ISN) and a Generative Adversarial Network (GAN), and the two parts of CLSCN are separately trained with our constructed dataset to output complete cotton leaves. Thirdly, with the fusion of the completed RGB images output by cascaded network segmentation and the point clouds captured by RGB-D sensor, the proposed FLPRA is used to filter, reconstruct, fuse and register the cotton canopy leaf point clouds, and to obtain the whole cotton canopy point-clouds with inner occluded structure recovery. Finally, the CLSCN and FLPRA are validated using the validation dataset of cotton leaf. The test results indicate that the front-end ISN of the proposed CLSCN can generate high-quality cotton leaf masks, with FID scores less than 35 and mIoU up to 84.65%. Additionally, the back-end GAN of CLSCN can complete the occluded leaves with an accuracy of over 94%. The reconstruction accuracy of the final three-dimensional model of the cotton canopy is as high as 82.70%. Therefore, the proposed neural network and algorithm effectively solve the problem of incomplete canopy point cloud caused by the occlusion of outer leaves and provide an effective way to recover the complete three-dimensional structure of crop canopy with internal occlusion. It is a meaningful theoretical and technical support to realize real-time crop status observation and precise field management in agriculture production.