A single view leaf reconstruction method based on the fusion of ResNet and differentiable render in plant growth digital twin system

Abstract

• A single view leaf reconstruction based on deep learning for plant growth digital twin system is proposed. • The model provides a feature extraction network with RestNet18 to improve pixel2mesh’ feature extraction capabilities. • The model provides a way to optimize the shape of the leaf mesh with differentiable rendering. In modern agriculture, plant growth digital twin system helps breeders monitor plant growth, increase yield, and provide growth management advice. Research on the single view leaf 3D reconstruction in digital twin systems has achieved relative success. However, in traditional single-view reconstruction algorithms, the leaf reconstruction often contains the problems of low precision, achieving complexity, and slow speed, making it difficult for recovering three-dimensional information about leaves. Consequently, the reconstruction precision is significantly reduced, which further affects the accuracy of single-view leaf 3D reconstruction. In response to this problem, this study proposed a single-view leaf reconstruction approach in plant growth digital twin systems based on deep learning. The method in this paper mainly fuses the advantages of ResNet and differentiable rendering, and the model is used for further enhancing feature extraction capability and reconstruction precision. Finally, the experiment presented in this paper suggests that the method allows for the 3D reconstruction of plant leaves with different shapes using a single view. Moreover, the experiment results show that the F-Score, CD, EMD reached 76.192, 0.808, and 3.567. Compared with other models, the proposed model in this study has higher reconstruction accuracy, 3D evaluation indicators, and prediction results, providing important ideas and methods for recovering the leaves from a single view in a plant growth digital twin system.