Modeling realistic 3D agricultural vegetations using a photometric-based approach and its application to weed detection

Abstract

3D computer graphics is one of the major approaches to create synthetic images to train, evaluate, or validate computer vision systems. Attempts to utilize computer graphics in agriculture have shown great potential. However, the complexity of agricultural vegetations has been hindering the development of 3D models for agricultural applications. Here, a framework is developed to facilitate the synthesis of 3D agricultural vegetation scenes. The framework fully relies on photometric approaches, thus requiring no sophisticated equipment. It starts with an efficient method to acquire dual-faced leaf models with details of leaf geometry, light reflectance and light transmittance. A parametric L-system template is used to organize leaf models in a geometric arrangement that resembles real plants. Finally, a ray-tracing approach is adopted to produce high levels of visual realism. The robustness of the proposed framework is illustrated by training neural networks with rendered images for the detection of weeds, which are major pests threatening crop production. A considerable boost of detection performance is granted by the rendered images, as well as the ability for instance segmentation. The promising results obtained here open up several areas for future work, one of which is the development of publicly available 3D crop and weed databases.