Monitoring of grain crops nitrogen status from uav multispectral images coupled with deep learning approaches

Abstract

Effective nitrogen nutrition is vital for better crop yield. In order to get the maximum yield from a field, nutrition must be spread evenly among all crops. Therefore, this paper proposes a combination of deep learning image segmentation methods to monitor nutrition across an agricultural field and detect areas with shortages of nutrients. In particular, the authors consider the applicability of five state-of-the-art neural network architectures based on U-Net to solve the nitrogen level rate segmentation problem for crops on an orthophotomap. Training, effectiveness assessment, and applicability of these neural network models are carried out by the authors on their own multi-datasets, collected by using UAS (Geoscan 401) at the Agrophysical Research Institute (ARI) experimental biopolygon for 2020–2021. The survey was performed using a MicaSense RedEdge-MX multispectral camera (5 channels in total). The total size of the collected dataset is more than 20 thousand images of two different agricultural fields (with a total area of about 62 ha). On each field, there are six test areas with known nitrogen nutrition levels (founded by agronomists). Images of these test areas are used for data augmentation and training of the above-mentioned neural network models (U-Net, Attention U-Net, R2-UNet, Attention R2-Unet, and U-Net3+). Also, in this research, an experiment was conducted to evaluate the influence of the choice of different bands of field images on the accuracy of the considered segmentation methods. The experiment showed that among all models, Attention R2U-Net (t2) proved to be more robust and reliable for different kinds of crops (accuracy 97.59–99.96%). The authors also evaluated the impact of using different combinations of image bands (such as RGB, RedEdge, NearIR, and NDVI) on the segmentation accuracy of the neural network model. The combination of RGB, NearIR, and NDVI channels allowed for the high values of all 8 metrics used in this research (0.41–1.77% more than the standard combination of RGB bands). The use of the RedEdge band has a significant negative impact on the quality of segmentation of the nitrogen level in the agricultural field. The proposed method based on Attention R2U-Net (t2) and a combination of RGB, NearIR, and NDVI bands is stable for different types of agricultural landscapes and can help to improve crop nutrition and yield.