Implementing Spatio-Temporal 3D-Convolution Neural Networks and UAV Time Series Imagery to Better Predict Lodging Damage in Sorghum

Sebastian Varela,Taylor L Pederson,Andrew D B Leakey

doi:10.3390/rs14030733

Sebastian Varela, Taylor L Pederson + Show 1 more

Open Access

https://doi.org/10.3390/rs14030733

Copy DOI

Abstract

Unmanned aerial vehicle (UAV)-based remote sensing is gaining momentum in a variety of agricultural and environmental applications. Very-high-resolution remote sensing image sets collected repeatedly throughout a crop growing season are becoming increasingly common. Analytical methods able to learn from both spatial and time dimensions of the data may allow for an improved estimation of crop traits, as well as the effects of genetics and the environment on these traits. Multispectral and geometric time series imagery was collected by UAV on 11 dates, along with ground-truth data, in a field trial of 866 genetically diverse biomass sorghum accessions. We compared the performance of Convolution Neural Network (CNN) architectures that used image data from single dates (two spatial dimensions, 2D) versus multiple dates (two spatial dimensions + temporal dimension, 3D) to estimate lodging detection and severity. Lodging was detected with 3D-CNN analysis of time series imagery with 0.88 accuracy, 0.92 Precision, and 0.83 Recall. This outperformed the best 2D-CNN on a single date with 0.85 accuracy, 0.84 Precision, and 0.76 Recall. The variation in lodging severity was estimated by the best 3D-CNN analysis with 9.4% mean absolute error (MAE), 11.9% root mean square error (RMSE), and goodness-of-fit (R2) of 0.76. This was a significant improvement over the best 2D-CNN analysis with 11.84% MAE, 14.91% RMSE, and 0.63 R2. The success of the improved 3D-CNN analysis approach depended on the inclusion of “before and after” data, i.e., images collected on dates before and after the lodging event. The integration of geometric and spectral features with 3D-CNN architecture was also key to the improved assessment of lodging severity, which is an important and difficult-to-assess phenomenon in bioenergy feedstocks such as biomass sorghum. This demonstrates that spatio-temporal CNN architectures based on UAV time series imagery have significant potential to enhance plant phenotyping capabilities in crop breeding and Precision agriculture applications.

Highlights

The overall accuracy (OA) of 2D-Convolution Neural Network (CNN) predictions of the lodging that occurred during the storm 82 days after planting (DAP) was very limited when analyzing images collected seven days prior to the lodging event used as a reference baseline (75 DAP; 0.61 to 0.73)
Modest gains were made in the qualitative task of lodging detection through the use of a 3D-CNN compared to a 2D-CNN, which had an overall accuracy of 0.79–0.88 depending on the number of spectral and geometric features of the input data
Unmanned aerial vehicle (UAV) remote sensing imaging that could be made with 3D-CNN versus 2D-CNN

Summary

Introduction

Lodging negatively affects yield in many crops [1,2,3]. This damage, defined as the displacement of plant stems from an upright position [4], is mainly induced by the strong winds of extreme weather events during the growing season of the crop. Lodging negatively impacts plant morphology [5], physiological processes [6], and growth [7], as well as impeding harvesting activities, to significantly reduce the final above-ground yield of the crop [8,9,10,11]. Due to its rapid vertical growth [12] and tall stature of up to 3-4 m, biomass-type Sorghum bicolor (L.)

Objectives

Results

Discussion

Conclusion