Segmenting ears of winter wheat at flowering stage using digital images and deep learning

Abstract

Segmenting ears of winter wheat from canopy images was considered to be an important procedure prior to the extraction of related traits. Current segmentation method based on computer vision was susceptible to noise, which is limited in practical applications. In this study, a two-stage segmentation method for ears of winter wheat based on digital images of unit ground area and the state-of-the-art deep learning techniques was proposed. In the coarse segmentation stage, a deep convolutional neural network (DCNN) was constructed to classify the superpixels generated by entropy rate superpixel algorithm, achieving the coarse results. In the fine segmentation stage, a fully convolutional network (FCN) allowing pixel-wise semantic segmentation was constructed to eliminate the non-ear pixels in the coarse results. To compare the results of the proposed two-stage segmentation method, conventionally adopted methods for image segmentation were used. Results showed that the proposed two-stage segmentation method was able to accurately segmenting ears of winter wheat from canopy images captured at flowering stage (Qseg = 0.7197, F1 score = 83.70%, SSIM = 0.8605), outperforming the other compared methods. Generalization tests were conducted to evaluate the utility of the proposed two-stage segmentation method. Results showed that the two-stage segmentation method was still capable of accurately segmenting ears of winter wheat, even though the performance slightly decreased. Change of winter wheat cultivar and lack of descriptive information were two factors that could degrade the performance of the two-stage segmentation method. Tests of the methods on Unmanned Aerial Vehicle (UAV) based RGB images showed the Fully Convolutional Network stride 8 predictions (FCN-8s) had a good chance to achieve satisfactory performances on UAV based canopy images.