Instance segmentation of root crops and simulation-based learning to estimate their physical dimensions for on-line machine vision yield monitoring

Abstract

Modern agriculture is based on control and optimization, where monitoring is essential. But yield monitoring limited to spatial mapping of biomass is unsatisfactory for uniformity optimization. Acquiring this information as soon as during the harvest could improve efficiency of precision agriculture. Current machine vision solutions do not allow this for several reasons, starting with difficulties with recognition of heavily cluttered, often mutually occluding objects, or requirements for elaborate hardware. Acquiring images in on-line conditions often results in perspective distortion which makes measurement difficult. This is even more challenging in the case of root crops, as they are usually graded basing on their minimal diameter, which is not directly observable via monocular vision.We propose a method for developing yield monitoring systems that can estimate physical dimensions of crops directly on-line. In our approach, individual plants are segmented using Mask R-CNN, allowing clear separation even in heavily cluttered conditions and in the presence of occlusions. Then, a nonlinear regression model is used to predict the minimal diameter of each plant basing on their observed contours. This model is capable of jointly correcting the perspective distortion and estimate the non-observed dimension. Training this model requires per-object annotations with their true diameters. Since this information is very difficult to acquire for real crops, we build a simulation environment where we model the crops, eventually rendering a synthetic dataset to train the regression model on. The size estimation model transfers the knowledge from simulated data, correctly predicting the sizes of real plants. We demonstrate our method on the case of potatoes, but it can be applied to other crops as well.