A system for classifying vegetative structures on coffee branches based on videos recorded in the field by a mobile device

Abstract

An expert system development to classify six coffee vegetative structures.A non-destructive system based on a smartphone camera and computer vision techniques.A system robust to illumination and occlusion changes using 2D and 3D features. As a drink, coffee is one of the most in demand products worldwide; as an agricultural product, it requires non-destructive tools for its monitoring and control. In order to create a non-destructive method which can be used in the field, a system was developed to find and classify six types of vegetative structures on coffee branches: leaves, stems, flowers, unripe fruits, semi-ripe fruits, and ripe fruits. Videos were obtained from 12 coffee branches in field conditions, using the rear camera of a mobile device. Approximately 90 frames, those which had the most information from the scene, were selected from each video. Next, a three-dimensional (3D) reconstruction was generated using the Structure from Motion (SfM) and Patch-based Multi-view Stereo (PMVS) techniques for each branch. All acquired images were manually recorded, and a Ground Truth point cloud was generated for each branch. The generated point clouds were filtered using a statistical outliers filter, in order to eliminate noise generated in the 3D reconstruction process. The points that were located in the deepest part were considered to be the scene background, and were removed using a band-pass filter. Point clouds were sub-sampled using a VoxelGrid filter, to reduce the number of points to 50% and therefore reduce computation time of the processes that followed. Various two-dimensional (2D) and 3D features were taken from the point clouds: 11 based on RGB, Lab, Luv, YCbCr, and HSV color space, four based on curvatures, and the remaining two based on shape and curvedness indexes. A Support Vector Machine (SVM) was trained with the previously mentioned features by using eight branches for the training stage, and four branches for the validation stage. Experimental results showed a precision of 0.82 and a recall of 0.79, when classifying said vegetative structures. The proposed system is economical, as only a mobile device is needed to obtain information. Remaining system processes were performed offline. Additionally, the system developed was not affected by changes in lighting conditions, when recording videos on a coffee plantation.