Cascade-SORT: A robust fruit counting approach using multiple features cascade matching

Abstract

Estimation of fruit yield is of great importance to agricultural management and production decision-making. Fruit counting based on computer vision is faced with many challenges, particularly dense occlusion and difficult detection. To address the problems that exist in agricultural scenarios, we propose a fruit counting pipeline based on multiple features matching. Fruit counting is regarded as a multiple object tracking problem based on tracking-by-detection framework. The proposed method combines object detection with deep learning, Kalman filter, and cascade matching, which integrated motion and appearance features for frame-by-frame data association. Using the detection results of YOLO-v3, cascade matching is leveraged to associate detection bounding boxes with tracks. In cascade matching, the appearance features of fruit, Mahalanobis distance, and intersection over union metric were fused to match objects frame-by-frame. Mahalanobis distance is used to screen detection bounding boxes initially. Furthermore, the vector of locally aggregated descriptors image retrieval method is used to calculate the similarity of appearance between the two objects. In the final step of cascade matching, residual unmatched tracks and detection candidates are matched using intersection over union metric. Moreover, the Kalman filter is optimized for predicting the trajectories of undetectable objects to enhance the accuracy and robustness of fruit counting. In the experiments, the results of predicted fruit counting for camellia is 44 while the ground truth is 38 for a video. For apple counting, the total predicted number of fruits for three videos is 310 while the actual number is 292. And compared to the method of SORT, our method is better in counting accuracy, reduced the number of ID switches, and had more robustness when the detector performance degenerated. All the above mentioned metrics indicate that the proposed method is well performance in fruit counting regardless of whether the fruit is sparsely or densely grown.