Method for assessing the quality of data used in evaluating the performance of recognition algorithms for fruits and vegetables

Abstract

In studies on agricultural robot vision systems, data used to evaluate algorithm performance, such as successful recognition rates, vary because of various factors. If the variation is too large, representation of the actual performance of algorithms by the data is bound to be poor. Here we present a method for analysing the quality of data used to evaluate the performance of a recognition algorithm for occluded tomatoes based on measurement system analysis. The measurement system included a soft measurement tool (a counting method for the number of successful recognitions), appraisers, measured objects (recognition results of 300 occluded tomato images), the usage method for the soft measurement tool and measurement environments. The measurement system was analysed on the basis of its repeatability and reproducibility. Repeatability and reproducibility were both evaluated based on Fleiss's Kappa values, free-marginal multirater Kappa values and Kendall coefficients. Test results showed that repeatability was excellent or fair to good based on Fleiss's Kappa values and excellent based on free-marginal multirater Kappa values and Kendall coefficients for the three appraisers. Further improvement in the soft type of measurement tool is necessary. Reproducibility was fair to good with Fleiss's Kappa values and free-marginal multirater Kappa values, and good with Kendall coefficients. Large values of measured feature resulted in inferior repeatability and reproducibility.