PERFORMANCE OF HUMAN EXPERTS GRADING POT ROSES IN A COMMERCIAL ENVIRONMENT

Abstract

In Denmark, ornamental products are graded by manual labour. The task is very demanding regarding labour and training of the employees. This paper addresses the characteristics of manual grading of pot-roses, with its variance and bias of plant evaluation, within and between experts. Evaluations from three experts were gathered for 2800+ pot roses. From these 2800 five additional experts, resulting in a total of eight evaluations per plant evaluate plants 300. The evaluations were given in a natural work environment that means that the estimate of performance by the single expert should reflect the consistency to be expected of commercial products. The reason for this investigation is to establish knowledge of the current consistency of product grading, to evaluate future automatic grading systems, based on computer vision. A very important issue in relation to subjective evaluation is the determination of an appropriate correct classification of the products. Pros and cons of the majority voting scheme is considered in relation to pot plant grading. The performance of each individual expert is analysed with respect to the majority decision, and the analysis covers overall error rates, class-specific error rates, certainty of the expert statement and hit-rate. Expert statements are merged to form expert-panels. The theory of panels is explained and the performance is compared with the performance of an individual expert. Further, the maximum achievable classification rate is estimated based on expert-panels and the correlation between these. The single expert classification rate varies between 84% and 92%, but the errors are not equally distribution between classes. The performance of expert-panels formed by single expert statements varies between 90.9% and 98.6%, which is a significant improvement. A conservative estimate of the maximum achievable classification rate is between 98.5-99.5%. This implies that an almost perfect classifier can be developed, if the same level of information is available for an automatic grading system, as for the human grader. This paper identifies the lower qualities (B) as the most difficult to grade. It is uncertain whether the low performances of grade B plants are caused by the small number of plants or the fact that the variance of class B plants is much larger than that of high quality plants (A). It is also found that a considerable improvement can be obtained by employing expert panels. They can be implemented in an automatic grading system, providing a good knowledge base.