Standardization of Criteria across Multiple Evaluators to Detect Objects

Abstract

For a typical object detection task with machine learning technique, there has an absolute correct label on which a mechanical model is constructed. However, there are many tasks in which labels vary with evaluators because they have different criteria for discrimination. It happens when the detection criteria are vague and undefined among evaluators. This problem is avoided by constructing an individual model, but it is not recommended from a long-term perspective, because the model depends on a specific evaluator.   This paper proposes a method to standardize the evaluator's criteria. The method is verified in the image detection task of asteroid powders in paint material as an example of ambiguity in the criteria.   The performance and variance of detection with the proposed method are compared with conventional ones to evaluate whether it allows us to standardize the evaluators’ criteria. It turns out a clear reduction in the variance of evaluators’ detection results without significant degradation of the performance for all evaluators. It is confirmed this method standardizes the criteria across multiple evaluators. In addition, the paper discusses how to obtain manuals formally expressing the standardized criteria with text mining.