An efficient method for creating benchmark classifications for automatic workpiece classification systems

Abstract

The usefulness of an automatic workpiece classification system depends primarily on the extent to which its classification results are consistent with users' judgments. Thus, to evaluate the effectiveness of an automatic classification system it is necessary to establish classification benchmarks based on users' judgments. Such benchmarks are typically established by having subjects perform pair comparisons of all workpieces in a set of sample workpieces. The result of such comparisons is called a full-data classification. However, when the number of sample workpieces is very large, such exhaustive comparisons become impractical. This paper proposes a more efficient method, called lean classification, in which data on comparisons between the samples and a small number of typical workpieces are used to infer the complete classification results. The proposed method has been verified by using a small set of 36 sample workpieces and by computer simulation with medium to large sets of 100 to 800 sample workpieces. The results reveal that the method could produce a classification that was 71% consistent with the full-data classification while using only 10% of the total data.