Beyond top‐N accuracy indicator: a comprehensive evaluation indicator of CNN models in image classification

Abstract

Nowadays, a large number of deep convolutional neural network (CNN) models are applied to image classification tasks. However, the authors find that the most widely used evaluation indicator, the Top‐N Accuracy indicator, cannot discriminate these models effectively. In this study, they propose a new indicator called Maximum‐Spanning‐Confusion‐Tree indicator to solve this problem. The Maximum‐Spanning‐Confusion‐Tree indicator is computed based on the hierarchical structure of the Maximum Spanning Confusion Tree of the deep CNN model on the dataset and reflect the ability of deep CNN models to discriminate confused categories in the dataset. The hierarchical structure of the Maximum Spanning Confusion Tree can reveal the confused category set of one selected category in the dataset efficiently and flexibly. Experiments show that they can discriminate ten different deep CNN models more accurately with the Maximum Spanning Confusion Tree indicator than the Top‐N Accuracy indicator and the Maximum Spanning Confusion Tree intuitively shows the distribution of confused category sets in the dataset so they can find out the weakness of deep CNN models effectively.