Performance visualization spaces for classification with rejection option

Abstract

The classification with reject option consists to train a classifier that rejects the examples when the confidence in its prediction is low. The objective is to improve the accuracy of the non-rejected examples and the reliability of the prediction. The performances of the reject classifiers depend on both the error rate and rejection rate. Since these two values are in opposition, we have to make a trade-off between them. This paper is focused on the visualization spaces the performances of the classifiers with rejection option. We analyze two common spaces, the ROC space and the error-rejection (ER) space, then we propose a new space: the cost-reject (CR) space. We show that the ROC space is the less convenient space to represent the performances of the reject classifier. However, it can be recommended for classification problems where the importance of the two classes is different. For the ER space, we point out that the linear interpolation that is commonly used to draw the error-reject curve is not correct and leads to an overestimation of the classifier performances. From the definition of the condition error and rejection rate, we propose a new interpolation of the error-rejection curve that is unbiased. We introduce a new visualization space called the cost reject space. The CR space plots the normalized classification cost in function on the normalized rejection cost. The performance of a classifier is represented in this space by a line. The three visualization spaces are compared on problems of classification algorithms comparison. The advantages and drawbacks of each spaces are discussed and some recommendations are provided in the conclusion.