A new evaluation measure for learning from imbalanced data

Abstract

Recently, researchers have shown that the Area Under the ROC Curve (AUC) has a serious deficiency since it implicitly uses different misclassification cost distributions for different classifiers. Thus, using the AUC can be compared to using different metrics to evaluate different classifiers [1]. To overcome this incoherence, the H measure was proposed, which uses a symmetric Beta distribution to replace the implicit cost weight distribution in the AUC. When learning from imbalanced data, misclassifying a minority class example is much more serious than misclassifying a majority class example. To take different misclassification costs into account, we propose using an asymmetric Beta distribution (B42) instead of a symmetric one. Experimental results on 36 imbalanced data sets using SVMs and logistic regression show that B42 is a good choice for evaluating on imbalanced data sets because it puts more weight on the minority class. We also show that balanced random undersampling does not work for large and highly imbalanced data sets, although it has been reported to be effective for small data sets.