Cal-Net: Jointly Learning Classification and Calibration On Imbalanced Binary Classification Tasks

Abstract

Datasets in critical domains are often class imbalanced, with a minority class far rarer than the majority class, and classification models face challenges to produce calibrated predictions on these datasets. A common approach to address this issue is to train classification models in the first step and subsequently use post-processing parametric or non-parametric calibration techniques to re-scale the model's outputs in the second step without tuning any underlying parameters in the model to improve calibration. In this study, we have shown that these common approaches are vulnerable to class imbalanced data, often producing unstable results that do not jointly optimize classification or calibration performance. We have introduced Cal-Net, a “self-calibrating” neural network architecture that simultaneously optimizes classification and calibration performances for class imbalanced datasets in a single training phase, thereby eliminating the need for any post-processing procedure for confidence calibration. Empirical results have shown that Cal-Net outperforms far more complex neural networks and post-processing calibration techniques in both classification and calibration performances on four synthetic and four benchmark class imbalanced binary classification datasets. Furthermore, Cal-Net can readily be extended to more complicated learning tasks, online learning and can be incorporated in more complex architectures as the final state.