Abstract
Calibration is a technique used to obtain accurate probability estimation for classification problems in real applications. Class imbalance can create considerable challenges in obtaining accurate probabilities for calibration methods. However, previous research has paid little attention to this issue. In this paper, we present an experimental investigation of some prevailing calibration methods in different imbalance scenarios. Several performance metrics are considered to evaluate different aspects of calibration performance. The experimental results show that the performance of different calibration techniques depends on the metrics and the degree of the imbalance ratio. Isotonic Regression has better overall performance on imbalanced datasets than parametric and other complex non-parametric methods. However, it performs unstably in highly imbalanced scenarios. This study provides some insights into calibration methods on imbalanced datasets, and it can be a reference for the future development of calibration methods in class imbalance scenarios.
Highlights
In many real-world classification applications, it is crucial to obtain accurate class probability estimation [1]
Each table indicates the performance of four calibration methods in terms of one metric
If there is no significant difference in terms of the Iman-Davenport test among the calibration methods for a classifier based on one metric, the method with the best rank is marked with a star
Summary
In many real-world classification applications, it is crucial to obtain accurate class probability estimation [1]. Many popular classifiers such as support vector machine, boosted decision tree and even modern deep neural networks cannot produce accurate class probability estimations [2], [3]. To deal with this issue, two ways have been developed. The other way is to use post-processing calibration method, which attempts to transform the output of classifiers to well-calibrated probabilities [4]. We perform an experimental investigation of calibration techniques on imbalanced datasets. The experimental results give some insights of calibration on imbalanced datasets and provide useful guidelines for the future development of calibration techniques.
Sign-in/Register to view highlights & summary Sign-in/Register to access full text options 
Talk to us
Join us for a 30 min session where you can share your feedback and ask us any queries you have
Schedule a callDisclaimer: All third-party content on this website/platform is and will remain the property of their respective owners and is provided on "as is" basis without any warranties, express or implied. Use of third-party content does not indicate any affiliation, sponsorship with or endorsement by them. Any references to third-party content is to identify the corresponding services and shall be considered fair use under The CopyrightLaw.
