How reliable is your reliability diagram?

Abstract

It is often necessary to evaluate probabilistic classifiers in terms of the quality of class probability estimates. A popular tool for assessing class probabilities is the reliability diagram, which is based on data binning. While the reliability diagram is visually appealing, it is difficult to statistically determine whether the probabilities are reliable. In this paper, we propose a standardized reliability diagram to assess a binary probabilistic classifier. The proposed method uses the transforms of the Poisson binomial distribution to the normal distribution. The results of the method provide valuable inferences over the (unscaled) reliability diagram. Moreover, we show that the assessment results may be undesirably dependent on the sample size in each bin. As a remedy, we also introduce an approach that chooses an appropriate number of bins for relatively consistent test results regardless of the sample size. Simulation and example results demonstrate the effectiveness of the proposed approaches.