Optimal Threshold from ROC and CAP Curves

Abstract

Receiver Operating Characteristic (ROC) and Cumulative Accuracy Profile (CAP) curves are used to assess the discriminatory power of different credit-rating approaches. The thresholds of optimal classification accuracy on an ROC curve and of maximal profit on a CAP curve can be found by using iso-performance tangent lines, which are based on the standard notion of accuracy. In this article, we propose another accuracy measure called the true rate. Using this rate, one can obtain alternative optimal thresholds on both ROC and CAP curves. For most real populations of borrowers, the number of the defaults is much less than that of the non defaults, and in such cases using the true rate may be more efficient than using the accuracy rate in terms of cost functions. Moreover, it is shown that both alternative optimal thresholds by using the true rate are the identical, and this single threshold coincides with the score corresponding to Kolmogorov–Smirnov statistic used to test the homogeneous distribution functions of the defaults and non defaults, whereas the optimal threshold by using the accuracy does not the same as the score corresponding to Kolmogorov–Smirnov statistic. These facts are explored with some simulation and illustrative examples.