An Odds Ratio Parameterization for ROC Diagram and Skill Score Indices

Abstract

Abstract The relative operating characteristic (ROC) diagram is often used to assess the performance of a classification system, like the categorical forecast of an event occurrence. Categorical forecasting can be obtained by imposing a threshold on a continuous variable in order to make it dichotomous. In practice this threshold could be varied to create different contingency tables. From each table, it is then possible to derive many statistical indices and skill scores, which are functions of the chosen threshold. The ROC curve is obtained by plotting two of these indices: probability of detection (POD) versus probability of false detection (POFD). In this work a simple approximation for another of these indices, the odds ratio (O), is proposed. Thus, O is parameterized as a function of POFD and that leads to a parameterization of all the theoretical ROC curves. Using this approximation, it is also possible to derive the theoretical maximum Hanssen and Kuipers skill score (KSS) and the theoretical maximum Heidke skill score (HSS), for each ROC. It is found that the maximum HSS depends explicitly on the database event frequency (α), while the KSS seems independent of it. Out of the approximation framework, some general properties of ROC points corresponding to the maximum KSS, to the maximum HSS, and to the BIAS = 1 condition have also been found. It is also suggested that many of these performance measures are influenced by the event frequency, which must be taken into account when comparing classifiers made for different databases. Another interesting outcome of this study is that it is shown how the KSS is also equitable (in the sense introduced by Gandin and Murphy) for a generic “cost ratio” (λ) between miss and false alarm cases, not only for the original case λ = 1.