Confidence intervals for low probability for false alarms and probability for a miss at low values based small sample space support

Abstract

When using ROCs (receiver operating characteristics) based upon experimental data derived from either experimental or simulated data, there is a tendency to concentrate on the high values of PFA and PM (miss probability) in the middle of the ROC figure. Nevertheless, most good systems want to have a performance near the upper left of an ROC (low PFA, low PM) and this is where results based upon finite sample support have the largest proportional errors. While this may now be an artifact that ‘‘probability paper’’ is no longer routinely used, fitting a curve through these regions is almost always misleading unless the sample support is very large. The problem is fundamentally one of fitting the ‘‘tail’’ of a distribution with little data. Various methods from probability such as ‘‘Chi squared’’ and Kolmogorov-Smirnov test can be used, but they almost always lead to rejection being tested in this low probability region. Here, we use an approach based upon ‘‘Chernov tilted densities’’ to capture the uncertainty of predictions in these regions when there is sparse, or small, sample support.