<title>Confidence intervals for ATR performance metrics</title>

Abstract

This paper describes confidence interval (CI) estimators (CIEs) for the metrics used to assess sensor exploitation algorithm (or ATR) performance. For the discrete distributions, small sample sizes and extreme outcomes encountered within ATR testing, the commonly used CIEs have limited accuracy. This paper makes available CIEs that are accurate over all conditions of interest to the ATR community. The approach is to search for CIs using an integration of the Bayesian posterior (IBP) to measure alpha (chance of the CI not containing the true value). The CIEs provided include proportion estimates based on Binomial distributions and rate estimates based on Poisson distributions. One or two-sided CIs may be selected. For two-sided CIEs, either minimal length, balanced tail probabilities, or balanced width may be selected. The CIEs' accuracies are reported based on a Monte Carlo validated integration of the posterior probability distribution and compared to the Normal approximation and `exact' (Clopper- Pearson) methods. While the IBP methods are accurate throughout, the conventional methods may realize alphas with substantial error (up to 50%). This translates to 10 to 15% error in the CI widths or to requiring 10 to 15% more samples for a given confidence level.