Calibrating Prediction Regions

Abstract

Abstract Suppose that the variable X to be predicted and the learning sample Yn that was observed have a joint distribution, which depends on an unknown parameter θ. The parameter θ can be finite- or infinite-dimensional. A prediction region Dn for X is a random set, depending on Yn , that contains X with prescribed probability α. This article studies methods for controlling simultaneously the conditional coverage proability of Dn , given Yn , and the overall (unconditional) coverage probability of Dn . The basic construction yields a prediction region Dn , which has the following properties in regular models: Both the conditional and overall coverage probabilities of Dn converge to α as the size n of the learning sample increases. The convergence of the former is in probability. Moreover, the asymptotic distribution of the conditional coverage probability about α is typically normal; and the overall coverage probability tends to α at rate n −1. Can one reduce the dispersion of the conditional coverage probability about α and increase the rate at which overall coverage probability converges to α? Both issues are addressed. The article establishes a lower bound for the asymptotic dispersion of conditional coverage probability. The article also shows how to calibrate Dn so as to make its overall coverage probability converge to α at the faster rate n −2. This calibration adjustment does not affect the asymptotic distribution or dispersion of the conditional coverage probability, in a first-order analysis. In general, a bootstrap Monte Carlo algorithm accomplishes the calibration of Dn . In special cases, analytical calibration is possible.