Universal predictive systems

Abstract

This paper describes probability forecasting systems that are universal, or universally consistent, in the sense of being consistent under any data-generating distribution, assuming that the observations are produced independently in the IID fashion. The notion of universal consistency is asymptotic and does not imply any small-sample guarantees of validity. On the other hand, the method of conformal prediction has been recently adapted to producing predictive distributions that satisfy a natural property of small-sample validity, namely they are automatically probabilistically calibrated. The main result of the paper is the existence of universal conformal predictive systems, which output predictive distributions that are both probabilistically calibrated and universally consistent.