Testable forecasts

Abstract

Predictions about the future are commonly evaluated through statistical tests. As shown by recent literature, many known tests are subject to adverse selection problems and cannot discriminate between forecasters who are competent and forecasters who are uninformed but predict strategically. We consider a framework where forecasters' predictions must be consistent with a paradigm, a set of candidate probability laws for the stochastic process of interest. This paper presents necessary and sufficient conditions on the paradigm under which it is possible to discriminate between informed and uninformed forecasters. We show that optimal tests take the form of likelihood‐ratio tests comparing forecasters' predictions against the predictions of a hypothetical Bayesian outside observer. In addition, the paper illustrates a new connection between the problem of testing strategic forecasters and the classical Neyman–Pearson paradigm of hypothesis testing.