An invalidation test for predictive models

Abstract

The standard means of establishing predictive ability in hydrological models is by finding how well predictions match independent validation data. This matching may not be particularly good in some situations such as seasonal flow forecasting and the question arises as to whether a given model has any predictive capacity. A model-independent significance test of the presence of predictive ability is proposed through random permutations of the predicted values. The null hypothesis of no model predictive ability is accepted if there is a sufficiently high probability that a random reordering of the predicted values will yield a better fit to the validation data. The test can achieve significance even with poor model predictions and its value is for invalidating bad models rather than verifying good models as suitable for application. Some preliminary applications suggest that test outcomes will often be similar at the 0.05 level for standard fit measures using absolute or squared residuals. In addition to hydrological application, the test may also find use as a base quality control measure for predictive models generally.