The role of prediction in evaluating econometric models

Abstract

Many statistical and economic criteria must influence the overall evalua­tion of econometric systems, but success when ‘predictive testing’ seems to establish most credibility, perhaps because the data were unavailable to the modellers. Yet the ability of prediction tests to detect even gross mis-specifications depends both on the properties of the data process and on the structure of the selected tests. To study the issues involved, a mixed analytical-Monte Carlo approach was adopted, based on asymp­totically valid approximations to forecast confidence intervals. Tests were calibrated for finite samples by simulation experiments on a Distributed Array Processor to yield numerical power function response surfaces. The main results of analysing prediction tests are that H successive one-step ahead tests are equivalent to one H -step test; that conditional multi-step forecast confidence bands are not necessarily monotonically increasing in H , so that ‘intercept corrections’ potentially have an objective justification; that tests based on averages (e. g. within-years) may be more useful than either one-step or H -step tests for detecting predictive failure; that the need to pool forecasts indicates non-encompassing models; and that a forecast-encompassing test should have some diagnostic power when applied to macroeconometric systems. Forecasting of the level of uncertainty is briefly considered.