Abstract
Are traditional tests of forecast evaluation well behaved when the competing (nested) model is biased? No, they are not. In this paper, we show analytically and via simulations that, under the null hypothesis of no encompassing, a bias in the nested model may severely distort the size properties of traditional out-of-sample tests in economic forecasting. Not surprisingly, these size distortions depend on the magnitude of the bias and the persistency of the additional predictors. We consider two different cases: (i) There is both in-sample and out-of-sample bias in the nested model. (ii) The bias is present exclusively out-of-sample. To address the former case, we propose a modified encompassing test (MENC-NEW) robust to a bias in the null model. Akin to the ENC-NEW statistic, the asymptotic distribution of our test is a functional of stochastic integrals of quadratic Brownian motions. While this distribution is not pivotal, we can easily estimate the nuisance parameters. To address the second case, we derive the new asymptotic distribution of the ENC-NEW, showing that critical values may differ remarkably. Our Monte Carlo simulations reveal that the MENC-NEW (and the ENC-NEW with adjusted critical values) is reasonably well-sized even when the ENC-NEW (with standard critical values) exhibits rejections rates three times higher than the nominal size.
Highlights
Publisher’s Note: MDPI stays neutral with regard to jurisdictional claims in published maps and institutional affiliations
We show that the use of CM critical values ignoring the effects of these misspecifications may lead to severe size distortions; in specific, we address the effects of a bias in the predictive models over the asymptotic distributions
We determine the critical values (CV) for the MENC-NEW according to Theorem 2
Summary
“ for serious minds, a bias recognized is a bias sterilized” Benjamin Haydon. We show that the use of CM critical values ignoring the effects of these misspecifications may lead to severe size distortions; in specific, we address the effects of a bias in the predictive models over the asymptotic distributions The interesting case here is an in-sample bias of zero but an out-of-sample bias different from zero (e.g., a shift in the drift of the target variable) In this case, there is no need for a re-centering term, we derive the new asymptotic distribution of the ENC-NEW (it is necessary to simulate new critical values).
Talk to us
Join us for a 30 min session where you can share your feedback and ask us any queries you have