Equal Predictive Ability Research Articles

Maximum-Subsampling Test of Equal Predictive Ability

In comparing the accuracy of any two competing forecasts, under the assumption that the loss differentials are covariance stationary, Diebold and Mariano (DM) proposed a test statistic that is asymptotically normal. This test is further studied by Giacomini and White (GW). However, under the DM-GW framework, the estimator of the variance of the DM test can be inaccurate in small samples, which yields size distortions; see, for example, Coroneo and Iacone (CI). To alleviate this challenge, we propose a maximum-subsampling (MS) test that does not require estimating the long-run variance induced by a number of autocovariances. Accordingly, the MS test is applicable for comparing the predictive ability between two competing forecasts even when the loss differentials are serially correlated with arbitrary autocovariance structures. We demonstrate that the MS test converges to the type I extreme value distribution under the DM-GW null hypothesis with proper conditions. In addition, under a set of alternative hypotheses, we show the MS test is consistent. We further compare the MS test with the DM test and two CI tests via five simulation settings, which are either modified or adapted from McCracken and Coroneo and Iacone. Simulation results show that the MS test performs satisfactorily.

Conditional monitoring and fault detection of wind turbines based on Kolmogorov–Smirnov non-parametric test

This research presents a new method for conditional monitoring based on the wind turbine power curve. The Kolmogorov-Smirnov (K-S) distribution test is employed in the assessment of turbine data and the detection of abnormality (faults) in wind turbines. The process begins with anomaly detection and filtration of faulty SCADA data by a quantile-based filtration approach. Suitable data comprising wind speed, air density, ambient temperature, and pitch angle are utilized in the development of wind turbine power curve models that represents actualities within wind farms. The radial basis function (RBF), multi-layer Perceptron (MLP), and gradient boosting (GBR) methods utilized for model development are compared for predictive accuracy using Mariano-Preve test. The null hypothesis assumes equal predictive ability (EPA); if rejected, an algorithm compares the coefficients of correlation of the models and selects the closest to one (unity). The most accurate model is utilized for the creation of a bin-wise distribution from past data, and bin-wise confidence levels from the plot of wind speed and output power. Cochran’s method was utilized to validate the minimum sample size that will possess a sampling distribution similar to that of the population, and a fault is detected if there is a reasonable difference between the sample distribution and population distribution. The K-S test, having a null hypothesis of equivalent distributions, signals a fault if the null hypothesis is rejected. Two wind turbine SCADA datasets associated with two fault events are used for the assessment of our method. The results indicate that our method effectively discovers abnormalities in power output relating to increased bearing temperature and reduced generator rpm, thereby aiding in the detection of faults long before they occur.

Estimation of value at risk for copper

We analyze various types of models for Value at Risk (VaR) forecasts for daily copper returns. The period of the analysis is from January 4, 2000 to January 14, 2021 including 5290 daily closing prices. The models considered are GARCH-type models, the Generalized Autoregressive Score model, the Dynamic Quantile Regression model, and the Conditional Autoregressive Value at Risk model specifications. The best model is selected using the Model Confidence Set approach. This approach provides a superior set of models by testing the null hypothesis of equal predictive ability. The findings suggest that the EGARCH model outperforms the rest of the models for the copper commodity under investigation.

Equal predictive ability tests based on panel data with applications to OECD and IMF forecasts

We propose two types of equal predictive ability (EPA) tests with panels to compare the predictions made by two forecasters. The first type, S-statistics, focuses on the overall EPA hypothesis, which states that the EPA holds, on average, over all panel units and over time. The second type, C-statistics, focuses on the clustered EPA hypothesis where the EPA holds jointly for a fixed number of clusters of panel units. The asymptotic properties of the proposed tests are evaluated under weak and strong cross-sectional dependence. An extensive Monte Carlo simulation shows that the proposed tests have very good finite sample properties, even with little information about the cross-sectional dependence in the data. The proposed framework is applied to compare the economic growth forecasts of the OECD and the IMF, and to evaluate the performance of the consumer price inflation forecasts of the IMF.

Equal Predictive Ability Tests Based on Panel Data with Applications to OECD and IMF Forecasts

Equal Predictive Ability Tests Based on Panel Data with Applications to OECD and IMF Forecasts

Understanding the conditional out-of-sample predictive impact of the price of crude oil on aggregate equity return volatility

Using monthly data from 1973 through 2020, we explore whether it is possible to improve the accuracy of one-month ahead log-aggregate equity return realized volatility point forecasts by conditioning on various nonlinear crude oil price measures widely relied on in the literature. When evaluating the evidence of unconditional relative equal predictive ability as specified in Diebold and Mariano (1995), we observe that similar to well-known economic variables, such as the dividend yield, the default yield spread and the rate of inflation, we rarely observe evidence of statistical gains in relative point forecast accuracy in favor of the crude oil price-based models. However, when evaluating the evidence of conditionalrelative equal predictive ability as specified in Giacomini and White (2006), we observe that contrary to well-known economic predictors, certain nonlinear crude oil price variables, such as the one-year net crude oil price increase suggested in Hamilton (1996) offer sizable point forecast accuracy gains relative to the benchmark. These statistical gains can also be translated into economic gains.

Combining p-values for Multivariate Predictive Ability Testing

In this paper, we propose an intersection-union test for multivariate forecast accuracy based on the combination of a sequence of univariate tests. The testing framework evaluates a global null hypothesis of equal predictive ability using any number of univariate forecast accuracy tests under arbitrary dependence structures, without specifying the underlying multivariate distribution. An extensive Monte-Carlo simulation exercise shows that our proposed test has very good size and power properties under several relevant scenarios, and performs well in both low- and high-dimensional settings. We illustrate the empirical validity of our testing procedure using a large dataset of 84 daily exchange rates running from 1 January 2011 to 1 April 2021. We show that our proposed test addresses inconclusive results that often arise in practice.

Comparison of Models for Growth-at-Risk Forecasting

During the past several decades, the importance of assessing the risk of GDP growth downturns has increased tremendously. The financial crisis of 2008–2009 and the global lockdown caused by the COVID-19 pandemic demonstrated the vulnerability of the modern economy. As a result, a new framework (Growth-at-Risk) has been developed which allows the estimation of the size of the potential downfall of future GDP growth. However, most of the research focuses on the performance of quantile regression. I apply different approaches to forecasting growth-at-risk, including quantile regression, quantile random forests, and generalised autoregressive conditional heteroscedastic (GARCH) models, using the US economy for the analysis. I find that GARCH-type models perform worse at 5% and 10% coverage levels, but that quantile random forests and quantile regressions seem to have equal predictive ability.

Agree to disagree? Predictions of U.S. nonfarm payroll changes between 2008 and 2020 and the impact of the COVID19 labor shock

We analyze an unbalanced panel of monthly predictions of nonfarm payroll (NFP) changes between January 2008 and December 2020 sourced from Bloomberg. Unsurprisingly, we find that prediction quality varies across economists and we reject the hypothesis of equal predictive ability. In an error decomposition, we find evidence of significantly biased forecasts. Participation rate in the survey is affecting this bias. We find that survey participants under-predict job losses in times of market turmoil while also under-predicting the recovery thereafter, especially during the COVID19 labor shock. For prediction of NFP changes, autoregressive models are outperformed by a deep learning long short-term memory network. However, the consensus forecast yields better forecasts than model-based approaches and are further improved by combining the forecasts of the best performing economists. The COVID19 labor shock is shown to have adverse effects on the prediction performance of economists. However, not all economists are affected equally.

Tests of Equal Forecasting Accuracy for Nested Models with Estimated CCE Factors*

In this article, we propose new tests of equal predictive ability between nested models when factor-augmented regressions are used to forecast. In contrast to the previous literature, the unknown factors are not estimated by principal components but by the common correlated effects (CCE) approach, which employs cross-sectional averages of blocks of variables. This makes for easy interpretation of the estimated factors, and the resulting tests are easy to implement and they account for the block structure of the data. Assuming that the number of averages is larger than the true number of factors, we establish the limiting distributions of the new tests as the number of time periods and the number of variables within each block jointly go to infinity. The main finding is that the limiting distributions do not depend on the number of factors but only on the number of averages, which is known. The important practical implication of this finding is that one does not need to estimate the number of factors consistently in order to apply our tests.

“Go Wild for a While!”: A New Test for Forecast Evaluation in Nested Models

In this paper, we present a new asymptotically normal test for out-of-sample evaluation in nested models. Our approach is a simple modification of a traditional encompassing test that is commonly known as Clark and West test (CW). The key point of our strategy is to introduce an independent random variable that prevents the traditional CW test from becoming degenerate under the null hypothesis of equal predictive ability. Using the approach developed by West (1996), we show that in our test, the impact of parameter estimation uncertainty vanishes asymptotically. Using a variety of Monte Carlo simulations in iterated multi-step-ahead forecasts, we evaluated our test and CW in terms of size and power. These simulations reveal that our approach is reasonably well-sized, even at long horizons when CW may present severe size distortions. In terms of power, results were mixed but CW has an edge over our approach. Finally, we illustrate the use of our test with an empirical application in the context of the commodity currencies literature.

How to Predict Energy Consumption in BRICS Countries?

Brazil, Russia, China, India, and the Republic of South Africa (BRICS) represent developing economies facing different energy and economic development challenges. The current study aims to predict energy consumption in BRICS at aggregate and disaggregate levels using the annual time series data set from 1992 to 2019 and to compare results obtained from a set of models. The time-series data are from the British Petroleum (BP-2019) Statistical Review of World Energy. The forecasting methodology bases on a novel Fractional-order Grey Model (FGM) with different order parameters. This study contributes to the literature by comparing the forecasting accuracy and the predictive ability of the FGM1,1 with traditional ones, like standard GM1,1 and ARIMA1,1,1 models. Moreover, it illustrates the view of BRICS’s nexus of energy consumption at aggregate and disaggregates levels using the latest available data set, which will provide a reliable and broader perspective. The Diebold-Mariano test results confirmed the equal predictive ability of FGM1,1 for a specific range of order parameters and the ARIMA1,1,1 model and the usefulness of both approaches for energy consumption efficient forecasting.

30-day mortality in patients after hip fracture surgery: A comparison of the Charlson Comorbidity Index score and ASA score used in two prediction models

Objective: To compare the Charlson Comorbidity Index (CCI) and American Society of Anesthesiologists (ASA) Physical Status Classification used in two prediction models for 30-day mortality after hip fracture surgery.Study Design and Setting: Data from 3651 patients (mean age: 83 years) from a Norwegian University Hospital were retrospectively obtained and randomly divided into two cohorts: a model cohort (n = 1825) to develop two prediction models with CCI and ASA as the main predictors, and a validation cohort (n = 1826) to assess the predictive ability of both models. A receiver operating characteristic (ROC) curve determined the best model to predict mortality.Results: Area under the ROC curve at 30 days was 0.726 (p = 0.988) for both the CCI- and ASA-model. The chosen cut-off-points on the ROC curve for CCI- and ASA-model corresponded to similar model sensitivities of 0.657 and specificities of 0.680 and 0.679, respectively. Hence, each model predicts correctly 66% (n = 96) of the mortalities and 68% (n = 1132 and n = 1131) of the survivals. 23% (n = 33) of the mortalities were predicted by neither model.Conclusion: The CCI- and ASA-model had equal predictive ability of 30-day mortality after hip fracture. Considering the effort involved in calculating Charlson Comorbidity Index score, the ASA score may be the preferred tool to predict the 30-day mortality after hip fracture.

The price of crude oil and (conditional) out-of-sample predictability of world industrial production

A measure of global economic activity (EA) is often used as input in macroeconometric models. Baumeister and Hamilton (2019) and Hamilton (2019) favor using the world industrial production index as a measure for global EA. Given the connections between variations in industrial production and demand for industrial commodities, such as crude oil, one is inclined to assume that changes in the world industrial production index can be predicted out-of-sample if one conditions on changes in the price of crude oil. Interestingly, we do not find any evidence of out-of-sample point forecast accuracy gains from our crude oil price-based models relative to the benchmark. Likewise, the unconditional equal predictive ability test suggested in Diebold and Mariano (1995) rarely indicates a statistical difference between point forecasts produced under the benchmark and the crude oil price-based models. However, the null hypothesis of equal conditional predictive ability as specified in Giacomini and White (2006) is often rejected. By relying on the information provided by the conditioning variables used in the Giacomini and White (2006) test, and devising a forecast selection strategy following Granziera and Sekhposyan (2019), we succeed at obtaining one-month ahead point forecast accuracy gains as high as 14% relative to the benchmark. The nonlinear model using the one-year asymmetric net crude oil price change performs very well when business conditions are bad or equity market uncertainty is high.

"Go Wild for a While!": A New Asymptotically Normal Test for Forecast Evaluation in Nested Models

In this paper we present a new asymptotically normal test for out-of-sample evaluation in nested models. Our approach is a simple modification of a traditional encompassing test that is commonly known as Clark and West test (CW). The key point of our strategy is to introduce an independent random variable that prevents the traditional CW test from becoming degenerate under the null hypothesis of equal predictive ability. Using the approach developed by West (1996), we show that in our test the impact of parameter estimation uncertainty vanishes asymptotically. Using a variety of Monte Carlo simulations in iterated multi-step-ahead forecasts we evaluate our test and CW in terms of size and power. These simulations reveal that our approach is reasonably well-sized even at long horizons when CW may present severe size distortions. In terms of power, results are mixed but CW has an edge over our approach. Finally, we illustrate the use of our test with an empirical application in the context of the commodity currencies literature.

Agree to Disagree? Predictions of U.S. Nonfarm Payroll Changes between 2008 and 2020 and the Impact of the COVID19 Labor Shock

We analyze an unbalanced panel of monthly predictions of nonfarm payroll (NFP) changes between January 2008 and December 2020 sourced from Bloomberg. Unsurprisingly, we find that prediction quality varies across economists and we reject the hypothesis of equal predictive ability. In an error decomposition, we find evidence of significantly biased forecasts. Participation rate in the survey is affecting this bias. We find that survey participants under-predict job losses in times of market turmoil while also under-predicting the recovery thereafter, especially during the COVID19 labor shock. For prediction of NFP changes, autoregressive models are outperformed by a deep learning long short-term memory network. However, the consensus forecast yields better forecasts than model-based approaches and are further improved by combining the forecasts of the best performing economists. The COVID19 labor shock is shown to have adverse effects on the prediction performance of economists. However, not all economists are affected equally.

Crude oil price point forecasts of the Norwegian GDP growth rate

Given the important role of the petroleum industry in the Norwegian economy, one would assume that changes in the price of crude oil would help greatly improve the accuracy of the Norwegian real gross domestic product growth rate point (density) forecasts out-of-sample. Surprisingly, evidence of one-quarter-ahead out-of-sample point (density) forecast accuracy gain relative to the benchmark model is very weak, at best close to 3%. Furthermore, results from the unconditional equal predictive ability test suggested in Diebold and Mariano (J Bus Econ Stat 13:253–263, 1995) document that these modest gains are not statistically significant. However, the null hypothesis of equal conditional predictive ability as specified in Giacomini and White (Econometrica 74:1545–1578, 2006) is rejected for a number of models. Moreover, by relying on the information provided by the conditioning variables used in the Giacomini and White (2006) test and devising a forecast selection strategy following Granziera and Sekhposyan (Int J Forecast 35:1636–1657, 2019), we succeed at obtaining point forecast accuracy gains as high as $$12\%$$ relative to the benchmark one-quarter ahead.

Diverging Tests of Equal Predictive Ability

We investigate claims made in Giacomini and White (2006) and Diebold (2015) regarding the asymptotic normality of a test of equal predictive ability. A counterexample is provided in which, instead, the test statistic diverges with probability 1 under the null.

Forecasting aggregate equity return volatility using crude oil price volatility: The role of nonlinearities and asymmetries

To assess the potential nonlinear predictive impact of crude oil price volatility on aggregate equity return volatility, we consider autoregressions of monthly aggregate equity return realized volatility augmented with nonlinear transformations of crude oil price realized volatility and evaluate if they improve point forecasts. Out-of-sample results based on data from 1885m1 through 1895m12 and from 1983m1 through 2017m12 illustrate that our conclusions depend heavily on the notion of forecast improvement. At the population level, the null hypothesis of no out-of-sample predictability from crude oil price realized volatility to aggregate equity return realized volatility is rejected for the linear as well as certain nonlinear specifications. On the other hand, the null hypothesis of finite-sample equal predictive ability is rejected less frequently. Among the range of models, the autoregression augmented with the one-year net crude oil price realized volatility increase is the top performer, producing statistically significant more accurate point forecasts than the benchmark.

Scale-Dependent Functional Redundancy in a Tropical Forest

Functional redundancy is an important tool for justifying and prioritizing species protection in forest ecosystem, but it is a scale-dependent. If functional redundancy really exists, functional trait composition tends to have higher predictive ability of community assembly than species composition. Thus, comparing the differences in the predictive ability of community assembly between species and functional trait compositions across spatial scale represents a useful tool to quantify how functional redundancy varies across spatial scales. Here, we used variation partitioning in combination with distance-based Moran’s eigenvector maps to compare the differences in the predictive ability of community assembly between species composition and functional trait composition across spatial scales (20, 30, 40, 50, and 100 m) in a 20-ha subtropical forest plot. We found that functional trait composition possessed higher predictive ability of niche-based abiotic filtering process than species composition within 40 m. At 50 and 100 m scales, both species and functional trait compositions had approximately equal predictive ability of dispersal limitation processes. Thus, functional redundancy can only exist within 40 m scale but not 50 and 100 m scales. As a result, priority species loss protection should be performed at 50 and 100 m scales.