Predictive Ability Test Research Articles

A consumer demands-resources model of engagement: Theoretical and managerial contributions from a cross-validated predictive ability test procedure

Despite the widespread economic and social interest in consumer engagement (CE) in do-it-yourself (DIY) or “making” activities, the construct remains underexplored in the literature. Integrating the theoretical frameworks of CE, the service-dominant logic, and conservation of resources theory, this study uses a mixed-methods approach to identify the resources that best predict CE in making activities. We conduct an exploratory qualitative study through 26 semi-structured interviews with consumers and professionals to refine our hypotheses. We then use a quantitative approach (N = 210) based on the latest advances in partial least squares structural equation modeling (PLS-SEM) to determine predictive validity. It thus offers an empirical application of the latest guidelines in this field. In particular, the study applies the new cross-validated predictive ability test (CVPAT) to test and compare several models. Combined with an importance-performance map analysis (IPMA), the CVPAT approach helps identify operational levers for decision-makers in the maker ecosystem.

The out-of-sample performance of carry trades

We carry out a large-scale investigation of the reliability of the profitability of carry trade strategies, using foreign exchange data for 48 countries over 36 years, employing reality check, superior predictive ability test, and stepwise tests to correct for data-snooping bias (the factor of luck in model selection). Carry trade strategies chosen as profitable in one period are generally not profitable in an ensuing out-of-sample sample period, especially after correcting for data-snooping and even after allowing for learning and stop-loss strategies. Any evidence of consistency in carry trade profitability that is found is concentrated in a relatively brief historical period, 1998-2005. We further investigate particular currency pairs that may drive the out-of-sample profitability during this period, and find their performance to be unstable in general. Our findings thus highlight the instability and the importance of chance or luck in generating profits from carry trades.

Maximum-Subsampling Test of Equal Predictive Ability

In comparing the accuracy of any two competing forecasts, under the assumption that the loss differentials are covariance stationary, Diebold and Mariano (DM) proposed a test statistic that is asymptotically normal. This test is further studied by Giacomini and White (GW). However, under the DM-GW framework, the estimator of the variance of the DM test can be inaccurate in small samples, which yields size distortions; see, for example, Coroneo and Iacone (CI). To alleviate this challenge, we propose a maximum-subsampling (MS) test that does not require estimating the long-run variance induced by a number of autocovariances. Accordingly, the MS test is applicable for comparing the predictive ability between two competing forecasts even when the loss differentials are serially correlated with arbitrary autocovariance structures. We demonstrate that the MS test converges to the type I extreme value distribution under the DM-GW null hypothesis with proper conditions. In addition, under a set of alternative hypotheses, we show the MS test is consistent. We further compare the MS test with the DM test and two CI tests via five simulation settings, which are either modified or adapted from McCracken and Coroneo and Iacone. Simulation results show that the MS test performs satisfactorily.

Beyond bricks and mortar: Work-life balance and project performance of Ghana's construction professionals – Does co-worker relationship quality matter?

Beyond bricks and mortar: Work-life balance and project performance of Ghana's construction professionals – Does co-worker relationship quality matter?

The value of self-determination theory in marketing studies: Insights from the application of PLS-SEM and NCA to anti-food waste apps

Self-determination theory (SDT) is a theory of human motivation that highlights the distinction between intrinsic and extrinsic motivations. Recent research has advocated its wider use in marketing studies, suggesting that it has strong predictive accuracy for consumer behaviors, and has proposed arguments about the necessity of both intrinsic and extrinsic motivations for marketing outcomes. However, these statements have not been empirically substantiated. We address this gap by studying the motivations for attitude and intention to use anti-food waste apps. Data from 141 users and 227 non-users of the app “Too Good To Go” are analyzed using partial least squares structural equation modeling (PLS-SEM) and several of its methodological extensions (e.g., multigroup analysis and the cross-validated predictive ability test), and necessary condition analysis (NCA). The findings support the argument that SDT accurately predicts consumer attitudes and behaviors, while NCA provides a nuanced view of the necessity of intrinsic and extrinsic motivations.

Abstract 11396: Circulating Biomarkers Improve ChA 2 DS 2 -VASC Risk Prediction in Atrial Fibrillation: The CHA 2 DS 2 -VASc-Biomarkers Model

Introduction: Anticoagulation reduces stroke risk in atrial fibrillation (AF) but increases bleeding. Risk calculators are used to inform decision making for anticoagulation in AF. Yet, existing tools have substantial limitations and improvement efforts failed to change practice. Hypotheses: Biomarkers of ischemic stroke risk for general populations are associated with stroke risk in AF and improve performance of a recommended risk score (CHA 2 DS 2 -VASc). Methods: REGARDS is a prospective cohort of 30,239 Black or White adults aged ≥45 monitored for stroke. Nine blood biomarkers were measured at baseline in participants with AF, no prior stroke, and not taking anticoagulation at baseline. Hazard ratios of ischemic stroke were estimated by Cox models adjusted for demographics and stroke risk factors. Models with CHA 2 DS 2 -VASc alone and with biomarkers (CHA 2 DS 2 -VASc-B) were compared using tests of fit (likelihood ratio test and Aikake information criterion [AIC]) and predictive ability (continuous net reclassification index [NRI]). Results: Among 2,411 participants with baseline AF (median age 69, 55% female, 36% Black), 163 (7%) developed stroke over 13 years. After adjustments, higher cystatin C, growth differentiation factor 15 (GDF-15), interleukin 6, lipoprotein (a), and N-terminal pro-B type natriuretic peptide (NTproBNP) were associated with higher stroke risk (Figure A). Including all biomarkers substantially improved CHA 2 DS 2 -VASc model fit (Figure B; p <0.001; ΔAIC -12.6; <-10 indicates marked improvement) and predictive ability (Figure C; 5-year NRI 0.42). Adding only GDF-15 and NTproBNP resulted in the best model fit and a similar NRI, compared to all biomarkers. Conclusions: Higher levels of five blood biomarkers were associated with ischemic stroke risk in a large general population cohort with AF. Moreover, the CHA 2 DS 2 -VASc-B model was superior to CHA 2 DS 2 -VASc. Findings suggest we can better tailor anticoagulation to reduce stroke in AF.

The roles of liquidity and delay in financial markets based on an optimal forecasting model.

We investigate the roles of liquidity and delay in financial markets through our proposed optimal forecasting model. The efficiency and liquidity of the financial market are examined using stochastic models that incorporate information delay. Based on machine learning, we estimate the in-sample and out-of-sample forecasting price performances of the six proposed methods using the likelihood function and Bayesian methods, and the out-of-sample prediction performance is compared with the benchmark model ARIMA-GARCH. We discover that the forecasting price performance of the proposed simplified delay stochastic model is superior to that of the benchmark methods by the test methods of a variety of loss function, superior predictive ability test (SPA), Akaike information criterion (AIC), and Bayesian information criterion (BIC). Using data from the Chinese stock market, the best forecasting model assesses the efficiency and liquidity of the financial market while accounting for information delay and trade probability. The rise in trade probability and delay time affects the stability of the return distribution and raises the risk, according to stochastic simulation. The empirical findings show that empirical and best forecasting approaches are compatible, that company size and liquidity (delay time) have an inverse relationship, and that delay time and liquidity have a nonlinear relationship. The most efficient have optimal liquidity.

The predictive ability of technical trading rules: an empirical analysis of developed and emerging equity markets

We investigate the predictability of leading equity indices of 23 developed and 18 emerging markets with a set of 6406 technical trading rules over up to 66 years. Using a state-of-the-art test for superior predictive ability to control for data snooping bias, we find in-sample evidence for technical heuristics with significant outperformance over a simple buy-and-hold strategy in the majority of markets. The proportion of heuristics with superior performance is much higher among emerging market indices, and the predictability diminishes drastically over time in all markets. In particular, markets turn unpredictable in the last years of our sample. Moreover, the results are very sensitive to the introduction of moderate transaction costs. An out-of-sample analysis shows that the performance of technical rules is not persistent in the sense that recently best-performing rules perform significantly worse than simple buy-and-hold strategies in the future. Overall, our results cast serious doubt on whether investors could have earned any excess profits using the broad range of considered technical trading rules.

Optimal out‐of‐sample forecast evaluation under stationarity

AbstractIt is a common practice to split a time series into an in‐sample and pseudo‐out‐of‐sample segments and estimate the out‐of‐sample loss for a given statistical model by evaluating forecasting performance over the pseudo‐out‐of‐sample segment. We propose an alternative estimator of the out‐of‐sample loss, which, contrary to the conventional wisdom, utilizes criteria measured both in‐ and out‐of‐sample via a carefully constructed system of affine weights. We prove that, provided that the time series is stationary, the proposed estimator is the best linear unbiased estimator of the out‐of‐sample loss and outperforms the conventional estimator in terms of sampling variability. Application of the optimal estimator to Diebold–Mariano type tests of predictive ability leads to a substantial power gain without increasing finite sample size distortions. An extensive evaluation on real‐world time series from the M4 forecasting competition confirms superiority of the proposed estimator and also demonstrates substantial robustness to violations of the underlying assumption of stationarity.

Cryptocurrencies and Lucky Factors: The value of technical and fundamental analysis

AbstractThis study explores the effectiveness of technical and fundamental analysis in predicting and trading the returns of 12 cryptocurrencies, namely Bitcoin, Ethereum, Ripple, Dash, Cardano, Avalanche, Binance Coin, Dogecoin, Polkadot, Litecoin, Terra and Solana. A universe of 7846 technical rules, five log moving average‐based ratios and 59 fundamental factors are used to test predictability and profitability through the Lucky Factors methodology and Superior Predictive Ability test. We observe predictability for a small set of technical and fundamental rules, while only the short‐term log moving average‐based ratio and Hashrate Index demonstrate genuine in‐sample and out‐of‐sample profitability. Our findings question the value of both technical and fundamental analysis on cryptocurrencies.

ON THE SIZE CONTROL OF THE HYBRID TEST FOR SUPERIOR PREDICTIVE ABILITY

This article analyzes the theoretical properties of the hybrid test for superior predictive ability. A simple example reveals that the test may not be size-controlled at common significance levels with rejection rates exceeding $11\%$ at a $5\%$ nominal level. Generalizing this observation, the main results show the pointwise asymptotic invalidity of the hybrid test under reasonable conditions. Monte Carlo simulations support these theoretical findings.

Equal predictive ability tests based on panel data with applications to OECD and IMF forecasts

We propose two types of equal predictive ability (EPA) tests with panels to compare the predictions made by two forecasters. The first type, S-statistics, focuses on the overall EPA hypothesis, which states that the EPA holds, on average, over all panel units and over time. The second type, C-statistics, focuses on the clustered EPA hypothesis where the EPA holds jointly for a fixed number of clusters of panel units. The asymptotic properties of the proposed tests are evaluated under weak and strong cross-sectional dependence. An extensive Monte Carlo simulation shows that the proposed tests have very good finite sample properties, even with little information about the cross-sectional dependence in the data. The proposed framework is applied to compare the economic growth forecasts of the OECD and the IMF, and to evaluate the performance of the consumer price inflation forecasts of the IMF.

Predicting Bond Return Predictability

This paper provides empirical evidence on predictable time variations in out-of-sample bond return predictability. Bond return predictability is associated with periods of high (low) economic activity (uncertainty), which implies that violations of the expectations hypothesis are state dependent and linked to features of the business cycle. These state dependencies in predictability, established by introducing a new multivariate test for equal conditional predictive ability, can be used in real time to improve out-of-sample bond risk premia estimates and investors’ economic utility through a novel dynamic forecast combination scheme that uses predicted forecasting performance to identify the best set of methods to include in the combined forecast. Dynamically combined forecasts exhibit strong countercyclical behavior and peak during recessions.This paper was accepted by Lukas Schmid, finance.Funding: This work was supported by the Danish Council of Independent Research [Grants DFF 7024–00020B and DFF 9033–00003B] and the Danish Finance Institute.Supplemental Material: The data files and online appendices are available at https://doi.org/10.1287/mnsc.2023.4713 .

Predictive Ability Tests with Possibly Overlapping Models

Predictive Ability Tests with Possibly Overlapping Models

Equal Predictive Ability Tests Based on Panel Data with Applications to OECD and IMF Forecasts

Equal Predictive Ability Tests Based on Panel Data with Applications to OECD and IMF Forecasts

Conditional evaluation of predictive models: The cspa command

In this article, we introduce a new command, cspa, that implements the conditional superior predictive ability test developed in Li, Liao, and Quaedvlieg (2022, Review of Economic Studies 89: 843–875). With the conditional performance of predictive methods measured nonparametrically by the conditional expectation functions of their predictive losses, we test the null hypothesis that a benchmark model weakly outperforms a collection of competitors uniformly across the conditioning space. The proposed command can implement this test for both independent cross-sectional data and serially dependent time-series data. Confidence sets for the most superior model can be obtained by inverting the test, for which the cspa command also offers a convenient implementation.

A nonparametric predictive regression model using partitioning estimators based on Taylor expansions

This article proposes a nonparametric predictive regression model. The unknown function modeling the predictive relationship is approximated using polynomial Taylor expansions applied over disjoint intervals covering the support of the predictor variable. The model is estimated using the theory on partitioning estimators that is extended to a stationary time series setting. We show pointwise and uniform convergence of the proposed estimator and derive its asymptotic normality. These asymptotic results are applied to test for the presence of predictive ability. We develop an asymptotic pointwise test of predictive ability using the critical values of a Normal distribution, and a uniform test with asymptotic distribution that is approximated using a p‐value transformation and Wild bootstrap methods. These theoretical insights are illustrated in an extensive simulation exercise and also in an empirical application to forecasting high‐frequency based realized volatility measures. Our results provide empirical support to the presence of nonlinear autoregressive predictability of these measures for the constituents of the Dow Jones index.

Predictive model assessment and selection in composite-based modeling using PLS-SEM: extensions and guidelines for using CVPAT

PurposeResearchers often stress the predictive goals of their partial least squares structural equation modeling (PLS-SEM) analyses. However, the method has long lacked a statistical test to compare different models in terms of their predictive accuracy and to establish whether a proposed model offers a significantly better out-of-sample predictive accuracy than a naïve benchmark. This paper aims to address this methodological research gap in predictive model assessment and selection in composite-based modeling.Design/methodology/approachRecent research has proposed the cross-validated predictive ability test (CVPAT) to compare theoretically established models. This paper proposes several extensions that broaden the scope of CVPAT and explains the key choices researchers must make when using them. A popular marketing model is used to illustrate the CVPAT extensions’ use and to make recommendations for the interpretation and benchmarking of the results.FindingsThis research asserts that prediction-oriented model assessments and comparisons are essential for theory development and validation. It recommends that researchers routinely consider the application of CVPAT and its extensions when analyzing their theoretical models.Research limitations/implicationsThe findings offer several avenues for future research to extend and strengthen prediction-oriented model assessment and comparison in PLS-SEM.Practical implicationsGuidelines are provided for applying CVPAT extensions and reporting the results to help researchers substantiate their models’ predictive capabilities.Originality/valueThis research contributes to strengthening the predictive model validation practice in PLS-SEM, which is essential to derive managerial implications that are typically predictive in nature.

Forecasting Industrial Production Using Its Aggregated and Disaggregated Series or a Combination of Both: Evidence from One Emerging Market Economy

In this paper, we address whether using a disaggregated series or combining an aggregated and disaggregated series improves the forecasting of the aggregated series compared to using the aggregated series alone. We used econometric techniques, such as the weighted lag adaptive least absolute shrinkage and selection operator, and Exponential Triple Smoothing (ETS), as well as the Autometrics algorithm to forecast industrial production in Brazil one to twelve months ahead. This is the novelty of the work, as is the use of the average multi-horizon Superior Predictive Ability (aSPA) and uniform multi-horizon Superior Predictive Ability (uSPA) tests, used to select the best forecasting model by combining different horizons. Our sample covers the period from January 2002 to February 2020. The disaggregated ETS has a better forecast performance when forecasting horizons that are more than one month ahead using the mean square error, and the aggregated ETS has better forecasting ability for horizons equal to 1 and 2. The aggregated ETS forecast does not contain information that is useful for forecasting industrial production in Brazil beyond the information already found in the disaggregated ETS forecast between two and twelve months ahead.

Analyzing and Forecasting Multi-Commodity Prices Using Variants of Mode Decomposition-Based Extreme Learning Machine Hybridization Approach

Because of the non-linearity inherent in energy commodity prices, traditional mono-scale smoothing methodologies cannot accommodate their unique properties. From this viewpoint, we propose an extended mode decomposition method useful for the time-frequency analysis, which can adapt to various non-stationarity signals relevant for enhancing forecasting performance in the era of big data. To this extent, we employ variants of mode decomposition-based extreme learning machines namely: (i) Complete Ensemble Empirical Mode Decomposition with Adaptive Noise-based ELM Model (CEEMDAN-ELM), (ii) Ensemble Empirical Mode Decomposition-based ELM Model (EEMD-ELM) and (iii) Empirical Mode Decomposition Based ELM Model (EMD-ELM), which cut-across soft computing and artificial intelligence to analyze multi-commodity time series data by decomposing them into seven independent intrinsic modes and one residual with varying frequencies that depict some interesting characterization of price volatility. Our findings show that in terms of the model-specific forecast accuracy measures different dynamics in the two scenarios namely the (non) COVID periods. However, the introduction of a benchmark, namely the autoregressive integrated moving average model (ARIMA) reveals a slight change in the earlier dynamics, where ARIMA outperform our proposed models in the Japan gas and the US gas markets. To check the superiority of our models, we apply the model-confidence set (MCS) and the Kolmogorov-Smirnov Predictive Ability test (KSPA) with more preference for the former in a multi-commodity framework, which reveals that in the pre-COVID era, CEEMDAN-ELM shows persistence and superiority in accurately forecasting Crude oil, Japan gas, and US gas. Nonetheless, this paradigm changed during the COVID-era, where CEEMDAN-ELM favored Japan gas, US gas, and coal market with different rankings via the Model confidence set evaluation methods. Overall, our numerical experiment indicates that all decomposition-based extreme learning machines are superior to the benchmark model.