Degree Of Misspecification Research Articles

Likelihood Inference for Factor Copula Models with Asymmetric Tail Dependence.

For multivariate non-Gaussian involving copulas, likelihood inference is dominated by the data in the middle, and fitted models might not be very good for joint tail inference, such as assessing the strength of tail dependence. When preliminary data and likelihood analysis suggest asymmetric tail dependence, a method is proposed to improve extreme value inferences based on the joint lower and upper tails. A prior that uses previous information on tail dependence can be used in combination with the likelihood. With the combination of the prior and the likelihood (which in practice has some degree of misspecification) to obtain a tilted log-likelihood, inferences with suitably transformed parameters can be based on Bayesian computing methods or with numerical optimization of the tilted log-likelihood to obtain the posterior mode and Hessian at this mode.

Technical Validation of Plot Designs by Use of Deep Learning

When does inspecting a certain graphical plot allow for an investigator to reach the right statistical conclusion? Visualizations are commonly used for various tasks in statistics—including model diagnostics and exploratory data analysis—and though attractive due to its intuitive nature, the lack of available methods for validating plots is a major drawback. We propose a new technical validation method for visual reasoning. Our method trains deep neural networks to distinguish between plots simulated under two different data generating mechanisms (null or alternative), and we use the classification accuracy as a technical validation score (TVS). The TVS measures the information content in the plots, and TVS values can be used to compare different plots or different choices of data generating mechanisms, thereby providing a meaningful scale that new visual reasoning procedures can be validated against. We apply the method to three popular diagnostic plots for linear regression, namely scatterplots, quantile-quantile plots and residual plots. We consider various types and degrees of misspecification, as well as different within-plot sample sizes. Our method produces TVSs that increase with increasing sample size and decrease with increasing difficulty, and hence the TVS is a meaningful measure of validity.

Bayesian backtesting for counterparty risk models

We introduce a new framework for counterparty risk model backtesting based on Bayesian methods. This provides a conceptually sound approach for analyzing model performance that is also straightforward to implement. We show that our methodology provides important advantages over a typical, classical backtesting setup. In particular, we find that the Bayesian approach outperforms the classical one in identifying whether a model is correctly specified, which is the principal aim of any backtesting framework. The power of the methodology is due to its ability to test individual parameters and thus identify not only the degree of misspecification but also which aspects of a model are misspecified. This greatly facilitates the impact assessment of model issues as well as their remediation.

Optimal probabilistic forecasts: When do they work?

Proper scoring rules are used to assess the out-of-sample accuracy of probabilistic forecasts, with different scoring rules rewarding distinct aspects of forecast performance. Herein, we re-investigate the practice of using proper scoring rules to produce probabilistic forecasts that are ‘optimal’ according to a given score and assess when their out-of-sample accuracy is superior to alternative forecasts, according to that score. Particular attention is paid to relative predictive performance under misspecification of the predictive model. Using numerical illustrations, we document several novel findings within this paradigm that highlight the important interplay between the true data generating process, the assumed predictive model and the scoring rule. Notably, we show that only when a predictive model is sufficiently compatible with the true process to allow a particular score criterion to reward what it is designed to reward, will this approach to forecasting reap benefits. Subject to this compatibility, however, the superiority of the optimal forecast will be greater, the greater is the degree of misspecification. We explore these issues under a range of different scenarios and using both artificially simulated and empirical data.

Combining a DSGE Model with Variational Bayesian Neural Networks

In the spirit of the DSGE-VAR approach, I employed Temporal Difference Variational Auto-Encoder (TDVAE) proposed by Gregor et al. (2019) as a reduced timeseries model on which theoretical restrictions from a DSGE model are imposed. TDVAE is virtually a general-form stochastic state-space model implemented via variational Bayesian inference (VI) with neural networks. With the flexibility of neural networks, this DSGE-TDVAE approach tries to address the Consolo et al. (2009)’s criticism on DSGE-VAR that an unrestricted VAR may not represent the data correctly due to its shortage in expressiveness. The empirical results showed that forecast performance is certainly gained by combining the standard New Keynesian DSGE with TDVAE for the actual macroeconomic data of Japan. It confirmed that some support for the DSGE model is still found in the data even when evaluated against the more general statistical benchmark of TDVAE rather than VAR. At the same time, this indicated that the DSGE holds considerable degree of misspecification as the optimal intensity of DSGE restriction was small. Furthermore, DSGE-TDVAE achieved the superior forecasting performance to DSGE-VAR.

Evaluating performance of covariate-constrained randomization (CCR) techniques under misspecification of cluster-level variables in cluster-randomized trials

Covariate constrained randomization (CCR) is a method of controlling imbalance in important baseline covariates in cluster-randomized trials (CRT). We use simulated CRTs to investigate the performance (control of imbalance) of CCR relative to simple randomization (SR) under conditions of misspecification of the cluster-level variable used in the CCR algorithm.We use data from a Patient-Centered Outcomes Research Institute (PCORI)-funded CRT evaluating the Mothers and Babies (MB) intervention (AD-1507-31,473). CCR methodology was used in the MB study to control imbalance in, among other baseline variables, the percent minority (i.e., non-White) participants at each study site. Simulation schemes explored variation in degree of misspecification in the baseline covariate of interest, and include correct report, observed misspecification, and a range of simulated misspecification for intervals within and beyond that observed in the MB study. We also consider three within-site sample size scenarios: that observed in the MB study, small (mean 10) and large (mean 50). Simulations at every level of baseline covariate misspecification suggest that use of the CCR strategy provides between-arm imbalance that is simultaneously lower and less variable, on average, than that produced from the SR strategy. We find that the gains to using CCR over SR are nearly twice as high with accurate reporting (Δ = −5.33) compared to the observed study-level misspecification (Δ = −3.03). Although CCR still outperforms SR as the level of misspecification increases, the gains to using CCR over SR decrease; thus, every effort should still be made to obtain high-quality baseline data.

Latent likelihood ratio tests for assessing spatial kernels in epidemic models

One of the most important issues in the critical assessment of spatio-temporal stochastic models for epidemics is the selection of the transmission kernel used to represent the relationship between infectious challenge and spatial separation of infected and susceptible hosts. As the design of control strategies is often based on an assessment of the distance over which transmission can realistically occur and estimation of this distance is very sensitive to the choice of kernel function, it is important that models used to inform control strategies can be scrutinised in the light of observation in order to elicit possible evidence against the selected kernel function. While a range of approaches to model criticism is in existence, the field remains one in which the need for further research is recognised. In this paper, building on earlier contributions by the authors, we introduce a new approach to assessing the validity of spatial kernels—the latent likelihood ratio tests—which use likelihood-based discrepancy variables that can be used to compare the fit of competing models, and compare the capacity of this approach to detect model mis-specification with that of tests based on the use of infection-link residuals. We demonstrate that the new approach can be used to formulate tests with greater power than infection-link residuals to detect kernel mis-specification particularly when the degree of mis-specification is modest. This new tests avoid the use of a fully Bayesian approach which may introduce undesirable complications related to computational complexity and prior sensitivity.

Identification strength with a large number of moments

This paper studies how identification is affected in GMM estimation as the number of moment conditions increases. We develop a general asymptotic theory extending the set up of Chao and Swanson and Antoine and Renault to the case where moment conditions have heterogeneous identification strengths and the number of them may diverge to infinity with the sample size. We also allow the models to be locally misspecified and examine how the asymptotic theory is affected by the degree of misspecification. The theory encompasses many cases including GMM models with many moments (Han and Phillips), partially linear models, and local GMM via kernel smoothing with a large number of conditional moment restrictions. We provide an understanding of the benefits of a large number of moments that compensate the weakness of individual moments by explicitly showing how an increasing number of moments improves the rate of convergence in GMM.

Statistical analysis of longitudinal data on tumour growth in mice experiments

We consider mice experiments where tumour cells are injected so that a tumour starts to grow. When the tumour reaches a certain volume, mice are randomized into treatment groups. Tumour volume is measured repeatedly until the mouse dies or is sacrificed. Tumour growth rates are compared between groups. We propose and evaluate linear regression for analysis accounting for the correlation among repeated measurements per mouse. More specifically, we examined five models with three different variance-covariance structures in order to recommend the least complex method for small to moderate sample sizes encountered in animal experiments. We performed a simulation study based on data from three previous experiments to investigate the properties of estimates of the difference between treatment groups. Models were estimated via marginal modelling using generalized least squares and restricted maximum likelihood estimation. A model with an autoregressive (AR-1) covariance structure was efficient and unbiased retaining nominal coverage and type I error when the AR-1 variance-covariance matrix correctly specified the association between repeated measurements. When the variance-covariance was misspecified, that model was still unbiased but the type I error and the coverage rates were affected depending on the degree of misspecification. A linear regression model with an autoregressive (AR-1) covariance structure is an adequate model to analyse experiments that compare tumour growth rates between treatment groups.

On uniform asymptotic risk of averaging GMM estimators

This paper studies the averaging GMM estimator that combines a conservative GMM estimator based on valid moment conditions and an aggressive GMM estimator based on both valid and possibly misspecified moment conditions, where the weight is the sample analog of an infeasible optimal weight. We establish asymptotic theory on uniform approximation of the upper and lower bounds of the finite‐sample truncated risk difference between any two estimators, which is used to compare the averaging GMM estimator and the conservative GMM estimator. Under some sufficient conditions, we show that the asymptotic lower bound of the truncated risk difference between the averaging estimator and the conservative estimator is strictly less than zero, while the asymptotic upper bound is zero uniformly over any degree of misspecification. The results apply to quadratic loss functions. This uniform asymptotic dominance is established in non‐Gaussian semiparametric nonlinear models.

Performance of Nonrecursive Latent Variable Models under Misspecification

A problem central to structural equation modeling is measurement model specification error and its propagation into the structural part of nonrecursive latent variable models. Full-information estimation techniques such as maximum likelihood are consistent when the model is correctly specified and the sample size large enough; however, any misspecification within the model can affect parameter estimates in other parts of the model. The goals of this study included comparing the bias, efficiency, and accuracy of hypothesis tests in nonrecursive latent variable models with indirect and direct feedback loops. We compare the performance of maximum likelihood, two-stage least-squares and Bayesian estimators in nonrecursive latent variable models with indirect and direct feedback loops under various degrees of misspecification in small to moderate sample size conditions.

On Lagrange Multiplier Tests in Multidimensional Item Response Theory: Information Matrices and Model Misspecification.

Lagrange multiplier (LM) or score tests have seen renewed interest for the purpose of diagnosing misspecification in item response theory (IRT) models. LM tests can also be used to test whether parameters differ from a fixed value. We argue that the utility of LM tests depends on both the method used to compute the test and the degree of misspecification in the initially fitted model. We demonstrate both of these points in the context of a multidimensional IRT framework. Through an extensive Monte Carlo simulation study, we examine the performance of LM tests under varying degrees of model misspecification, model size, and different information matrix approximations. A generalized LM test designed specifically for use under misspecification, which has apparently not been previously studied in an IRT framework, performed the best in our simulations. Finally, we reemphasize caution in using LM tests for model specification searches.

Spurious Inference in Reduced-Rank Asset-Pricing Models

This note studies some seemingly anomalous results that arise in possibly misspecified, reduced-rank linear asset-pricing models estimated by the continuously updated generalized method of moments. When a spurious factor (that is, a factor that is uncorrelated with the returns on the test assets) is present, the test for correct model specification has asymptotic power that is equal to the nominal size. In other words, applied researchers will erroneously conclude that the model is correctly specified even when the degree of misspecification is arbitrarily large. The rejection probability of the test for overidentifying restrictions typically decreases further in underidentified models where the dimension of the null space is larger than 1.

Credit Value Adjustment with Market-Implied Recovery

We present a model for CVA calculation in which the recovery rate is inferred from the term structure of CDS spreads. We show that the assumption of constant recovery is inconsistent with observed CDS spreads and leads to substantial underestimation of the CVA. We further document that the degree of misspecification induced by the constant recovery assumption is affected by the ratings of the two counterparties and the phase of the business cycle. Assuming constant recovery biases CVA calculations through two channels: estimating the default probability and measuring the impact of correlation and the associated wrong-way risk.

Spurious Inference in Unidentified Asset-Pricing Models

This paper studies some seemingly anomalous results that arise in possibly misspecified and unidentified linear asset-pricing models estimated by maximum likelihood and one-step generalized method of moments (GMM). Strikingly, when useless factors (that is, factors that are independent of the returns on the test assets) are present, the models exhibit perfect fit, as measured by the squared correlation between the model's fitted expected returns and the average realized returns, and the tests for correct model specification have asymptotic power that is equal to the nominal size. In other words, applied researchers will erroneously conclude that the model is correctly specified even when the degree of misspecification is arbitrarily large. We also derive the highly nonstandard limiting behavior of these invariant estimators and their t-tests in the presence of identification failure. These results reveal the spurious nature of inference as useless factors are selected with high probability, while useful factors are driven out from the model. The practical relevance of our findings is demonstrated using simulations and an empirical application.

Uniform Asymptotic Risk of Averaging GMM Estimator Robust to Misspecification, Second Version

This paper studies the averaging generalized method of moments (GMM) estimator that combines a conservative GMM estimator based on valid moment conditions and an aggressive GMM estimator based on both valid and possibly misspecified moment conditions, where the weight is the sample analog of an infeasible optimal weight. It is an alternative to pre-test estimators that switch between the conservative and aggressive estimators based on model specification tests. This averaging estimator is robust in the sense that it uniformly dominates the conservative estimator by reducing the risk under any degree of misspecification, whereas the pre-test estimators reduce the risk in parts of the parameter space and increase it in other parts. To establish uniform dominance of one estimator over another, we establish asymptotic theories on uniform approximations of the finite-sample risk differences between two estimators. These asymptotic results are developed along drifting sequences of data generating processes (DGPs) that model various degrees of local misspecification as well as global misspecification. Extending seminal results on the James-Stein estimator, the uniform dominance is established in non-Gaussian semiparametric nonlinear models. The proposed averaging estimator is applied to estimate the human capital production function in a life-cycle labor supply model.

An Investigation of the Factor Structure of the 16PF, Version 5

In order to examine its higher-order factor structure, we applied confirmatory factor and invariance analysis to item level data from the US standardization sample of the 16PF5, divided into a calibration sample (N = 5,130) and a validation sample (N = 5,131). Using standard assessments of model fit, all primary factors displayed good to excellent model fit, thus suggesting the scales to be broadly unidimensional. Results indicated a drop in model fit in both the structural and configurally invariant second order models, suggesting some level of misspecification in the global scales of Extraversion, Anxiety, Tough-Mindedness, Independence, and Self-Control. However, the degree of misspecification was slight. Overall, the analyses generally supported the proposed structure of the 16PF5.

Using Fit Indexes to Select a Covariance Model for Longitudinal Data

This study investigated the performance of fit indexes in selecting a covariance structure for longitudinal data. Data were simulated to follow a compound symmetry, first-order autoregressive, first-order moving average, or random-coefficients covariance structure. We examined the ability of the likelihood ratio test (LRT), root mean square error of approximation (RMSEA), comparative fit index (CFI), and Tucker–Lewis Index (TLI) to reject misspecified models with varying degrees of misspecification. With a sample size of 20, RMSEA, CFI, and TLI are high in both Type I and Type II error rates, whereas LRT has a high Type II error rate. With a sample size of 100, these indexes generally have satisfactory performance, but CFI and TLI are affected by a confounding effect of their baseline model. Akaike's Information Criterion (AIC) and Bayesian Information Criterion (BIC) have high success rates in identifying the true model when sample size is 100. A comparison with the mixed model approach indicates that separately modeling the means and covariance structures in structural equation modeling dramatically improves the success rate of AIC and BIC.

Merging expert and empirical data for rare event frequency estimation: Pool homogenisation for empirical Bayes models

Empirical Bayes provides one approach to estimating the frequency of rare events as a weighted average of the frequencies of an event and a pool of events. The pool will draw upon, for example, events with similar precursors. The higher the degree of homogeneity of the pool, then the Empirical Bayes estimator will be more accurate. We propose and evaluate a new method using homogenisation factors under the assumption that events are generated from a Homogeneous Poisson Process. The homogenisation factors are scaling constants, which can be elicited through structured expert judgement and used to align the frequencies of different events, hence homogenising the pool. The estimation error relative to the homogeneity of the pool is examined theoretically indicating that reduced error is associated with larger pool homogeneity. The effects of misspecified expert assessments of the homogenisation factors are examined theoretically and through simulation experiments. Our results show that the proposed Empirical Bayes method using homogenisation factors is robust under different degrees of misspecification.

On the Fit of New Keynesian Models

This article provides new tools for the evaluation of dynamic stochastic general equilibrium (DSGE) models and applies them to a large-scale new Keynesian model. We approximate the DSGE model by a vector autoregression, and then systematically relax the implied cross-equation restrictions and document how the model fit changes. We also compare the DSGE model's impulse responses to structural shocks with those obtained after relaxing its restrictions. We find that the degree of misspecification in this large-scale DSGE model is no longer so large as to prevent its use in day-to-day policy analysis, yet is not small enough to be ignored.