Detecting Model Misspecification Research Articles

A Chi-Square Goodness-of-Fit Test for Autoregressive Logistic Regression Models with Applications to Patient Screening

We propose a chi-square goodness-of-fit test for autoregressive logistic regression models. General guidelines for a two-dimensional binning strategy are provided, which make use of two types of maximum likelihood parameter estimates. For smaller sample sizes, a bootstrap p-value procedure is discussed. Simulation studies indicate that the test procedure satisfactorily approximates the correct size and has good power for detecting model misspecification. In particular, the test is very good at detecting the need for an additional lag. An application to a dataset relating to screening patients for late-onset Alzheimer’s disease is provided.

Detecting Model Misspecification in Inflated Beta Regressions

Beta regression models are useful for modeling data that assume values in the standard unit interval (0, 1), such as rates and proportions. These models, however, cannot be used when the data contain observations that equal zero or one (the limits of the unit interval). Ospina and Ferrari (2012) developed the class of inflated beta regressions to handle situations in which the data contain positive mass at zero and/or one. The model is quite general and contains three submodels: for the mean, for the precision, and for the probability that the variate equals zero or one (α). In this article, we propose a misspecification test for inflated beta regressions with fixed and variable dispersion. In particular, we propose two variants of the test. In the first variant, we only add testing variables to the mean submodel. The second variant follows from adding testing variables to all submodels. We perform extensive Monte Carlo simulations in order to assess the finite-sample properties of the tests (size and power), and also to gain insight on which variables should be used as testing variable and on which asymptotic testing procedure delivers the most reliable inferences. We consider a number of different misspecifications, namely: neglected nonlinearities, omitted independent variables, incorrect link functions, neglected spatial correlation, and neglected variable dispersion. Finally, an empirical illustration is presented and discussed.

Specification Tests of Calibrated Option Pricing Models

In spite of the popularity of model calibration in finance, empirical researchers have put more emphasis on model estimation than on the equally important goodness-of-fit problem. This is due partly to the ignorance of modelers, and more to the ability of existing statistical tests to detect specification errors. In practice, models are often calibrated by minimizing the sum of squared difference between the modelled and actual observations. It is challenging to disentangle model error from estimation error in the residual series. To circumvent the difficulty, we study an alternative way of estimating the model by exact calibration. We argue that standard time series tests based on the exact approach can better reveal model misspecifications than the error minimizing approach. In the context of option pricing, we illustrate the usefulness of exact calibration in detecting model misspecification. Under heteroskedastic observation error structure, our simulation results shows that the Black-Scholes model calibrated by exact approach delivers more accurate hedging performance than that calibrated by error minimization.

The Ability for Posterior Predictive Checking to Identify Model Misspecification in Bayesian Growth Mixture Modeling

Proper model specification is an issue for researchers, regardless of the estimation framework being utilized. Typically, indexes are used to compare the fit of one model to the fit of an alternate model. These indexes only provide an indication of relative fit and do not necessarily point toward proper model specification. There is a procedure in the Bayesian framework called posterior predictive checking that is designed theoretically to detect model misspecification for observed data. However, the performance of the posterior predictive check procedure has thus far not been directly examined under different conditions of mixture model misspecification. This article addresses this task and aims to provide additional insight into whether or not posterior predictive checks can detect model misspecification within the context of Bayesian growth mixture modeling. Results indicate that this procedure can only identify mixture model misspecification under very extreme cases of misspecification.

Why Should Prediction Discrepancies Be Renamed Standardized Visual Predictive Check?

We have read the paper by Wang and Zhang[1]. We are pleased to see that other authors further illustrate and also find nice properties for an evaluation tool that we first used more than ten years ago[2], presented at PAGE in 20003 and published extensively first in 2006[4]. However we disagree with some of the conclusions made by the authors, and we feel that an important reference should have been included in their manuscript. As Wang and Zhang point out themselves, what they call SVPC is nothing other than the prediction discrepancies (pd) named that way, after being called pseudo-residuals by Mentre and Escolano in 2006[4]. It is therefore misleading to present SVPC as something novel when in fact it goes back on something that our group has published and presented in conferences. Plots of SVPC versus time in the paper by Wang and Zhang[1] are very similar to plots of pd versus time in Mentre and Escolano[4], therefore we disagree with the statement on page 2 that “Neither pd nor npde was intended/recommended for evaluation of model predictions over a time course”. pd and npde were developed for their improved statistical properties over linearisation-based residuals but are used as visual diagnostic tools in a similar way. Also the paper is not complete when it comes to the current state of literature. Wang and Zang[1] missed one important reference, in which we have compared pd, npde, VPC, as well as tests based on prediction intervals with or without decorrelation[5]. We have also proposed tests for covariate models illustrating how npde can be used to evaluate covariate models. Their paper is rather inaccurate when comparing the properties of the pd to those of the npde. The idea behind the npde, as recalled in the paper, is to decorrelate observations within individuals. In Mentre and Escolano[4], it had been shown that the type I error of the test based on pd was inflated when subjects contribute several observations, because of the within-subject correlation between observations, and it was anticipated that the decorrelation would improve this feature. Indeed, in Brendel et al.[5] we have shown that the type I error of the npde is close to the expected 5% while the type I error for the pd or the test based on prediction intervals of the VPC was larger. In that paper, we performed a full simulation study, i.e. with several replications of the simulated data set, while in the manuscript of Wang and Zhang only one simulated dataset is given, from which it is very hard to draw meaningful conclusions. It is quite possible that for one simulated dataset the pd detects something that the npde does not, indeed, our simulations have shown that on a large number of simulated datasets, the tests with npde maintain type I error while the tests with pd have an increased type I error and hence a higher power. A discussion of the power to detect model misspecification should therefore take into account the increase in type I error under the null hypothesis. Also in their table V, there was a significant departure from 0 of the mean npde for the wrong model (p=0.03 with a Wilcoxon test). We would also like to take this opportunity to mention that more informative VPC graphs have been proposed by Wilkins[6] et al in 2006, which use prediction intervals around the simulated percentiles, and that we have adapted these graphs to pd and npde[7]. In that paper, we made a case of using pd instead of npde to plot diagnostic graphs because the decorrelation tends to blur the relationship with time when used for visual diagnostics. Finally, we join Wang and Zhang to stress the nice properties of simulation-based model evaluation tools such as pd (SVPC) over VPC and to encourage readers to use them.

Goodness-of-fit measures for individual-level models of infectious disease in a Bayesian framework

In simple models there are a variety of tried and tested ways to assess goodness-of-fit. However, in complex non-linear models, such as spatio-temporal individual-level models, less research has been done on how best to ascertain goodness-of-fit. Often such models are fitted within a Bayesian statistical framework, since such a framework is ideally placed to account for the many areas of data uncertainty. Within a Bayesian context, a major tool for assessing goodness-of-fit is the posterior predictive distribution. That is, a distribution for a test statistic is found through simulation from the posterior distribution and then compared with the observed test statistic for the data. Here, we examine different test statistics and ascertain how well they can detect model misspecification via a simulation study.

Detecting Growth Shape Misspecifications in Latent Growth Models: An Evaluation of Fit Indexes

In this study, the authors compared the likelihood ratio test and fit indexes for detection of misspecifications of growth shape in latent growth models through a simulation study and a graphical analysis. They found that the likelihood ratio test, MFI, and root mean square error of approximation performed best for detecting model misspecification when a linear model was fit to scores presenting nonlinear growth trajectories, in terms of being sensitive to severity of misspecification, and providing stable results with different types of nonlinearity and sample sizes.

Diagnostic Measures for Generalized Linear Models with Missing Covariates

In this paper, we carry out an in-depth investigation of diagnostic measures for assessing the influence of observations and model misspecification in the presence of missing covariate data for generalized linear models. Our diagnostic measures include case-deletion measures and conditional residuals. We use the conditional residuals to construct goodness-of-fit statistics for testing possible misspecifications in model assumptions, including the sampling distribution. We develop specific strategies for incorporating missing data into goodness-of-fit statistics in order to increase the power of detecting model misspecification. A resampling method is proposed to approximate the p-value of the goodness-of-fit statistics. Simulation studies are conducted to evaluate our methods and a real data set is analysed to illustrate the use of our various diagnostic measures.

Graphical diagnostics to check model misspecification for the proportional odds regression model.

The cumulative logit or the proportional odds regression model is commonly used to study covariate effects on ordinal responses. This paper provides some graphical and numerical methods for checking the adequacy of the proportional odds regression model. The methods focus on evaluating functional misspecification for specific covariate effects, but misspecification of the link function can also be dealt with under the same framework. For the logistic regression model with binary responses, Arbogast and Lin (Statist. Med. 2005; 24:229-247) developed similar graphical and numerical methods for assessing the adequacy of the model using the cumulative sums of residuals. The paper generalizes their methods to ordinal responses and illustrates them using an example from the VA Normative Aging Study. Simulation studies comparing the performance of the different diagnostic methods indicate that some of the graphical methods are more powerful in detecting model misspecification than the Hosmer-Lemeshow-type goodness-of-fit statistics for the class of models studied.

Increasing the power: A practical approach to goodness-of-fit test for logistic regression models with continuous predictors

When continuous predictors are present, classical Pearson and deviance goodness-of-fit tests to assess logistic model fit break down. The Hosmer–Lemeshow test can be used in these situations. While simple to perform and widely used, it does not have desirable power in many cases and provides no further information on the source of any detectable lack of fit. Tsiatis proposed a score statistic to test for covariate regional effects. While conceptually elegant, its lack of a general rule for how to partition the covariate space has, to a certain degree, limited its popularity. We propose a new method for goodness-of-fit testing that uses a very general partitioning strategy (clustering) in the covariate space and either a Pearson statistic or a score statistic. Properties of the proposed statistics are discussed, and a simulation study demonstrates increased power to detect model misspecification in a variety of settings. An application of these different methods on data from a clinical trial illustrates their use. Discussions on further improvement of the proposed tests and extending this new method to other data situations, such as ordinal response regression models are also included.

Clustering Time Series with Clipped Data

Clustering time series is a problem that has applications in a wide variety of fields, and has recently attracted a large amount of research. Time series data are often large and may contain outliers. We show that the simple procedure of clipping the time series (discretising to above or below the median) reduces memory requirements and significantly speeds up clustering without decreasing clustering accuracy. We also demonstrate that clipping increases clustering accuracy when there are outliers in the data, thus serving as a means of outlier detection and a method of identifying model misspecification. We consider simulated data from polynomial, autoregressive moving average and hidden Markov models and show that the estimated parameters of the clipped data used in clustering tend, asymptotically, to those of the unclipped data. We also demonstrate experimentally that, if the series are long enough, the accuracy on clipped data is not significantly less than the accuracy on unclipped data, and if the series contain outliers then clipping results in significantly better clusterings. We then illustrate how using clipped series can be of practical benefit in detecting model misspecification and outliers on two real world data sets: an electricity generation bid data set and an ECG data set.

Testing Exogeneity in Cross-Section Regression by Sorting Data

We introduce a framework to test for exogeneity of a variable in a regression based on cross-sectional data. By sorting data with respect to a function (sorting score) of known exogeneous variables it is possible to utilize a battery of tools originally develped to detecting model misspecification in at time series context. Thus, we are able to propose graphical tools for the identification of endogeneity, as well as formal tests, including a simple-to-use Chow test, needing a minimum of assumptions on the alternative endogeneity hypothesis. Models of endogenous treatment and selectivity are utilized to illustrate the methods. With Monte Carlo experiments, including continous and discrete response cases, we compare small sample performances with existing tests for exogeneity.

Estimation of HIV dynamic parameters.

Investigation of HIV viral dynamics is important for understanding the HIV pathogenesis and for development of treatment strategies. Perelson et al. demonstrated that simple viral dynamic models fit to data on viral load as measured by plasma HIV-RNA could produce estimates of rates of clearance of virus and of infected CD4+ T-lymphocytes. In this paper we extend the work of Perelson et al. by proposing models with less restrictive assumptions about drug activity. Our models take into account the fact that infectious and non-infectious virions are produced by infected T-cells both before and after the treatment. We also show that direct measurement of infectious virus load provides sufficient information for estimation of antiretroviral drug efficacy parameter. For characterizing viral dynamics of populations and estimation of dynamic parameters, we propose a hierarchical non-linear model. Compared to other methods such as the non-linear least square method used by Perelson et al., we show that the proposed approach has the following advantages: (i) it is more appropriate for modelling within-patient and between-patient variation and to characterize the population dynamics; (ii) it is flexible enough to deal with both rich and sparse individual data; (iii) it has more power to detect model misspecification; (iv) it allows incorporation of covariates for viral dynamic parameters; (v) it makes more efficient use of between-subject information to get better parameter estimates. We give two simulation examples to illustrate the proposed approach and its advantages. Finally, we discuss practical issues regarding the clinical trial design for viral dynamic studies.

An Asymptotic Theory for Estimating Beta‐Pricing Models Using Cross‐Sectional Regression

ABSTRACTWithout the assumption of conditional homoskedasticity, a general asymptotic distribution theory for the two‐stage cross‐sectional regression method shows that the standard errors produced by the Fama–MacBeth procedure do not necessarily overstate the precision of the risk premium estimates. When factors are misspecified, estimators for risk premiums can be biased, and the t‐value of a premium may converge to infinity in probability even when the true premium is zero. However, when a beta‐pricing model is misspecified, the t‐values for firm characteristics generally converge to infinity in probability, which supports the use of firm characteristics in cross‐sectional regressions for detecting model misspecification.

Empirical Analysis of a Dynamic Duopoly Model of Competition

Empirically validating and testing the specification of game theoretic models has received limited attention in the marketing literature. The authors provide an econometric framework for estimating the parameters of response functions when the observed data in the market place are the Nash equilibrium outcomes of an underlying dynamic duopoly game specification. Specifically, the estimation procedure accounts for the joint endogeneity of market shares and marketing efforts of market rivals using a system of simultaneous equations that included the market response function and the Nash equilibrium conditions. A formal statistical test is used to detect model misspecification. The empirical analysis is carried out using data from four product markets: pharmaceutical, soft drink, beer, and detergent. Comparisons are provided with conventional estimation of the response function parameters in which the equilibrium conditions are ignored in the estimation. Managerial implications of the empirical results are discussed.

RECRES: A FORTRAN-77 program for detecting model misspecification in multiple linear regression

RECRES is an interactive FORTRAN-77 program for detecting model misspecification in multiple linear regression problems. The program produces plots of ordinary least-squares (OLS) residuals, normal probability plots of recursive residuals, cumulative sums of recursive residuals, the Quandt log-likelihood statistic, and recursive estimates of the model parameters. It uses orthogonal triangularization to compute diagnostic statistics and allows flexibility in the production of diagnostic plots. Examples show that the program yields reliable results and can detect model misspecifications that are not apparent readily to users of conventional multiple linear regression programs. A source listing of the program and an example of an interactive session are provided.

Residual analysis in the grouped and censored normal linear model

Graphical and numerical analysis of residual can be informative about model misspecification even when data are censored or grouped. This paper provides simple procedures for calculating diagnostic statistics to detect model misspecification when grouped or censored data are analysed using a normal linear model. Graphs can reveal the nature of the correlations that these statistics pick up but when grouping or censoring is severe they can be difficult to interpret. We discuss the processing of graphs so that they are more easily interpreted and provide examples based on artificial data and on grouped data relating to unemployment durations.