Consistent Information Criterion Research Articles

PanIC: Consistent information criteria for general model selection problems

PanIC: Consistent information criteria for general model selection problems

Blockwise AICc and its consistency properties in model selection

Akaike information criterion (AIC) and corrected Akaike information criterion (AICc) are two widely used information criteria. It is well known that neither of them is consistent because there is a positive probability to select an over-specified candidate model. In this paper, with the assumption that the sample size tends to be infinite, we derive the probability of the true model’s AICc (AIC) value less than an over-specified model’s AICc (AIC) value, and we also derive the lower bound of probability of selecting the true model using AICc (AIC) when the candidate model set includes all possible candidate models. We also prove that blockwise AICc, a new information criterion, is a consistent information criterion if the number of blocks and sample size both tend to be infinite. Furthermore, compared with the other popular information criteria, simulations and real data analysis also show that bAICc performs well for moderate and large sample sizes.

Errors in Statistical Inference Under Model Misspecification: Evidence, Hypothesis Testing, and AIC.

The methods for making statistical inferences in scientific analysis have diversified even within the frequentist branch of statistics, but comparison has been elusive. We approximate analytically and numerically the performance of Neyman-Pearson hypothesis testing, Fisher significance testing, information criteria, and evidential statistics (Royall, 1997). This last approach is implemented in the form of evidence functions: statistics for comparing two models by estimating, based on data, their relative distance to the generating process (i.e., truth) (Lele, 2004). A consequence of this definition is the salient property that the probabilities of misleading or weak evidence, error probabilities analogous to Type 1 and Type 2 errors in hypothesis testing, all approach 0 as sample size increases. Our comparison of these approaches focuses primarily on the frequency with which errors are made, both when models are correctly specified, and when they are misspecified, but also considers ease of interpretation. The error rates in evidential analysis all decrease to 0 as sample size increases even under model misspecification. Neyman-Pearson testing on the other hand, exhibits great difficulties under misspecification. The real Type 1 and Type 2 error rates can be less, equal to, or greater than the nominal rates depending on the nature of model misspecification. Under some reasonable circumstances, the probability of Type 1 error is an increasing function of sample size that can even approach 1! In contrast, under model misspecification an evidential analysis retains the desirable properties of always having a greater probability of selecting the best model over an inferior one and of having the probability of selecting the best model increase monotonically with sample size. We show that the evidence function concept fulfills the seeming objectives of model selection in ecology, both in a statistical as well as scientific sense, and that evidence functions are intuitive and easily grasped. We find that consistent information criteria are evidence functions but the MSE minimizing (or efficient) information criteria (e.g., AIC, AICc, TIC) are not. The error properties of the MSE minimizing criteria switch between those of evidence functions and those of Neyman-Pearson tests depending on models being compared.

Model selection with estimated factors and idiosyncratic components

SummaryThis paper provides consistent information criteria for the selection of forecasting models that use a subset of both the idiosyncratic and common factor components of a big dataset. This hybrid model approach has been explored by recent empirical studies to relax the strictness of pure factor‐augmented model approximations, but no formal model selection procedures have been developed. The main difference to previous factor‐augmented model selection procedures is that we must account for estimation error in the idiosyncratic component as well as the factors. Our main contribution is to show the conditions required for selection consistency of a class of information criteria that reflect this additional source of estimation error. We show that existing factor‐augmented model selection criteria are inconsistent in circumstances where N is of larger order than , where N and T are the cross‐section and time series dimensions of the dataset respectively, and that the standard Bayesian information criterion is inconsistent regardless of the relationship between N and T. We therefore propose a new set of information criteria that guarantee selection consistency in the presence of estimated idiosyncratic components. The properties of these new criteria are explored through a Monte Carlo simulation study. The paper concludes with an empirical application to long‐horizon exchange rate forecasting using a recently proposed model with country‐specific idiosyncratic components from a panel of global exchange rates.

A consistent information criterion for support vector machines in diverging model spaces

Information criteria have been popularly used in model selection and proved to possess nice theoretical properties. For classification, Claeskens et al. (2008) proposed support vector machine information criterion for feature selection and provided encouraging numerical evidence. Yet no theoretical justification was given there. This work aims to fill the gap and to provide some theoretical justifications for support vector machine information criterion in both fixed and diverging model spaces. We first derive a uniform convergence rate for the support vector machine solution and then show that a modification of the support vector machine information criterion achieves model selection consistency even when the number of features diverges at an exponential rate of the sample size. This consistency result can be further applied to selecting the optimal tuning parameter for various penalized support vector machine methods. Finite-sample performance of the proposed information criterion is investigated using Monte Carlo studies and one real-world gene selection problem.

Finite sample performance of the model selection approach in co-integration analysis

We investigate the finite sample performances of three consistent information criteria, Hannan and Quinn (HQ), Schwarz information criterion (SIC), and Posterior information criterion (PIC), in the selection of co-integrating rank, or in joint determination of the lag order and co-integrating rank of vector error correction models. No single approach dominates the others across all specifications. When compared with SIC and PIC, HQ is less sensitive to parameter distributions and the variance structure of innovations. The performance of SIC is close to that of the maximum eigenvalue test in models with both small and large sample sizes. PIC performs better than SIC in small sample sizes when the model does not contain a trend, the trend signal is weak, or when the stationary component is not close to a non-stationary one. PIC also performs better than SIC and to a lesser extent than HQ in a real world data-based simulation, where the data generating processes (DGP) has high dimension and includes several exogenous variables.

A parametric estimation method for dynamic factor models of large dimensions

The estimation of dynamic factor models for large sets of variables has attracted considerable attention recently, due to the increased availability of large datasets. In this paper we propose a new parametric methodology for estimating factors from large datasets based on state space models and discuss its theoretical properties. In particular, we show that it is possible to estimate consistently the factor space. We also develop a consistent information criterion for the determination of the number of factors to be included in the model. Finally, we conduct a set of simulation experiments that show that our approach compares well with existing alternatives.

A Parametric Estimation Method for Dynamic Factor Models of Large Dimensions

The estimation of dynamic factor models for large sets of variables has attracted considerable attention recently, due to the increased availability of large datasets. In this paper we propose a new parametric methodology for estimating factors from large datasets based on state space models and discuss its theoretical properties. In particular, we show that it is possible to estimate consistently the factor space. We also develop a consistent information criterion for the determination of the number of factors to be included in the model. Finally, we conduct a set of simulation experiments that show that our approach compares well with existing alternatives.

Is model selection using Akaike's information criterion appropriate for catch per unit effort standardization in large samples?

Akaike’s information criterion (A/C), which is widely used as a criterion of model selection in fish population dynamics, is known to have a bias in not only small samples but also large samples. Consistency was proposed as a property of the information criteria available in large samples. We carried out model selection in ANOVA-type model corresponding to catch per unit effort (CPUE) standardization using consistent information criteria (Bayesian information criterion, Hannan-Quinn, or consistent A/C), which satisfy the asymptotic desirable property called consistency. The results of the model selections between these consistent criteria and A/C are different. Computer simulations using a linear regression model show that the selection performances of consistent information criteria in large samples are good compared with that of A/C.

A strongly consistent information criterion for linear model selection based on M-estimation

In this paper, a linear model selection procedure based on M-estimation is proposed, which includes many classical model selection criteria as its special cases. It is shown that the proposed criterion is strongly consistent under certain mild conditions, for instance without assuming normality of the distribution of the random errors. The results from a simulation study are also presented.