Generalized Information Criterion Research Articles

The E-MS Algorithm: Model Selection With Incomplete Data

We propose a procedure associated with the idea of the E-M algorithm for model selection in the presence of missing data. The idea extends the concept of parameters to include both the model and the parameters under the model, and thus allows the model to be part of the E-M iterations. We develop the procedure, known as the E-MS algorithm, under the assumption that the class of candidate models is finite. Some special cases of the procedure are considered, including E-MS with the generalized information criteria (GIC), and E-MS with the adaptive fence (AF; Jiang et al.). We prove numerical convergence of the E-MS algorithm as well as consistency in model selection of the limiting model of the E-MS convergence, for E-MS with GIC and E-MS with AF. We study the impact on model selection of different missing data mechanisms. Furthermore, we carry out extensive simulation studies on the finite-sample performance of the E-MS with comparisons to other procedures. The methodology is also illustrated on a real data analysis involving QTL mapping for an agricultural study on barley grains. Supplementary materials for this article are available online.

Model Order Selection Based on Information Theoretic Criteria: Design of the Penalty

Information theoretic criteria (ITC) have been widely adopted in engineering and statistics for selecting among an ordered set of candidate models the one that better fits the observed sample data. The selected model minimizes a penalized likelihood metric, where the penalty is determined by the criterion adopted. While rules for choosing a penalty that guarantees a consistent estimate of the model order are known, theoretical tools for its design with finite samples have never been provided in a general setting. In this paper, we study model order selection for finite samples under a design perspective, focusing on the generalized information criterion (GIC), which embraces the most common ITC. The theory is general, and as case studies we consider: a) the problem of estimating the number of signals embedded in additive white Gaussian noise (AWGN) by using multiple sensors; b) model selection for the general linear model (GLM), which includes, e.g., the problem of estimating the number of sinusoids in AWGN. The analysis reveals a trade-off between the probabilities of overestimating and underestimating the order of the model. We then propose to design the GIC penalty to minimize underestimation while keeping the overestimation probability below a specified level. For the considered problems this method leads to analytical derivation of the optimal penalty for a given sample size. A performance comparison between the penalty optimized GIC and common AIC and BIC is provided, demonstrating the effectiveness of the proposed design strategy.

Multimodel adaptive subtraction with regularized parameter selection via generalized information criterion

Adaptive subtraction remains a critical part of today’s seismic data processing. Many existing adaptive subtraction methods can be formulated as a parameter estimation problem, and all of them, although fundamentally different, have a common restriction that the dimension of the unknown parameters must be determined in advance. We have developed an adaptive subtraction framework called multimodel adaptive subtraction (MMAS) that aims to relax this restriction as well as regularize the estimation of the parameters through a generalized information criterion combined with a nonconcave penalized likelihood function. MMAS is a fully general framework that can be applied to many existing adaptive subtraction methods. As an example, we determined how MMAS can be applied to the popular least-squares adaptive subtraction (LSAS) method, and we call the resulting algorithm the multimodel LSAS ([Formula: see text]); furthermore, we studied its computational complexity and developed an efficient implementation for [Formula: see text]. We applied our proposed [Formula: see text] method in the context of multiple attenuation, and we used computer simulations to demonstrate the advantages of our method over the traditional LSAS method, which included the capabilities to be more aggressive (use longer filters) in regions in which multiples occurred, to be less aggressive (use smaller filters) in regions dominated by primary reflections, and to be passive and take no effect (no filters used) in regions in which multiples were absent from the data.

Asymptotic theory of generalized information criterion for geostatistical regression model selection

Information criteria, such as Akaike's information criterion and Bayesian information criterion are often applied in model selection. However, their asymptotic behaviors for selecting geostatistical regression models have not been well studied, particularly under the fixed domain asymptotic framework with more and more data observed in a bounded fixed region. In this article, we study the generalized information criterion (GIC) for selecting geostatistical regression models under a more general mixed domain asymptotic framework. Via uniform convergence developments of some statistics, we establish the selection consistency and the asymptotic loss efficiency of GIC under some regularity conditions, regardless of whether the covariance model is correctly or wrongly specified. We further provide specific examples with different types of explanatory variables that satisfy the conditions. For example, in some situations, GIC is selection consistent, even when some spatial covariance parameters cannot be estimated consistently. On the other hand, GIC fails to select the true polynomial order consistently under the fixed domain asymptotic framework. Moreover, the growth rate of the domain and the degree of smoothness of candidate regressors in space are shown to play key roles for model selection.

Identification of multicollinearity and it’s effect in model selection

Multicollinearity is the problem experienced by the statisticians while at the time of evaluating a regression model. This paper explored the relationship between the sample variance inflation factor (vif) and F-ratio, based on this we proposed an exact F-test for the identification of multicollinearity and it overcomes the traditional procedures of rule of thumb. The authors critically identified that the variance inflation factor not only inflates the variance of the estimated regression coefficient and it also inflates the residual error variance of a given fitted regression model in various inflation level. Moreover, we also found a link between the problem of multicollinearity and its impact on the model selection decision. For this, the authors proposed multicollinearity corrected version of generalized information criteria which incorporates the effect of multicollinearity and help the statisticians to select a best model among the various competing models. This procedure numerically illustrated by fitting 12 different types of stepwise regression models based on 44 independent variables in a BSQ (Bank service Quality) study. Finally. Simulation study shows the transition in model selection after the correction of multicollinearity effect.

Adaptive basis expansion via $$\ell _1$$ ℓ 1 trend filtering

We propose a new approach for nonlinear regression modeling by employing basis expansion for the case where the underlying regression function has inhomogeneous smoothness. In this case, conventional nonlinear regression models tend to be over- or underfitting, where the function is more or less smoother, respectively. First, the underlying regression function is roughly approximated with a locally linear function using an $$\ell _1$$ penalized method, where this procedure is executed by extending an algorithm for the fused lasso signal approximator. We then extend the fused lasso signal approximator and develop an algorithm. Next, the residuals between the locally linear function and the data are used to adaptively prepare the basis functions. Finally, we construct a nonlinear regression model with these basis functions along with the technique of a regularization method. To select the optimal values of the tuning parameters for the regularization method, we provide an explicit form of the generalized information criterion. The validity of our proposed method is then demonstrated through several numerical examples.

The Optimal Selection for Restricted Linear Models with Average Estimator

The essential task of risk investment is to select an optimal tracking portfolio among various portfolios. Statistically, this process can be achieved by choosing an optimal restricted linear model. This paper develops a statistical procedure to do this, based on selecting appropriate weights for averaging approximately restricted models. The method of weighted average least squares is adopted to estimate the approximately restricted models under dependent error setting. The optimal weights are selected by minimizing ak-class generalized information criterion (k-GIC), which is an estimate of the average squared error from the model average fit. This model selection procedure is shown to be asymptotically optimal in the sense of obtaining the lowest possible average squared error. Monte Carlo simulations illustrate that the suggested method has comparable efficiency to some alternative model selection techniques.

Tuning Parameter Selection in High Dimensional Penalized Likelihood

SummaryDetermining how to select the tuning parameter appropriately is essential in penalized likelihood methods for high dimensional data analysis. We examine this problem in the setting of penalized likelihood methods for generalized linear models, where the dimensionality of covariates p is allowed to increase exponentially with the sample size n. We propose to select the tuning parameter by optimizing the generalized information criterion with an appropriate model complexity penalty. To ensure that we consistently identify the true model, a range for the model complexity penalty is identified in the generlized information criterion. We find that this model complexity penalty should diverge at the rate of some power of log (p) depending on the tail probability behaviour of the response variables. This reveals that using the Akaike information criterion or Bayes information criterion to select the tuning parameter may not be adequate for consistently identifying the true model. On the basis of our theoretical study, we propose a uniform choice of the model complexity penalty and show that the approach proposed consistently identifies the true model among candidate models with asymptotic probability 1. We justify the performance of the procedure proposed by numerical simulations and a gene expression data analysis.

Efficient Regularization Parameter Selection Via Information Criteria

In a nonlinear regression model based on a regularization method, selection of appropriate regularization parameters is crucial. Information criteria such as generalized information criterion (GIC) and generalized Bayesian information criterion (GBIC) are useful for selecting the optimal regularization parameters. However, the optimal parameter is often determined by calculating information criterion for all candidate regularization parameters, and so the computational cost is high. One simple method by which to accomplish this is to regard GIC or GBIC as a function of the regularization parameters and to find a value minimizing GIC or GBIC. However, it is unclear how to solve the optimization problem. In the present article, we propose an efficient Newton–Raphson type iterative method for selecting optimal regularization parameters with respect to GIC or GBIC in a nonlinear regression model based on basis expansions. This method reduces the computational time remarkably compared to the grid search and can select more suitable regularization parameters. The effectiveness of the method is illustrated through real data examples.

Selection of NARX models estimated using weighted least squares method via GIC-based method and l 1-norm regularization methods

We investigate the model selection problem for nonlinear autoregressive with exogenous variables models estimated using the weighted least squares (WLS) method. Because WLS changes the statistical property of data under study and violates the assumptions imposed on the well-developed model evaluation and selection methods (e.g. Akaike’s information criterion, Schwarz’s Bayesian information criterion, and the error reduction ratio based methods), therefore, new approaches should be investigated. In this research, an information criterion based method and two l 1-norm regularization methods are taken into consideration: (a) in the former method, for models estimated using WLS, we first derive an information criterion in terms of the generalized information criterion (GIC, proposed by Konishi and Kitagawa in Biometrica 83(4):875–890, 1996), which is a theoretical framework for the analysis and extension of information criteria via a statistical functional approach. Then we develop a robust selection procedure by combining the GIC-based forward stepwise method with Subsampling; (b) in the latter two methods, we employ the l 1-norm regularization methods, including Lasso and adaptive Lasso, to select models estimated with WLS. Finally, a numerical example is given to test and compare the performance of the three methods.

Further asymptotic properties of the generalized information criterion

Asymptotic properties of the generalized information criterion for model selection are examined and new conditions under which this criterion is overfitting, consistent, or underfitting are derived.

Iterative Bias Correction of the Cross‐Validation Criterion

Abstract. The cross‐validation (CV) criterion is known to be asecond‐order unbiased estimator of the risk function measuring the discrepancy between the candidate model and the true model, as well as the generalized information criterion (GIC) and the extended information criterion (EIC). In the present article, we show that the 2kth‐order unbiased estimator can be obtained using a linear combination from the leave‐one‐out CV criterion to the leave‐k‐out CV criterion. The proposed scheme is unique in that a bias smaller than that of a jackknife method can be obtained without any analytic calculation, that is, it is not necessary to obtain the explicit form of several terms in an asymptotic expansion of the bias. Furthermore, the proposed criterion can be regarded as a finite correction of a bias‐corrected CV criterion by using scalar coefficients in a bias‐corrected EIC obtained by the bootstrap iteration.

Tumor Grade Improves the Prognostic Ability of American Joint Committee on Cancer Stage in Patients With Penile Carcinoma

Penile cancer is rare. Thus, predicting cancer specific mortality may be difficult. We devised an accurate and yet easily applicable predictive rule that compares favorably with 2 previous models (73.8% and 74.7% accuracy, respectively). We identified patients treated with primary tumor excision for all stages of penile squamous cell carcinoma between 1998 and 2006. Disease stage definitions using Surveillance, Epidemiology and End Results stage, American Joint Committee on Cancer stage and TNM classification, and tumor grade were used to predict cancer specific mortality. Predictive accuracy estimates were compared using the DeLong method for related AUCs. Surveillance, Epidemiology and End Results stage alone (1 predictor variable) was least accurate (74.5%). American Joint Committee on Cancer stage with tumor grade (2 predictor variables) was the most simple and most accurate (80.9%, p <0.001). A benefit similar to that of American Joint Committee on Cancer stage with tumor grade was seen for TNM classification and TG (80.7%, p = 0.8). However, this rule (4 predictor variables) was more complex than American Joint Committee on Cancer stage and tumor grade. American Joint Committee on Cancer stage combined with tumor grade is the simplest, most accurate cancer specific mortality prediction rule after primary tumor excision for penile squamous cell carcinoma. This method is also more accurate than 2 previous cancer specific mortality prediction rules.

Regularization Parameter Selections via Generalized Information Criterion

We apply the nonconcave penalized likelihood approach to obtain variable selections as well as shrinkage estimators. This approach relies heavily on the choice of regularization parameter, which controls the model complexity. In this paper, we propose employing the generalized information criterion, encompassing the commonly used Akaike information criterion (AIC) and Bayesian information criterion (BIC), for selecting the regularization parameter. Our proposal makes a connection between the classical variable selection criteria and the regularization parameter selections for the nonconcave penalized likelihood approaches. We show that the BIC-type selector enables identification of the true model consistently, and the resulting estimator possesses the oracle property in the terminology of Fan and Li (2001). In contrast, however, the AIC-type selector tends to overfit with positive probability. We further show that the AIC-type selector is asymptotically loss efficient, while the BIC-type selector is not. Our simulation results confirm these theoretical findings, and an empirical example is presented. Some technical proofs are given in the online supplementary material.

Optimal tuning parameter estimation in maximum penalized likelihood method

In maximum penalized or regularized methods, it is important to select a tuning parameter appropriately. This paper proposes a direct plug-in method for tuning parameter selection. The tuning parameters selected using a generalized information criterion (Konishi and Kitagawa, Biometrika, 83, 875–890, 1996) and cross-validation (Stone, Journal of the Royal Statistical Society, Series B, 58, 267–288, 1974) are shown to be asymptotically equivalent to those selected using the proposed method, from the perspective of estimation of an optimal tuning parameter. Because of its directness, the proposed method is superior to the two selection methods mentioned above in terms of computational cost. Some numerical examples which contain the penalized spline generalized linear model regressions are provided.

Variable Selection in Logistic Discrimination Based on Local Likelihood

We consider the variable selection problem in the nonlinear discriminant procedure using local likelihood. The local likelihood method is an effective technique for analyzing data with complex structure,and various bandwidth selection methods have been suggested in recent years. Variable selection in a nonlinear model,however, is more complex than bandwidth selection,since the optimal bandwidth depends on the combination of the variables. We propose a technique for variable selection using generalized information criteria in logistic discrimination based on local likelihood. We derive the logistic discrimination method with a sample covariance matrix to account for the correlation of the variables. Real data examples are given to examine the effectiveness of our technique.

Generalized Information Criteria in Model Selection for Locally Stationary Processes

The problem of fitting a parametric model of time series with time varying parameters attracts our attention. We evaluate a goodness of time varying spectral models from an information theoretic point of view. We propose model selection criteria for locally stationary processes based on nonlinear functionals of a time varying spectral density without assuming that the true time varying spectral density belongs to the model. Also, we obtain a sufficient condition such that our information criteria coincide with Akaike’s information criterion.

A selection problem for a constrained linear regression model

Selecting a good estimate for a constricted linear regression model is investigated by using the generalized information criterion. Some asymptotic properties of the selection procedure with the model average technique are established. It is shown that the selection procedure is asymptotically efficient in the sense that a fitted estimate asymptotically obtains the minimum average squared error from a class of model average estimators.

Modeling shark bycatch: The zero-inflated negative binomial regression model with smoothing

The zero-inflated negative binomial (ZINB) regression model with smoothing is introduced for modeling count data with many zero-valued observations, and its use is illustrated with shark bycatch data from the eastern Pacific Ocean tuna purse-seine fishery for 1994–2004. Based on the generalized information criterion, the ZINB regression model provided a better fit to the data than either Poisson, negative binomial or zero-inflated Poisson regression models. To demonstrate the utility of the ZINB regression model for the standardization of catch data, standardized temporal trends in bycatch rates estimated with the ZINB regression model are computed and compared to those obtained from fits of the other three types of models to the same data. With the exception of the negative binomial, estimated temporal trends were more similar among models than would have been inferred from an analysis of model fit. Comparison of trends among models suggests that the negative binomial regression model may overestimate model coefficients when fitted to data with many zero-valued observations.

Asymptotic properties of model selection procedures in linear regression

In the regression analysis, there is typically a large collection of competing models available from which we want to select an appropriate one. This article is concerned with asymptotic properties of procedures for selecting linear models, which are based on certain data-dependent criteria such as Mallows’ C p , cross-validation and the generalized information criterion. We avoid the assumption of an adequate (‘correct’) model and allow the maximal model dimension to increase with the sample size. General asymptotic concepts are introduced, covering the usual ones of consistency and asymptotic optimality. The focus is on conditions for penalizing the model complexity, which are necessary to obtain the different optimality properties. For example, the consistency of a procedure is decided by the interplay between these penalties, the complexity of the class of model candidates, and some quantity describing the ability to identify ‘wrong’ (pseudo-inadequate) models. Many results known from the literature appear as special cases or are slightly modified.