Adaptive group lasso penalized variable selection for high-dimensional varying-coefficient panel data models with fixed effects

<p>High-dimensional data offers researchers increased ability to find useful factors in predicting a response. However, determination of the most important factors requires careful selection of the explanatory variables. In order to tackle this challenge, much work has been done on single or grouped variable selection under the penalized regression framework. Although the topic of variable selection has been extensively studied under the parametric framework, its applications to more flexible nonparametric models are yet to be explored.</p> <p>In order to implement the variable selection in nonparametric additive models, I introduce and study two nonconvex selection methods under the penalized regression framework, namely the group MCP and the adaptive group LASSO, aiming at improvements on the selection performances of the more widely known group LASSO method in such models. One major part of the dissertation focuses on the theoretical properties of the group MCP and the adaptive group LASSO. I derive their selection and estimation properties. The application of the presently proposed methods to nonparametric additive models are further examined using simulation. Their applications to areas such as the economics and genomics are presented as well. Under both the simulation studies and data applications, the group MCP and the adaptive group LASSO have shown their advantages over the more traditionally used group LASSO method.</p> <p>For the proposed adaptive group LASSO that uses the newly proposed weights, whose recursive application is therefore never studied before, I also derive its theoretical properties under a very general framework. Simulation studies under linear regression are included.</p> <p>In addition to the theoretical and empirical investigations, throughout the dissertation, several other important issues have been briefly discussed, including the computing algorithms and different ways of selecting tuning parameters.</p>

Nonparametric varying coefficient models are useful for studying the time-dependent effects of variables. Many procedures have been developed for estimation and variable selection in such models. However, existing work has focused on the case when the number of variables is fixed or smaller than the sample size. In this paper, we consider the problem of variable selection and estimation in varying coefficient models in sparse, high-dimensional settings when the number of variables can be larger than the sample size. We apply the group Lasso and basis function expansion to simultaneously select the important variables and estimate the nonzero varying coefficient functions. Under appropriate conditions, we show that the group Lasso selects a model of the right order of dimensionality, selects all variables with the norms of the corresponding coefficient functions greater than certain threshold level, and is estimation consistent. However, the group Lasso is in general not selection consistent and tends to select variables that are not important in the model. In order to improve the selection results, we apply the adaptive group Lasso. We show that, under suitable conditions, the adaptive group Lasso has the oracle selection property in the sense that it correctly selects important variables with probability converging to one. In contrast, the group Lasso does not possess such oracle property. Both approaches are evaluated using simulation and demonstrated on a data example.

We show that the adaptive Lasso (aLasso) and the adaptive group Lasso (agLasso) are oracle efficient in stationary vector autoregressions where the number of parameters per equation is smaller than the number of observations. In particular, this means that the parameters are estimated consistently at a √T-rate, that the truly zero parameters are classified as such asymptotically, and that the non-zero parameters are estimated as efficiently as if only the relevant variables had been included in the model from the outset. The group adaptive Lasso differs from the adaptive Lasso by dividing the covariates into groups whose members are all relevant or all irrelevant. Both estimators have the property that they perform variable selection and estimation in one step. We evaluate the forecasting accuracy of these estimators for a large set of macroeconomic variables. The plain Lasso is found to be the most precise procedure overall. The adaptive and the adaptive group Lasso are less stable but mostly perform at par with common factor models.

In this paper, we consider the adaptive group Lasso in high-dimensional linear regression. Some extensions have been done with other fitting procedures, such as adaptive Lasso, nonconcave penalized likelihood and adaptive elastic-net. Under appropriate conditions, we establish the consistency and asymptotic normality, which means that the adaptive group Lasso shares the oracle property in high-dimensional linear regression when the number of group variables diverges with the sample size.

A Note on Adaptive Group Lasso for Structural Break Time Series

A Note on Adaptive Group Lasso for Structural Break Time Series

Conducting model selection on data gives rise to selection uncertainty which, when ignored, invalidates subsequent classical inference which assumes that the model is given before the analysis and is in all its aspects correctly specified. In selective inference, the randomness induced by selection is dealt with by conditioning confidence intervals and p-values on the subspace of the data which leads to the same model selection as the observed data. The main challenge is the characterization of this selection event. We develop an algorithm for conducting approximate post-selection inference for parameters after model selection events which may not be characterizable as polyhedrons. We apply this on the adaptive lasso, the adaptive elastic net and the group lasso. We conduct experiments on simulated and real data, illustrating that the algorithm can both successfully control the false-positive rate and is computationally efficient.

A combination of high-dimensional sparse data and multicollinearity problems can lead to instabilities in a predictive model when applied to a new data set. The least absolute shrinkage and selection operator (Lasso) is widely employed in machine-learning algorithm for variable selection and parameter estimations. Although this method is computationally feasible for high-dimensional data, it has some drawbacks. Thus, the adaptive Lasso was developed using the adaptive weight on penalty function. This adaptive weight is related to the power order of the estimators. Hence, we focus on the power of adaptive weight on two penalty functions: adaptive Lasso and adaptive elastic net. This study aimed to compare the performances of the power of the adaptive Lasso and adaptive elastic net methods under high-dimensional sparse data with multicollinearity. Moreover, the performances of four penalized methods were compared: Lasso, elastic net, adaptive Lasso, and adaptive elastic net. They were compared using the mean of the predicted mean squared error for the simulation study and the classification accuracy for a real-data application. The results showed that the higher-order of the adaptive Lasso method performed best on very high-dimensional sparse data with multicollinearity when the initial weight was determined using a ridge estimator. However, in the case of high-dimensional sparse data with multicollinearity, the square root of the adaptive Lasso together with the initial weight using Lasso was the best option.

We consider a nonparametric additive model of a conditional mean function in which the number of variables and additive components may be larger than the sample size but the number of nonzero additive components is "small" relative to the sample size. The statistical problem is to determine which additive components are nonzero. The additive components are approximated by truncated series expansions with B-spline bases. With this approximation, the problem of component selection becomes that of selecting the groups of coefficients in the expansion. We apply the adaptive group Lasso to select nonzero components, using the group Lasso to obtain an initial estimator and reduce the dimension of the problem. We give conditions under which the group Lasso selects a model whose number of components is comparable with the underlying model, and the adaptive group Lasso selects the nonzero components correctly with probability approaching one as the sample size increases and achieves the optimal rate of convergence. The results of Monte Carlo experiments show that the adaptive group Lasso procedure works well with samples of moderate size. A data example is used to illustrate the application of the proposed method.

When a series of (related) linear models has to be estimated it is often appropriate to combine the different data-sets to construct more efficient estimators. We use ℓ1-penalized estimators like the Lasso or the Adaptive Lasso which can simultaneously do parameter estimation and model selection. We show that for a time-course of high-dimensional linear models the convergence rates of the Lasso and of the Adaptive Lasso can be improved by combining the different time-points in a suitable way. Moreover, the Adaptive Lasso still enjoys oracle properties and consistent variable selection. The finite sample properties of the proposed methods are illustrated on simulated data and on a real problem of motif finding in DNA sequences.

Penalization has been extensively adopted for variable selection in regression. In some applications, covariates have natural grouping structures, where those in the same group have correlated measurements or related functions. Under such settings, variable selection should be conducted at both the group-level and within-group-level, that is, a bi-level selection. In this study, we propose the adaptive sparse group Lasso (adSGL) method, which combines the adaptive Lasso and adaptive group Lasso (GL) to achieve bi-level selection. It can be viewed as an improved version of sparse group Lasso (SGL) and uses data-dependent weights to improve selection performance. For computation, a block coordinate descent algorithm is adopted. Simulation shows that adSGL has satisfactory performance in identifying both individual variables and groups and lower false discovery rate and mean square error than SGL and GL. We apply the proposed method to the analysis of a household healthcare expenditure data set.

Penalized regression methods are widely used for variable selection. Non-negative garrote (NNG) was one of the earliest methods to combine variable selection with shrinkage of regression coefficients, followed by lasso. About a decade after the introduction of lasso, adaptive lasso (ALASSO) was proposed to address lasso’s limitations. ALASSO has two tuning parameters (λ and γ), and its penalty resembles that of NNG when γ=1, though NNG imposes additional constraints. Given ALASSO’s greater flexibility, which may increase instability, this study investigates whether NNG provides any practical benefit or can be replaced by ALASSO. We conducted simulations in both low- and high-dimensional settings to compare selected variables, coefficient estimates, and prediction accuracy. Ordinary least squares and ridge estimates were used as initial estimates. NNG and ALASSO (γ=1) showed similar performance in low-dimensional settings with low correlation, large samples, and moderate to high R2. However, under high correlation, small samples, and low R2, their selected variables and estimates differed, though prediction accuracy remained comparable. When γ≠1, the differences between NNG and ALASSO became more pronounced, with ALASSO generally performing better. Assuming linear relationships between predictors and the outcome, the results suggest that NNG may offer no practical advantage over ALASSO. The γ parameter in ALASSO allows for adaptability to model complexity, making ALASSO a more flexible and practical alternative to NNG.

A systematic review on model selection in high-dimensional regression

In this paper we study the asymptotic properties of the adaptive Lasso estimate in high-dimensional sparse linear regression models with heteroscedastic errors. It is demonstrated that model selection properties and asymptotic normality of the selected parameters remain valid but with a suboptimal asymptotic variance. A weighted adaptive Lasso estimate is introduced and investigated. In particular, it is shown that the new estimate performs consistent model selection and that linear combinations of the estimates corresponding to the non-vanishing components are asymptotically normally distributed with a smaller variance than those obtained by the “classical” adaptive Lasso. The results are illustrated in a data example and by means of a small simulation study.

The objective of this research is to compare the parameter estimation of penalized regression and penalized logistic regression using the lasso, elastic net, adaptive lasso, and adaptive elastic net methods on high-dimensional data. The parameter estimation of the multiple linear regression model is an important problem in two related variables consisting of dependent and independent variables. Usually, the number of independent variables is less than the number of sample sizes, so the ordinary least squares give a unique solution. However, the number of independent variables is larger than a number of sample sizes, which is called the high-dimensional data. The traditional regression analysis does not estimate the solution to this problem in the case of high-dimensional data.  To overcome this problem, penalized regression analysis concerns to solve high-dimensional data. The computational part focuses on estimating the lasso, adaptive lasso, elastic net, and adaptive elastic net methods called penalized regression analysis. Lasso (least absolute shrinkage and selection operator) is added the penalty term as the scaled sum of the absolute value of the coefficients. The elastic net mixes between ridge regression and lasso on the penalty term. The lasso and elastic net methods can shrink the coefficients for variable selection. The adaptive lasso and elastic net methods use the adaptive weights on the penalty term based on the lasso and elastic net estimates. The adaptive weight is related to the power order of the estimator. Commonly, these methods focus on estimating parameters in linear regression models based on the dependent variable and independent variable as a continuous scale.  Moreover, these methods can apply the penalized regression based on logistic regression to classify high-dimensional data. The classification is used to classify the categorical data for dependent variables dependent on the independent variables, called the penalized logistic regression model. The categorical data are considered a binary variable, and the independent variables are used as the continuous variable. In this case, the independent variables are generated from the normal distribution on several variances at 20, 30, 40, and 50 when the sample sizes are less than the independent variables. For penalized regression, the comparison criterion is the average mean square error. The average percentage of accuracy is used to compare penalized logistic regression performance.