Dimensional Linear Regression Research Articles

Sequential Model Averaging for High Dimensional Linear Regression Models

In high dimensional data analysis, we propose a sequential model averaging (SMA) method to make accurate and stable predictions. Specifically, we in- troduce a hybrid approach that combines a sequential screening process with a model averaging algorithm, where the weight of each model is determined by its Bayesian information (BIC) score (Schwarz, 1978; Chen and Chen, 2008). The sequential technique makes SMA computationally feasible with high dimensional data, because the averaging process assures the prediction’s accuracy and sta- bility. Theoretical results show that SMA not only yields a good model, but also mitigates overfitting. In addition, we demonstrate that SMA provides con- sistent estimators for the regression coefficients and yields reliable predictions under mild conditions. Both simulations and empirical examples are presented to illustrate the usefulness of the proposed method.

Fused LASSO penalized least absolute deviation estimator for high dimensional linear regression

The least absolute shrinkage and selection operator (LASSO) has been playing an important role in variable selection and dimensionality reduction for high dimensional linear regression under the zero-mean or Gaussian assumptions of the noises. However, these assumptions may not hold in practice. In this case, the least absolute deviation is a popular and useful method. In this paper, we focus on the least absolute deviation via Fused LASSO, called Robust Fused LASSO, under the assumption that the unknown vector is sparsity for both the coefficients and its successive differences. Robust Fused LASSO estimator does not need any knowledge of standard deviation of the noises or any moment assumptions of the noises. We show that the Robust Fused LASSO estimator possesses near oracle performance, i.e. with large probability, the \begin{document} $\ell_2$ \end{document} norm of the estimation error is of order \begin{document} $O(\sqrt{k(\log p)/n})$ \end{document} . The result is true for a wide range of noise distributions, even for the Cauchy distribution. In addition, we apply the linearized alternating direction method of multipliers to find the Robust Fused LASSO estimator, which possesses the global convergence. Numerical results are reported to demonstrate the efficiency of our proposed method.

Simultaneous selection of predictors and responses for high dimensional multivariate linear regression

Most existing variable selection methods for multivariate linear models focus only on predictor selection. In this article, we propose a two-step (double group lasso step and sparse canonical correlation step) method to conduct variable selection for predictors and responses simultaneously.

The flare package for high dimensional linear regression and precision matrix estimation in R

This paper describes an R package named flare, which implements a family of new high dimensional regression methods (LAD Lasso, SQRT Lasso, ℓ q Lasso, and Dantzig selector) and their extensions to sparse precision matrix estimation (TIGER and CLIME). These methods exploit different nonsmooth loss functions to gain modeling exibility, estimation robustness, and tuning insensitiveness. The developed solver is based on the alternating direction method of multipliers (ADMM), which is further accelerated by the multistage screening approach. The package flare is coded in double precision C, and called from R by a user-friendly interface. The memory usage is optimized by using the sparse matrix output. The experiments show that flare is efficient and can scale up to large problems.

Sequential Model Averaging for High Dimensional Linear Regression Models

In high dimensional data analysis, we propose a sequential model averaging (SMA) method to make accurate and stable predictions. Specifically, we in- troduce a hybrid approach that combines a sequential screening process with a model averaging algorithm, where the weight of each model is determined by its Bayesian information (BIC) score (Schwarz, 1978; Chen and Chen, 2008). The sequential technique makes SMA computationally feasible with high dimensional data, because the averaging process assures the prediction’s accuracy and sta- bility. Theoretical results show that SMA not only yields a good model, but also mitigates overfitting. In addition, we demonstrate that SMA provides con- sistent estimators for the regression coefficients and yields reliable predictions under mild conditions. Both simulations and empirical examples are presented to illustrate the usefulness of the proposed method.

Recovery of weak signal in high dimensional linear regression by data perturbation

How to recover weak signals (i.e., small nonzero regression coefficients) is a difficult task in high dimensional feature selection problems. Both convex and nonconvex regularization methods fail to fully recover the true model whenever there exist strong columnwise correlations in design matrices or small nonzero coefficients below some threshold. To address the two challenges, we propose a procedure, Perturbed LASSO (PLA), that weakens correlations in the design matrix and strengthens signals by adding random perturbations to the design matrix. Moreover, a quantitative relationship between the selection accuracy and computing cost of PLA is derived. We theoretically prove and demonstrate using simulations that PLA substantially improves the chance of recovering weak signals and outperforms comparable methods at a limited cost of computation.

Minimax optimal rates of estimation in high dimensional additive models

We establish minimax optimal rates of convergence for estimation in a high dimensional additive model assuming that it is approximately sparse. Our results reveal a behavior universal to this class of high dimensional problems. In the sparse regime when the components are sufficiently smooth or the dimensionality is sufficiently large, the optimal rates are identical to those for high dimensional linear regression and, therefore, there is no additional cost to entertain a nonparametric model. Otherwise, in the so-called smooth regime, the rates coincide with the optimal rates for estimating a univariate function and, therefore, they are immune to the “curse of dimensionality.”

EigenPrism: inference for high dimensional signal-to-noise ratios.

Consider the following three important problems in statistical inference, namely, constructing confidence intervals for (1) the error of a high-dimensional (p > n) regression estimator, (2) the linear regression noise level, and (3) the genetic signal-to-noise ratio of a continuous-valued trait (related to the heritability). All three problems turn out to be closely related to the little-studied problem of performing inference on the [Formula: see text]-norm of the signal in high-dimensional linear regression. We derive a novel procedure for this, which is asymptotically correct when the covariates are multivariate Gaussian and produces valid confidence intervals in finite samples as well. The procedure, called EigenPrism, is computationally fast and makes no assumptions on coefficient sparsity or knowledge of the noise level. We investigate the width of the EigenPrism confidence intervals, including a comparison with a Bayesian setting in which our interval is just 5% wider than the Bayes credible interval. We are then able to unify the three aforementioned problems by showing that the EigenPrism procedure with only minor modifications is able to make important contributions to all three. We also investigate the robustness of coverage and find that the method applies in practice and in finite samples much more widely than just the case of multivariate Gaussian covariates. Finally, we apply EigenPrism to a genetic dataset to estimate the genetic signal-to-noise ratio for a number of continuous phenotypes.

Prediction consistency of forward iterated regression and selection technique

Recently, Hwang et al. (2009) introduced a penalized forward selection technique for high dimensional linear regression which appears to possess excellent prediction and variable selection properties. In this article, we show that the procedure is prediction consistent.

A Split-and-Merge Bayesian Variable Selection Approach for Ultrahigh Dimensional Regression

SummaryWe propose a Bayesian variable selection approach for ultrahigh dimensional linear regression based on the strategy of split and merge. The approach proposed consists of two stages: split the ultrahigh dimensional data set into a number of lower dimensional subsets and select relevant variables from each of the subsets, and aggregate the variables selected from each subset and then select relevant variables from the aggregated data set. Since the approach proposed has an embarrassingly parallel structure, it can be easily implemented in a parallel architecture and applied to big data problems with millions or more of explanatory variables. Under mild conditions, we show that the approach proposed is consistent, i.e. the true explanatory variables can be correctly identified by the approach as the sample size becomes large. Extensive comparisons of the approach proposed have been made with penalized likelihood approaches, such as the lasso, elastic net, sure independence screening and iterative sure independence screening. The numerical results show that the approach proposed generally outperforms penalized likelihood approaches: the models selected by the approach tend to be more sparse and closer to the true model.

Adaptive Global Testing for Functional Linear Models

This article studies global testing of the slope function in functional linear regression models. A major challenge in functional global testing is to choose the dimension of projection when approximating the functional regression model by a finite dimensional multivariate linear regression model. We develop a new method that simultaneously tests the slope vectors in a sequence of functional principal components regression models. The sequence of models being tested is determined by the sample size and is an integral part of the testing procedure. Our theoretical analysis shows that the proposed method is uniformly powerful over a class of smooth alternatives when the signal to noise ratio exceeds the detection boundary. The methods and results reflect the deep connection between the functional linear regression model and the Gaussian sequence model. We also present an extensive simulation study and a real data example to illustrate the finite sample performance of our method. Supplementary materials for this article are available online.

Nonnegative-lasso and application in index tracking

This paper proposes the nonnegative-lasso method for variable selection in high dimensional sparse linear regression models with the nonnegative constraints on the coefficients. This method is an extension of Lasso and is shown to have variable selection consistency and estimation consistency under certain condition similar to Irrepresentable Condition in Lasso. To get the solution of the nonnegative-lasso, many algorithms such as Lars, coordinate decent can be used, among which multiplicative updates approach is preferred since it is faster and simpler. The constrained index tracking problem in stock market without short sales is studied in the latter part. The tracking results indicate that nonnegative-lasso can get small tracking error and is successful in assets selection.

Testing Covariates in High Dimensional Regression

In a high dimensional linear regression model, we propose a new procedure for testing statistical significance of a subset of regression coefficients. Specifically, we employ the partial covariances between the response variable and the tested covariates to obtain a test statistic. The resulting test is applicable even if the predictor dimension is much larger than the sample size. Under the null hypothesis, together with boundedness and moment conditions on the predictors, we show that the proposed test statistic is asymptotically standard normal, which is further supported by Monte Carlo experiments. A similar test can be extended to generalized linear models. The practical usefulness of the test is illustrated via an empirical example on paid search advertising.

Generalized [formula omitted] test for high dimensional linear regression coefficients

To test the regression coefficients of linear models, the conventional F-test has been suggested. This paper investigates the performance of the generalized F-test for testing regression coefficients in high dimensional linear regression under the case of p/n⟶ρ(0<ρ<1). The asymptotic normality of generalized F-statistic is obtained under some regular conditions, and then the power of the F-test is derived. Some comparisons and an illustrated example are also presented.

A New Model for the Regulation Width of Waterway Based on Hydraulic Geometry Relation

Based on the results of the width of channel regulation on the field observations and indoor experiments, a new model of the width of channel regulation on hydraulic geometry relation was proposed and its coefficients were determined in theory in terms of the principle of dimensional analysis and multiple linear regression method in this paper. Considering the action of channel width, average flow velocity, average water depth and the slope of bed river, the median grain sizeD50was introduced. Through field data were verified well, and flow gradient formulas and hydraulic formulas results were compared with more consistent ones, and it was shown that this model could better reflect relatively stable riverbed. Moreover, the established expression not only was simple and explicit, but also better laid a good theoretical basis for waterway project design. Last, taking the waterway regulation project which was from Shaoguan to Qingyuan for calculation, the results showed that new calculation method was feasible and it could be a useful reference for the design of waterway regulation.

SOCP based variance free Dantzig Selector with application to robust estimation

Sparse estimation methods based on ℓ1 relaxation, such as Lasso and Dantzig Selector, are powerful tools for estimating high dimensional linear models. However, in order to properly tune these methods, the variance of the noise is often used. In this paper, we propose a new approach to the joint estimation of the sparse vector and the noise variance in a high dimensional linear regression. The method is closely related to the maximum a posteriori estimation and has the attractive feature of being computable by solving a simple second-order cone program (SOCP). We establish nonasymptotic sharp risk bounds for the proposed estimator and show how it can be applied in the problem of robust estimation.

Transductive versions of the LASSO and the Dantzig Selector

Transductive methods are useful in prediction problems when the training dataset is composed of a large number of unlabeled observations and a smaller number of labeled observations. In this paper, we propose an approach for developing transductive prediction procedures that are able to take advantage of the sparsity in the high dimensional linear regression. More precisely, we define transductive versions of the LASSO (Tibshirani, 1996) and the Dantzig Selector (Candès and Tao, 2007). These procedures combine labeled and unlabeled observations of the training dataset to produce a prediction for the unlabeled observations. We propose an experimental study of the transductive estimators that shows that they improve the LASSO and Dantzig Selector in many situations, and particularly in high dimensional problems when the predictors are correlated. We then provide non-asymptotic theoretical guarantees for these estimation methods. Interestingly, our theoretical results show that the Transductive LASSO and Dantzig Selector satisfy sparsity inequalities under weaker assumptions than those required for the “original” LASSO.

Least Median of Squares Solution of Multiple Linear Regression Models Through the Origin

Barreto and Maharry (2006) showed that PROGRESS algorithm fails to find a correct minimum “Least Median of Squares/LMS” estimate for bivariate regression models which have no intercept. Kayhan and Gunay (2008) presented a different approach for the regression models through the origin which includes at most two unknown parameters. However, LMS estimate for multiple linear regression models still remains an open issue. The aim of this study is to show that finding true LMS estimate for zero intercept multiple linear regression models can be treated as a convex optimization problem and to provide a more general algorithm for any dimensional linear regression models.

Variance Estimation Using Refitted Cross-Validation in Ultrahigh Dimensional Regression

Variance estimation is a fundamental problem in statistical modelling. In ultrahigh dimensional linear regression where the dimensionality is much larger than the sample size, traditional variance estimation techniques are not applicable. Recent advances in variable selection in ultrahigh dimensional linear regression make this problem accessible. One of the major problems in ultrahigh dimensional regression is the high spurious correlation between the unobserved realized noise and some of the predictors. As a result, the realized noises are actually predicted when extra irrelevant variables are selected, leading to serious underestimate of the level of noise. We propose a two-stage refitted procedure via a data splitting technique, called refitted cross-validation, to attenuate the influence of irrelevant variables with high spurious correlations. Our asymptotic results show that the resulting procedure performs as well as the oracle estimator, which knows in advance the mean regression function. The simulation studies lend further support to our theoretical claims. The naive two-stage estimator and the plug-in one-stage estimators using the lasso and smoothly clipped absolute deviation are also studied and compared. Their performances can be improved by the reffitted cross-validation method proposed.

Generalization of [formula omitted] constraints for high dimensional regression problems

We focus on the high dimensional linear regression Y ∼ N ( X β ∗ , σ 2 I n ) , where β ∗ ∈ R p is the parameter of interest. In this setting, several estimators such as the LASSO ( Tibshirani, 1996) and the Dantzig Selector ( Candes and Tao, 2007) are known to satisfy interesting properties whenever the vector β ∗ is sparse. Interestingly, both the LASSO and the Dantzig Selector can be seen as orthogonal projections of 0 into DC ( s ) = { β ∈ R p , ‖ X ′ ( Y − X β ) ‖ ∞ ≤ s } , using an ℓ 1 distance for the Dantzig Selector and ℓ 2 for the LASSO. For a well chosen s > 0 , this set is actually a confidence region for β ∗ . In this paper, we investigate the properties of estimators defined as projections on DC ( s ) using general distances. We prove that the obtained estimators satisfy oracle properties close to the one of the LASSO and the Dantzig Selector. On top of that, it turns out that these estimators can be tuned to exploit a different sparsity or/and slightly different estimation objectives.