Sparse High-dimensional Linear Regression Research Articles

Sparse high-dimensional linear regression. Estimating squared error and a phase transition

We consider a sparse high-dimensional regression model where the goal is to recover a k-sparse unknown binary vector β∗ from n noisy linear observations of the form Y=Xβ∗+W∈Rn where X∈Rn×p has i.i.d. N(0,1) entries and W∈Rn has i.i.d. N(0,σ2) entries. In the high signal-to-noise ratio regime and sublinear sparsity regime, while the order of the sample size needed to recover the unknown vector information-theoretically is known to be n∗:=2klogp/log(k/σ2+1), no polynomial-time algorithm is known to succeed unless n>nalg:=(2k+σ2)logp. In this work, we offer a series of results investigating multiple computational and statistical aspects of the recovery task in the regime n∈[n∗,nalg]. First, we establish a novel information-theoretic property of the MLE of the problem happening around n=n∗ samples, which we coin as an “all-or-nothing behavior”: when n>n∗ it recovers almost perfectly the support of β∗, while if n<n∗ it fails to recover any fraction of it correctly. Second, at an attempt to understand the computational hardness in the regime n∈[n∗,nalg], we prove that at order nalg samples there is an Overlap Gap Property (OGP) phase transition occurring at the landscape of the MLE: for constants c,C>0 when n<cnalg OGP appears in the landscape of MLE while if n>Cnalg OGP disappears. OGP is a geometric “disconnectivity” property, which initially appeared in the theory of spin glasses and is known to suggest algorithmic hardness when it occurs. Finally, using certain technical results obtained to establish the OGP phase transition, we additionally establish various novel positive and negative algorithmic results for the recovery task of interest, including the failure of LASSO with access to n<cnalg samples and the success of a simple local search method with access to n>Cnalg samples.

An adaptive shortest-solution guided decimation approach to sparse high-dimensional linear regression

High-dimensional linear regression model is the most popular statistical model for high-dimensional data, but it is quite a challenging task to achieve a sparse set of regression coefficients. In this paper, we propose a simple heuristic algorithm to construct sparse high-dimensional linear regression models, which is adapted from the shortest-solution guided decimation algorithm and is referred to as ASSD. This algorithm constructs the support of regression coefficients under the guidance of the shortest least-squares solution of the recursively decimated linear models, and it applies an early-stopping criterion and a second-stage thresholding procedure to refine this support. Our extensive numerical results demonstrate that ASSD outperforms LASSO, adaptive LASSO, vector approximate message passing, and two other representative greedy algorithms in solution accuracy and robustness. ASSD is especially suitable for linear regression problems with highly correlated measurement matrices encountered in real-world applications.

Honest Confidence Sets for High-Dimensional Regression by Projection and Shrinkage

The issue of honesty in constructing confidence sets arises in nonparametric regression. While optimal rate in nonparametric estimation can be achieved and utilized to construct sharp confidence sets, severe degradation of confidence level often happens after estimating the degree of smoothness. Similarly, for high-dimensional regression, oracle inequalities for sparse estimators could be utilized to construct sharp confidence sets. Yet, the degree of sparsity itself is unknown and needs to be estimated, which causes the honesty problem. To resolve this issue, we develop a novel method to construct honest confidence sets for sparse high-dimensional linear regression. The key idea in our construction is to separate signals into a strong and a weak group, and then construct confidence sets for each group separately. This is achieved by a projection and shrinkage approach, the latter implemented via Stein estimation and the associated Stein unbiased risk estimate. Our confidence set is honest over the full parameter space without any sparsity constraints, while its size adapts to the optimal rate of when the true parameter is indeed sparse. Moreover, under some form of a separation assumption between the strong and weak signals, the diameter of our confidence set can achieve a faster rate than existing methods. Through extensive numerical comparisons on both simulated and real data, we demonstrate that our method outperforms other competitors with big margins for finite samples, including oracle methods built upon the true sparsity of the underlying model.

Strong selection consistency of Bayesian vector autoregressive models based on a pseudo-likelihood approach

Vector autoregressive (VAR) models aim to capture linear temporal interdependencies among multiple time series. They have been widely used in macroeconomics and financial econometrics and more recently have found novel applications in functional genomics and neuroscience. These applications have also accentuated the need to investigate the behavior of the VAR model in a high-dimensional regime, which will provide novel insights into the role of temporal dependence for regularized estimates of the models parameters. However, hardly anything is known regarding posterior model selection consistency for Bayesian VAR models in such regimes. In this work, we develop a pseudo-likelihood based Bayesian approach for consistent variable selection in high-dimensional VAR models by considering hierarchical normal priors on the autoregressive coefficients, as well as on the model space. We establish strong selection consistency of the proposed method, namely that the posterior probability assigned to the true underlying VAR model converges to one under high-dimensional scaling where the dimension p of the VAR system grows nearly exponentially with the sample size n. Further, the result is established under mild regularity conditions on the problem parameters. Finally, as a by-product of these results, we also establish strong selection consistency for the sparse high-dimensional linear regression model with serially correlated regressors and errors.

Variational Bayes for High-Dimensional Linear Regression With Sparse Priors

We study a mean-field spike and slab variational Bayes (VB) approximation to Bayesian model selection priors in sparse high-dimensional linear regression. Under compatibility conditions on the design matrix, oracle inequalities are derived for the mean-field VB approximation, implying that it converges to the sparse truth at the optimal rate and gives optimal prediction of the response vector. The empirical performance of our algorithm is studied, showing that it works comparably well as other state-of-the-art Bayesian variable selection methods. We also numerically demonstrate that the widely used coordinate-ascent variational inference algorithm can be highly sensitive to the parameter updating order, leading to potentially poor performance. To mitigate this, we propose a novel prioritized updating scheme that uses a data-driven updating order and performs better in simulations. The variational algorithm is implemented in the R package sparsevb. Supplementary materials for this article are available online.

Sparse regression for extreme values

We study the problem of selecting features associated with extreme values in high dimensional linear regression. Normally, in linear modeling problems, the presence of abnormal extreme values or outliers is considered an anomaly which should either be removed from the data or remedied using robust regression methods. In many situations, however, the extreme values in regression modeling are not outliers but rather the signals of interest; consider traces from spiking neurons, volatility in finance, or extreme events in climate science, for example. In this paper, we propose a new method for sparse high-dimensional linear regression for extreme values which is motivated by the Subbotin, or generalized normal distribution, which we call the extreme value linear regression model. For our method, we utilize an ℓp norm loss where p is an even integer greater than two; we demonstrate that this loss increases the weight on extreme values. We prove consistency and variable selection consistency for the extreme value linear regression with a Lasso penalty, which we term the Extreme Lasso, and we also analyze the theoretical impact of extreme value observations on the model parameter estimates using the concept of influence functions. Through simulation studies and a real-world data example, we show that the Extreme Lasso outperforms other methods currently used in the literature for selecting features of interest associated with extreme values in high-dimensional regression.

Nonnegative estimation and variable selection via adaptive elastic-net for high-dimensional data

This paper proposes the nonnegative adaptive elastic-net for simultaneous nonnegative estimation and variable selection in sparse high-dimensional linear regression models. By inheriting the good features of adaptive elastic-net, the nonnegative adaptive elastic-net enjoys the oracle property even in high-dimensional settings where the dimension of covariates can be much larger than the sample size. Through the simulation, we show that the newly proposed method deals with the collinearity problem better than alternative procedures in the literature. To make the proposed method practically feasible, we extend the multiplicative updates algorithm for implementation. Finally, we illustrate the favorable finite-sample performance of the proposed method through tracking the CSI 300 index, an important stock market index in China.

Rate optimal estimation and confidence intervals for high-dimensional regression with missing covariates

We consider the problems of estimation and of constructing component-wise confidence intervals in a sparse high-dimensional linear regression model when some covariates of the design matrix are missing completely at random. We analyze a variant of the Dantzig selector for estimating the regression model and we use a de-biasing argument to construct component-wise confidence intervals. We also complement our mathematical study in the supplementary materials with extensive simulations on synthetic and semi-synthetic data that show the accuracy of our asymptotic predictions for finite sample sizes.

Nonnegative estimation and variable selection under minimax concave penalty for sparse high-dimensional linear regression models

As a promising alternative to the Lasso penalty, the minimax concave penalty (MCP) produces nearly unbiased shrinkage estimate. In this paper, we propose the nonnegative MCP estimator in the specific high-dimensional settings where the regression coefficients are imposed with the nonnegativity constraint, which is particularly relevant when modelling nonnegative data. We prove that the asymptotic theory for the nonnegative MCP estimator requires much weaker assumptions than those in the literature. In particular, we do not impose the restrictive distributional conditions on the random errors and do not assume the irrepresentable condition or its variants on the design matrix. For the nonnegative MCP estimator, despite its good asymptotic properties, the corresponding optimization problem is non-convex, and consequently much hard to solve. Thus, we develop an efficient computation algorithm, which is a coupling of the difference convex algorithm with the multiplicative updates algorithm, for the implementation of the nonnegative MCP estimator. Simulation studies are carried out to examine superior performance of the nonnegative MCP estimator against alternative methods and a real data analysis for index tracking is also presented for illustration.

Posterior asymptotic normality for an individual coordinate in high-dimensional linear regression

We consider the sparse high-dimensional linear regression model $Y=Xb+\epsilon$ where $b$ is a sparse vector. For the Bayesian approach to this problem, many authors have considered the behavior of the posterior distribution when, in truth, $Y=X\beta+\epsilon$ for some given $\beta$. There have been numerous results about the rate at which the posterior distribution concentrates around $\beta$, but few results about the shape of that posterior distribution. We propose a prior distribution for $b$ such that the marginal posterior distribution of an individual coordinate $b_i$ is asymptotically normal centered around an asymptotically efficient estimator, under the truth. Such a result gives Bayesian credible intervals that match with the confidence intervals obtained from an asymptotically efficient estimator for $b_i$. We also discuss ways of obtaining such asymptotically efficient estimators on individual coordinates. We compare the two-step procedure proposed by Zhang and Zhang (2014) and a one-step modified penalization method.

Improved bounds for Square-Root Lasso and Square-Root Slope

Extending the results of Bellec, Lecue and Tsybakov [1] to the setting of sparse high-dimensional linear regression with unknown variance, we show that two estimators, the Square-Root Lasso and the Square-Root Slope can achieve the optimal minimax prediction rate, which is $(s/n)\log\left (p/s\right )$, up to some constant, under some mild conditions on the design matrix. Here, $n$ is the sample size, $p$ is the dimension and $s$ is the sparsity parameter. We also prove optimality for the estimation error in the $l_{q}$-norm, with $q\in[1,2]$ for the Square-Root Lasso, and in the $l_{2}$ and sorted $l_{1}$ norms for the Square-Root Slope. Both estimators are adaptive to the unknown variance of the noise. The Square-Root Slope is also adaptive to the sparsity $s$ of the true parameter. Next, we prove that any estimator depending on $s$ which attains the minimax rate admits an adaptive to $s$ version still attaining the same rate. We apply this result to the Square-root Lasso. Moreover, for both estimators, we obtain valid rates for a wide range of confidence levels, and improved concentration properties as in [1] where the case of known variance is treated. Our results are non-asymptotic.

Multiple outliers detection in sparse high-dimensional regression

ABSTRACTThe presence of outliers would inevitably lead to distorted analysis and inappropriate prediction, especially for multiple outliers in high-dimensional regression, where the high dimensionality of the data might amplify the chance of an observation or multiple observations being outlying. Noting that the detection of outliers is not only necessary but also important in high-dimensional regression analysis, we, in this paper, propose a feasible outlier detection approach in sparse high-dimensional linear regression model. Firstly, we search a clean subset by use of the sure independence screening method and the least trimmed square regression estimates. Then, we define a high-dimensional outlier detection measure and propose a multiple outliers detection approach through multiple testing procedures. In addition, to enhance efficiency, we refine the outlier detection rule after obtaining a relatively reliable non-outlier subset based on the initial detection approach. By comparison studies based on Monte Carlo simulation, it is shown that the proposed method performs well for detecting multiple outliers in sparse high-dimensional linear regression model. We further illustrate the application of the proposed method by empirical analysis of a real-life protein and gene expression data.

Asymptotic properties of adaptive Dantzig selector

In this paper, we study the asymptotic properties of the adaptive Dantzig selector estimators in sparse high-dimensional linear regression models when the number of covariates $p$ may be large or even larger than the sample size $n$ at an exponential rate of the sample size $n$. When initial estimators, as the weights of the adaptive Dantzig selector, are consistent to constants, under regularity conditions, we show that the adaptive Dantzig selector has the Oracle property. Available initial estimators are also provided for both $p\leq n$ and $p>n$. Simulations are carried out to witness our theoretical conclusions.

A note on standardization in penalized regressions

AbstractWe consider sparse high-dimensional linear regression models. Penalized regressionshave been used as e ective methods for variable selection and estimation in high-dimensional models. In penalized regressions, it is common practice to standardizevariables before tting a penalized model and then t a penalized model with stan-dardized variables. Finally, the estimated coecients from a penalized model are re-covered to the scale on original variables. However, these procedures produce a slightlydi erent solution compared to the corresponding original penalized problem. In thispaper, we investigate issues on the standardization of variables in penalized regressionsand formulate the de nition of the standardized penalized estimator. In addition, wecompare the original penalized estimator with the standardized penalized estimatorthrough simulation studies and real data analysis.Keywords: LASSO, nonconvex penalties, penalized regression, standardization. 1. Introduction Consider the linear regression modely = X + ; (1.1)where y = (y

An Additive Sparse Penalty for Variable Selection in High-Dimensional Linear Regression Model

We consider a sparse high-dimensional linear regression model. Penalized methods using LASSO or nonconvex penalties have been widely used for variable selection and estimation in high-dimensional regression models. In penalized regression, the selection and prediction performances depend on which penalty function is used. For example, it is known that LASSO has a good prediction performance but tends to select more variables than necessary. In this paper, we propose an additive sparse penalty for variable selection using a combination of LASSO and minimax concave penalties (MCP). The proposed penalty is designed for good properties of both LASSO and MCP. We develop an efficient algorithm to compute the proposed estimator by combining a concave convex procedure and coordinate descent algorithm. Numerical studies show that the proposed method has better selection and prediction performances compared to other penalized methods.

A small review and further studies on the LASSO

Abstract High-dimensional data analysis arises from almost all scienti c areas, evolving withdevelopment of computing skills, and has encouraged penalized estimations that playimportant roles in statistical learning. For the past years, various penalized estimationshave been developed, and the least absolute shrinkage and selection operator (LASSO)proposed by Tibshirani (1996) has shown outstanding ability, earning the rst place onthe development of penalized estimation. In this paper, we rst introduce a number ofrecent advances in high-dimensional data analysis using the LASSO. The topics includevarious statistical problems such as variable selection and grouped or structured variableselection under sparse high-dimensional linear regression models. Several unsupervisedlearning methods including inverse covariance matrix estimation are presented. In ad-dition, we address further studies on new applications which may establish a guidelineon how to use the LASSO for statistical challenges of high-dimensional data analysis.Keywords: High dimension, LASSO, penalized estimation, review.

Asymptotic properties of Lasso+mLS and Lasso+Ridge in sparse high-dimensional linear regression

We study the asymptotic properties of Lasso+mLS and Lasso+ Ridge under the sparse high-dimensional linear regression model: Lasso selecting predictors and then modified Least Squares (mLS) or Ridge estimating their coefficients. First, we propose a valid inference procedure for parameter estimation based on parametric residual bootstrap after Lasso+ mLS and Lasso+Ridge. Second, we derive the asymptotic unbiasedness of Lasso+mLS and Lasso+Ridge. More specifically, we show that their biases decay at an exponential rate and they can achieve the oracle convergence rate of $s/n$ (where $s$ is the number of nonzero regression coefficients and $n$ is the sample size) for mean squared error (MSE). Third, we show that Lasso+mLS and Lasso+Ridge are asymptotically normal. They have an oracle property in the sense that they can select the true predictors with probability converging to $1$ and the estimates of nonzero parameters have the same asymptotic normal distribution that they would have if the zero parameters were known in advance. In fact, our analysis is not limited to adopting Lasso in the selection stage, but is applicable to any other model selection criteria with exponentially decay rates of the probability of selecting wrong models.