Abstract
We study the asymptotic properties of Lasso+mLS and Lasso+ Ridge under the sparse high-dimensional linear regression model: Lasso selecting predictors and then modified Least Squares (mLS) or Ridge estimating their coefficients. First, we propose a valid inference procedure for parameter estimation based on parametric residual bootstrap after Lasso+ mLS and Lasso+Ridge. Second, we derive the asymptotic unbiasedness of Lasso+mLS and Lasso+Ridge. More specifically, we show that their biases decay at an exponential rate and they can achieve the oracle convergence rate of $s/n$ (where $s$ is the number of nonzero regression coefficients and $n$ is the sample size) for mean squared error (MSE). Third, we show that Lasso+mLS and Lasso+Ridge are asymptotically normal. They have an oracle property in the sense that they can select the true predictors with probability converging to $1$ and the estimates of nonzero parameters have the same asymptotic normal distribution that they would have if the zero parameters were known in advance. In fact, our analysis is not limited to adopting Lasso in the selection stage, but is applicable to any other model selection criteria with exponentially decay rates of the probability of selecting wrong models.
Highlights
Consider the sparse linear regression model Y = Xβ∗ + ǫ, (1)where ǫ = (ǫ1, . . . , ǫn)T is a vector of independent and identically distributed (i.i.d.) random variables with mean 0 and variance σ2
As we show in Theorem 3 and Corollary 2, these two post-Lasso estimators display an oracle property that the Lasso does not have: they can select the true predictors with probability converging to 1 and the estimates of nonzero parameters have the same asymptotic normal distribution that they would have if the zero parameters were known in advance
We have derived for the first time the asymptotic properties of Lasso+modified Least Squares (mLS) and Lasso+Ridge in sparse high-dimensional linear regression models where p ≫ n
Summary
Thresholded Lasso and Dantzig estimators were introduced in [31] and the authors proved their model selection consistency under less restrictive conditions on the decay rates of the nonzero regression coefficients. We should mention that previous work [3] has obtained l2 convergence rate (||βLasso+OLS − β∗||22 = Op(s/n)) of Lasso+OLS estimator under weaker conditions Their results hold in probability and it is not clear whether Lasso+OLS can achieve the oracle convergence rate of O(s/n) in L2-expectation, i.e., whether E||β− β∗||22 = O(s/n) holds, which we need to prove the validity of residual bootstrap. We begin with a precise definition of the modified Least Squares or Ridge after model selection, and study their asymptotic properties, including asymptotic unbiasedness, asymptotic normality and the validity of residual bootstrap
Talk to us
Join us for a 30 min session where you can share your feedback and ask us any queries you have