Abstract
Penalized regression methods such as the lasso and elastic net (EN) have become popular for simultaneous variable selection and coefficient estimation. Implementation of these methods require selection of the penalty parameters. We propose an empirical Bayes (EB) methodology for selecting these tuning parameters as well as computation of the regularization path plots. The EB method does not suffer from the “double shrinkage problem” of frequentist EN. Also it avoids the difficulty of constructing an appropriate prior on the penalty parameters. The EB methodology is implemented by efficient importance sampling method based on multiple Gibbs sampler chains. Since the Markov chains underlying the Gibbs sampler are proved to be geometrically ergodic, Markov chain central limit theorem can be used to provide asymptotically valid confidence band for profiles of EN coefficients. The practical effectiveness of our method is illustrated by several simulation examples and two real life case studies. Although this article considers lasso and EN for brevity, the proposed EB method is general and can be used to select shrinkage parameters in other regularization methods.
Highlights
Consider the standard linear model y = μ1n + Xβ +, where y ∈ Rn is the vector of responses, μ ∈ R is the overall mean, 1n is the n × 1 vector of 1’s, X = (X1, X2, . . . , Xp) is the n × p covariate matrix, β ∈ Rp is the unknown vector of regression coefficients, and is the n × 1 vector of iid normal errors with mean zero and unknown variance parameter σ2
It has been recently shown by Khare and Hobert (2013) that when λ in (1) is assumed fixed, the Gibbs sampler Markov chain for Bayesian lasso of Park and Casella (2008) has a geometric rate of convergence, while no such convergence results is currently known about the Markov chain Monte Carlo (MCMC) algorithm for the full Bayesian lasso model. (See Section 2 for the definition of geometric convergence.) In section 2, we prove that, the Bayesian elastic net Gibbs sampler that do not update λ1 and λ2 is geometrically ergodic
In this paper we develop an empirical Bayes (EB) approach with efficient generalized importance sampling methods based on multiple Markov chains for estimating the shrinkage parameters in penalized regression methods
Summary
In this paper we develop an EB approach with efficient generalized importance sampling methods based on multiple Markov chains for estimating the shrinkage parameters in penalized regression methods. In penalized regression methods such as lasso and EN, a plot of the profiles of the estimated regression coefficients as a function of the penalty parameter is used to display the amount of shrinkage corresponding to different tuning parameter values. As mentioned above, Khare and Hobert (2013) have recently shown that the Bayesian lasso Gibbs sampler is geometrically ergodic—which allows for calculation of asymptotically valid standard errors of lasso estimates (see Section 2 for details). An efficient importance sampling method for selecting the shrinkage parameters Proofs of results are relegated to the Web based supplementary materials (Roy and Chakraborty, 2016)
Sign-in/Register to view highlights & summary Sign-in/Register to access full text options 
Talk to us
Join us for a 30 min session where you can share your feedback and ask us any queries you have
Schedule a callDisclaimer: All third-party content on this website/platform is and will remain the property of their respective owners and is provided on "as is" basis without any warranties, express or implied. Use of third-party content does not indicate any affiliation, sponsorship with or endorsement by them. Any references to third-party content is to identify the corresponding services and shall be considered fair use under The CopyrightLaw.
