Abstract
We develop methodology and theory for a mean field variational Bayes approximation to a linear model with a spike and slab prior on the regression coefficients. In particular we show how our method forces a subset of regression coefficients to be numerically indistinguishable from zero; under mild regularity conditions estimators based on our method consistently estimate the model parameters with easily obtainable and (asymptotically) appropriately sized standard error estimates; and select the true model at an exponential rate in the sample size. We also develop a practical method for simultaneously choosing reasonable initial parameter values and tuning the main tuning parameter of our algorithms which is both computationally efficient and empirically performs as well or better than some popular variable selection approaches. Our method is also faster and highly accurate when compared to MCMC.
Highlights
Variable selection is one of the key problems in statistics as evidenced by papers too numerous to mention all but a small subset
Our contributions are: (i) We show how our variational Bayes (VB) method induces sparsity upon the regression coefficients; (ii) We show, under mild assumptions, that our estimators for the model parameters are consistent with obtainable and appropriately sized standard error estimates; (iii) Under these same assumptions our VB method selects the true model at an exponential rate in n; and (iv) We develop a practical method for simultaneously choosing reasonable initial parameter values and tuning the main tuning parameter of our algorithms
In this paper we have provided theory for a new approach which induces sparsity on the estimates of the regression coefficients for a Bayesian linear model
Summary
Variable selection is one of the key problems in statistics as evidenced by papers too numerous to mention all but a small subset. In this paper we consider a spike and slab prior on the regression coefficients (see Mitchell and Beauchamp, 1988; George and McCulloch, 1993) in order to encourage sparse estimators This entails using VB to approximate the posterior distribution of indicator variables to select which variables are to be included in the model. It is not difficult to extend the methodology developed here to handle elaborate responses (Wand et al, 2011), missing data (Faes et al, 2011) or measurement error (Pham et al, 2013) This contrasts with criteria based procedures, penalized regression and some Bayesian procedures (for example Liang et al, 2008; Maruyama and George, 2011, where the models are chosen carefully so that an exact expression for marginal likelihood is obtainable).
Sign-in/Register to view highlights & summary Sign-in/Register to access full text options 
Talk to us
Join us for a 30 min session where you can share your feedback and ask us any queries you have
Schedule a callDisclaimer: All third-party content on this website/platform is and will remain the property of their respective owners and is provided on "as is" basis without any warranties, express or implied. Use of third-party content does not indicate any affiliation, sponsorship with or endorsement by them. Any references to third-party content is to identify the corresponding services and shall be considered fair use under The CopyrightLaw.
