Abstract
We consider exact algorithms for Bayesian inference with model selection priors (including spike-and-slab priors) in the sparse normal sequence model. Because the best existing exact algorithm becomes numerically unstable for sample sizes over n=500, there has been much attention for alternative approaches like approximate algorithms (Gibbs sampling, variational Bayes, etc.), shrinkage priors (e.g. the Horseshoe prior and the Spike-and-Slab LASSO) or empirical Bayesian methods. However, by introducing algorithmic ideas from online sequential prediction, we show that exact calculations are feasible for much larger sample sizes: for general model selection priors we reach n=25000, and for certain spike-and-slab priors we can easily reach n=100000. We further prove a de Finetti-like result for finite sample sizes that characterizes exactly which model selection priors can be expressed as spike-and-slab priors. The computational speed and numerical accuracy of the proposed methods are demonstrated in experiments on simulated data, on a differential gene expression data set, and to compare the effect of multiple hyper-parameter settings in the beta-binomial prior. In our experimental evaluation we compute guaranteed bounds on the numerical accuracy of all new algorithms, which shows that the proposed methods are numerically reliable whereas an alternative based on long division is not.
Highlights
In the sparse normal sequence model we observe a sequence Y = (Y1, . . . , Yn) that satisfiesYi = θi + εi, i = 1, . . . , n, (1)for independent standard normal random variables εi, where θ = (θ1, . . . , θn) is the unknown signal of interest
For general model selection priors we propose a model selection Hidden Markov Model (HMM) algorithm, and for spike-and-slab priors we introduce a faster method based on discretization of the α hyper-parameter
We observe that the model selection HMM and the algorithm based on discretization performed superior to the preceding methods: the model selection HMM algorithm has run time O(n2) and the largest sample size it managed to complete within half an hour was n = 25 000, while the algorithm with discretized mixing parameter in the spike-and-slab prior has run time O(n3/2) and reached the time limit after sample size n = 100 000
Summary
The sparse normal sequence model, which is called the sparse normal means model, has been extensively studied from a frequentist perspective (see, for instance, [27, 7, 1]), but here we consider Bayesian approaches, which endow θ with a prior distribution This prior serves as a natural way to introduce sparsity into the model and the corresponding posterior can be used for model comparison and uncertainty quantification (see [24, 53, 37, 5] and references therein). One natural and well-understood class of priors are model selection priors that take the following hierarchical form: i.) First a sparsity level s is chosen from a prior πn on {0, 1, . . . , n}
Sign-in/Register to view highlights & summary Talk to us
Join us for a 30 min session where you can share your feedback and ask us any queries you have
Schedule a callDisclaimer: All third-party content on this website/platform is and will remain the property of their respective owners and is provided on "as is" basis without any warranties, express or implied. Use of third-party content does not indicate any affiliation, sponsorship with or endorsement by them. Any references to third-party content is to identify the corresponding services and shall be considered fair use under The CopyrightLaw.
