Parallel Bayesian Computation

Abstract

The use of Bayesian inference for the analysis of complex statistical models has increased dramatically in recent years, in part due to the increasing availability of computing power. There are a range of techniques available for carrying out Bayesian inference, but the lack of analytic tractability for the vast majority of models of interest means that most of the techniques are numeric, and many are computationally demanding. Indeed, for high-dimensional non-linear models, the only practical methods for analysis are based on Markov chain Monte Carlo (MCMC) techniques, and these are notoriously compute intensive, with some analyses requiring weeks of CPU time on powerful computers. It is clear therefore that the use of parallel computing technology in the context of Bayesian computation is of great interest to many who analyse complex models using Bayesian techniques. Section 18.2 considers the key elements of Bayesian inference, and the notion of graphical representation of the conditional independence structure underlying a statistical model. This turns out to be key to exploiting partitioning of computation in a parallel environment. Section 18.3 looks at the issues surrounding Monte Carlo simulation techniques in a parallel environment, laying the foundations for the examination of parallel MCMC in Section 18.4. Standard pseudo-random number generators are not suitable for use in a parallel setting, so this section examines the underlying reasons and the solution to the problem provided by parallel pseudo-random number generators. Parallel MCMC is the topic of Section 18.4. There are two essentially different strategies which can be used for parallelising an MCMC scheme (though these may be combined in a variety of ways). One is based on running multiple MCMC chains in parallel and the other is based on parallelisation of a single MCMC chain. There are different issues related to the different strategies, and each is appropriate in different situations. Indeed, since MCMC in complex models is somewhat of an art-form anyway, with a range of different possible algorithms and trade-offs even in the context of a non-parallel computing environment, the use of a parallel computer adds an additional layer of complexity to the MCMC algorithm design process. That is, the trade-offs one would adopt for the design of an efficient MCMC algorithm for the analysis of a given statistical algorithm in a non-parallel environment may