Parallel Markov Chain Monte Carlo Research Articles

Bayesian cross-validation by parallel Markov chain Monte Carlo

Brute force cross-validation (CV) is a method for predictive assessment and model selection that is general and applicable to a wide range of Bayesian models. Naive or ‘brute force’ CV approaches are often too computationally costly for interactive modeling workflows, especially when inference relies on Markov chain Monte Carlo (MCMC). We propose overcoming this limitation using massively parallel MCMC. Using accelerator hardware such as graphics processor units, our approach can be about as fast (in wall clock time) as a single full-data model fit. Parallel CV is flexible because it can easily exploit a wide range data partitioning schemes, such as those designed for non-exchangeable data. It can also accommodate a range of scoring rules. We propose MCMC diagnostics, including a summary of MCMC mixing based on the popular potential scale reduction factor (R^\\documentclass[12pt]{minimal} \\usepackage{amsmath} \\usepackage{wasysym} \\usepackage{amsfonts} \\usepackage{amssymb} \\usepackage{amsbsy} \\usepackage{mathrsfs} \\usepackage{upgreek} \\setlength{\\oddsidemargin}{-69pt} \\begin{document}$$\\widehat{\ extrm{R}}$$\\end{document}) and MCMC effective sample size (ESS^\\documentclass[12pt]{minimal} \\usepackage{amsmath} \\usepackage{wasysym} \\usepackage{amsfonts} \\usepackage{amssymb} \\usepackage{amsbsy} \\usepackage{mathrsfs} \\usepackage{upgreek} \\setlength{\\oddsidemargin}{-69pt} \\begin{document}$$\\widehat{\ extrm{ESS}}$$\\end{document}) measures. We also describe a method for determining whether an R^\\documentclass[12pt]{minimal} \\usepackage{amsmath} \\usepackage{wasysym} \\usepackage{amsfonts} \\usepackage{amssymb} \\usepackage{amsbsy} \\usepackage{mathrsfs} \\usepackage{upgreek} \\setlength{\\oddsidemargin}{-69pt} \\begin{document}$$\\widehat{\ extrm{R}}$$\\end{document} diagnostic indicates approximate stationarity of the chains, that may be of more general interest for applications beyond parallel CV. Finally, we show that parallel CV and its diagnostics can be implemented with online algorithms, allowing parallel CV to scale up to very large blocking designs on memory-constrained computing accelerators.

Automated model calibration with parallel MCMC: Applications for a cardiovascular system model.

Computational physiological models continue to increase in complexity, however, the task of efficiently calibrating the model to available clinical data remains a significant challenge. One part of this challenge is associated with long calibration times, which present a barrier for the routine application of model-based prediction in clinical practice. Another aspect of this challenge is the limited available data for the unique calibration of complex models. Therefore, to calibrate a patient-specific model, it may be beneficial to verify that task-specific model predictions have acceptable uncertainty, rather than requiring all parameters to be uniquely identified. We have developed a pipeline that reduces the set of fitting parameters to make them structurally identifiable and to improve the efficiency of a subsequent Markov Chain Monte Carlo (MCMC) analysis. MCMC was used to find the optimal parameter values and to determine the confidence interval of a task-specific prediction. This approach was demonstrated on numerical experiments where a lumped parameter model of the cardiovascular system was calibrated to brachial artery cuff pressure, echocardiogram volume measurements, and synthetic cerebral blood flow data that approximates what can be obtained from 4D-flow MRI data. This pipeline provides a cerebral arterial pressure prediction that may be useful for determining the risk of hemorrhagic stroke. For a set of three patients, this pipeline successfully reduced the parameter set of a cardiovascular system model from 12 parameters to 8-10 structurally identifiable parameters. This enabled a significant efficiency improvement in determining confidence intervals on predictions of pressure compared to performing a naive MCMC analysis with the full parameter set. This demonstrates the potential that the proposed pipeline has in helping address one of the key challenges preventing clinical application of such models. Additionally, for each patient, the MCMC approach yielded a 95% confidence interval on systolic blood pressure prediction in the middle cerebral artery smaller than ±10mmHg (±1.3kPa). The proposed pipeline exploits available high-performance computing parallelism to allow straightforward automation for general models and arbitrary data sets, enabling automated calibration of a parameter set that is specific to the available clinical data with minimal user interaction.

Reducing Geometric Uncertainty in Computational Hemodynamics by Deep Learning-Assisted Parallel-Chain MCMC.

Computational hemodynamic modeling has been widely used in cardiovascular research and healthcare. However, the reliability of model predictions is largely dependent on the uncertainties of modeling parameters and boundary conditions, which should be carefully quantified and further reduced with available measurements. In this work, we focus on propagating and reducing the uncertainty of vascular geometries within a Bayesian framework. A novel deep learning (DL)-assisted parallel Markov chain Monte Carlo (MCMC) method is presented to enable efficient Bayesian posterior sampling and geometric uncertainty reduction. A DL model is built to approximate the geometry-to-hemodynamic map, which is trained actively using online data collected from parallel MCMC chains and utilized for early rejection of unlikely proposals to facilitate convergence with less expensive full-order model evaluations. Numerical studies on two-dimensional aortic flows are conducted to demonstrate the effectiveness and merit of the proposed method.

Bayesian neural networks for stock price forecasting before and during COVID-19 pandemic.

Recently, there has been much attention in the use of machine learning methods, particularly deep learning for stock price prediction. A major limitation of conventional deep learning is uncertainty quantification in predictions which affect investor confidence. Bayesian neural networks feature Bayesian inference for providing inference (training) of model parameters that provides a rigorous methodology for uncertainty quantification in predictions. Markov Chain Monte Carlo (MCMC) sampling methods have been prominent in implementing inference of Bayesian neural networks; however certain limitations existed due to a large number of parameters and the need for better computational resources. Recently, there has been much progress in the area of Bayesian neural networks given the use of Langevin gradients with parallel tempering MCMC that can be implemented in a parallel computing environment. The COVID-19 pandemic had a drastic impact in the world economy and stock markets given different levels of lockdowns due to rise and fall of daily infections. It is important to investigate the performance of related forecasting models during the COVID-19 pandemic given the volatility in stock markets. In this paper, we use novel Bayesian neural networks for multi-step-ahead stock price forecasting before and during COVID-19. We also investigate if the pre-COVID-19 datasets are useful of modelling stock price forecasting during COVID-19. Our results indicate due to high volatility in the stock-price during COVID-19, it is more challenging to provide forecasting. However, we found that Bayesian neural networks could provide reasonable predictions with uncertainty quantification despite high market volatility during the first peak of the COVID-19 pandemic.

PMBA: A Parallel MCMC Bayesian Computing Accelerator

Bayesian computing, including sampling probability distributions, learning graphic model, and Bayesian reasoning, is a powerful class of machine learning algorithms with such wide applications as biologic computing, financial analysis, natural language processing, autonomous driving, and robotics. The central pattern of Bayesian computing is the Markov Chain Monte Carlo (MCMC) computing, which is compute-intensive and lacks explicit parallelism. In this work, we propose a parallel MCMC Bayesian computing accelerator (PMBA) architecture. Designed as a probabilistic computing platform with native support for efficient single-chain parallel Metropolis-Hastings based MCMC sampling, PMBA boosts the performance of probabilistic programs with a massive-parallelism microarchitecture. PMBA is equipped with on-chip random number generators as the built-in source of randomness. The sampling units of PMBA are designed for parallel random sampling through a customized SIMD pipeline supporting data synchronization every iteration. A respective computing framework supporting automatic parallelization and mapping of probabilistic programs is also developed. Evaluation results demonstrate that PMBA enables a 17-21 folds speedup over a TITAN X GPU on MCMC sampling workload. On probabilistic benchmarks, PMBA outperforms prior best solutions by factor of 3.6 to 10.3. An exemplar based visual category learning algorithm is implemented on PMBA to demonstrate its efficiency and effectiveness for complex statistical learning problems.

Surrogate-assisted parallel tempering for Bayesian neural learning

Due to the need for robust uncertainty quantification, Bayesian neural learning has gained attention in the era of deep learning and big data. Markov Chain Monte-Carlo (MCMC) methods typically implement Bayesian inference which faces several challenges given a large number of parameters, complex and multimodal posterior distributions, and computational complexity of large neural network models. Parallel tempering MCMC addresses some of these limitations given that they can sample multimodal posterior distributions and utilize high-performance computing. However, certain challenges remain given large neural network models and big data. Surrogate-assisted optimization features the estimation of an objective function for models which are computationally expensive. In this paper, we address the inefficiency of parallel tempering MCMC for large-scale problems by combining parallel computing features with surrogate assisted likelihood estimation that describes the plausibility of a model parameter value, given specific observed data. Hence, we present surrogate-assisted parallel tempering for Bayesian neural learning for simple to computationally expensive models. Our results demonstrate that the methodology significantly lowers the computational cost while maintaining quality in decision making with Bayesian neural networks. The method has applications for a Bayesian inversion and uncertainty quantification for a broad range of numerical models.

Bayesreef: A Bayesian inference framework for modelling reef growth in response to environmental change and biological dynamics

Estimating the impact of environmental processes on vertical reef development in geological time is a very challenging task. pyReef-Core is a deterministic carbonate stratigraphic forward model designed to simulate the key biological and environmental processes that determine vertical reef accretion and assemblage changes in fossil reef drill cores. We present a Bayesian framework called Bayesreef for the estimation and uncertainty quantification of parameters in pyReef-Core that represent environmental conditions affecting the growth of coral assemblages in geological timescales. Weencounter multimodal posterior distributions and investigate the challenges of sampling using Markov chain Monte-Carlo (MCMC) methods, which includes parallel tempering MCMC. We use a synthetic reef-core to investigate fundamental issues and then apply the methodology to a selected reef-core from the Great Barrier Reef in Australia. The results show that Bayesreef accurately estimates and provides uncertainty quantification of the selected parameters that represent environment and ecological conditions in pyReef-Core. Bayesreef provides insights into the complex posterior distributions of the parameters in pyReef-Core, which provides the groundwork for future research in this area.

Parallel MCMC sampling of AR-HMMs for prediction based option trading

In this paper we propose stochastic time series prediction by autoregressive Hidden Markov Models (AR-HMM). The model parameter estimation, hence the prediction, is carried out by Markov chain Monte Carlo (MCMC) sampling instead of finding a single m

MCMC Techniques for Parameter Estimation of ODE Based Models in Systems Biology

Ordinary differential equation systems (ODEs) are frequently used for dynamical system modelling in many science fields such as economics, physics, engineering, and systems biology. A special challenge in systems biology is that ODE systems typically contain kinetic rate parameters, which are unknown and have to be estimated from data. However, non-linearity of ODE systems together with noise in the data raise severe identifiability issues. Hence, Markov Chain Monte Carlo (MCMC) approaches have been frequently used to estimate posterior distributions of rate parameters. However, designing a good MCMC sampler for high dimensional and multi-modal parameter distributions remains a challenging task. Here we performed a systematic comparison of different MCMC techniques for this purpose using five public domain models. The comparison included Metropolis-Hastings, parallel tempering MCMC, adaptive MCMC and parallel adaptive MCMC. In conclusion, we found specifically parallel adaptive MCMC to produce superior parameter estimates while benefitting from inclusion of our suggested informative Bayesian priors for rate parameters and noise variance.

Parallel MCMC methods for global optimization

Abstract We introduce a parallel scheme for simulated annealing, a widely used Markov chain Monte Carlo (MCMC) method for optimization. Our method is constructed and analyzed under the classical framework of MCMC. The benchmark function for optimization is used for validation and verification of the parallel scheme. The experimental results, along with the proof based on statistical theory, provide us with insights into the mechanics of the parallelization of simulated annealing for high parallel efficiency or scalability for large parallel computers.

Langevin-gradient parallel tempering for Bayesian neural learning

Bayesian inference provides a rigorous approach for neural learning with knowledge representation via the posterior distribution that accounts for uncertainty quantification. Markov Chain Monte Carlo (MCMC) methods typically implement Bayesian inference by sampling from the posterior distribution. This not only provides point estimates of the weights, but the ability to propagate and quantify uncertainty in decision making. However, these techniques face challenges in convergence and scalability, particularly in settings with large datasets and neural network architectures. This paper addresses these challenges in two ways. First, parallel tempering MCMC sampling method is used to explore multiple modes of the posterior distribution and implemented in multi-core computing architecture. Second, we make within-chain sampling scheme more efficient by using Langevin gradient information for creating Metropolis–Hastings proposal distributions. We demonstrate the techniques using time series prediction and pattern classification applications. The results show that the method not only improves the computational time, but provides better decision making capabilities when compared to related methods.

Parallel Metropolis–Hastings Coupler

Bayesian methods and their implementations by means of sophisticated Monte Carlo techniques, such as Markov chain Monte Carlo (MCMC) and particle filters, have become very popular in signal processing over the last years. In this letter, we present a novel interacting parallel MCMC scheme, called parallel Metropolis–Hastings coupler (PMHC), where the information provided by different parallel MCMC chains is properly combined by the use of another advanced MCMC method, called normal kernel coupler (NKC). The NKC employs a mixture of densities as proposal density, which is updated according to a population of states. The PMHC is particularly efficient in multimodal scenarios, since it obtains a faster exploration of the state space with respect to other benchmark techniques. Several numerical simulations are provided showing the efficiency and robustness of the proposed method.

Parallel Markov chain Monte Carlo for Bayesian hierarchical models with big data, in two stages

ABSTRACTDue to the escalating growth of big data sets in recent years, new Bayesian Markov chain Monte Carlo (MCMC) parallel computing methods have been developed. These methods partition large data sets by observations into subsets. However, for Bayesian nested hierarchical models, typically only a few parameters are common for the full data set, with most parameters being group specific. Thus, parallel Bayesian MCMC methods that take into account the structure of the model and split the full data set by groups rather than by observations are a more natural approach for analysis. Here, we adapt and extend a recently introduced two-stage Bayesian hierarchical modeling approach, and we partition complete data sets by groups. In stage 1, the group-specific parameters are estimated independently in parallel. The stage 1 posteriors are used as proposal distributions in stage 2, where the target distribution is the full model. Using three-level and four-level models, we show in both simulation and real data studies that results of our method agree closely with the full data analysis, with greatly increased MCMC efficiency and greatly reduced computation times. The advantages of our method versus existing parallel MCMC computing methods are also described.

Parallel multiple-chain DRAM MCMC for large-scale geosteering inversion and uncertainty quantification

Geosteering is the proactive control of a wellbore placement based on the downhole measurements, aiming at maximizing economic production from the well. The recent development of azimuthal resistivity logging-while-drilling (LWD) tools have a much larger depth of detection, thus sets higher demands for geosteering inversion as more unknown parameters need to be taken into account in the earth model to be inverted. For complicated non-linear problems, traditional deterministic inversion methods are more likely to be trapped near local optima. In general, Markov chain Monte Carlo (MCMC) inversion methods are more capable of finding the global optimal solution and providing additional uncertainty analyses by sampling from the target distribution. However, MCMC methods usually have the problem of slow convergence. Even though optimization methods like delayed rejection (DR) and adaptive Metropolis (AM) were proposed to speed up the convergence, using MCMC methods to solve geosteering inversion problems may still incur unacceptable time cost. To reduce the sampling cost of MCMC methods, we use parallel multiple-chain DRAM MCMC methods to solve geosteering inverse problems and estimate the corresponding uncertainty. A clustering method, density-based spatial clustering of applications with noise (DBSCAN), is applied to select the best solution among many results. The simulation results show that running many relatively short MCMC chains for one problem can obtain a similar result as running a long single chain. Besides, avoiding communications between multiple Markov chains during the sampling process yields almost linear scalability.

APT-MCMC, a C++/Python implementation of Markov Chain Monte Carlo for parameter identification

The inverse problem associated with fitting parameters of an ordinary differential equation (ODE) system to data is nonlinear and multimodal, which is of great challenge to gradient-based optimizers. Markov Chain Monte Carlo (MCMC) techniques provide an alternative approach to solving these problems and can escape local minima by design. APT-MCMC was created to allow users to setup ODE simulations in Python and run as compiled C++ code. It combines affine-invariant ensemble of samplers and parallel tempering MCMC techniques to improve the simulation efficiency. Simulations use Bayesian inference to provide probability distributions of parameters, which enable analysis of multiple minima and parameter correlation.Benchmark tests result in a 20×–60× speedup but 14% increase in memory usage against emcee, a similar MCMC package in Python. Several MCMC hyperparameters were analyzed: number of temperatures, ensemble size, step size, and swap attempt frequency. Heuristic tuning guidelines are provided for setting these hyperparameters.

Powered embarrassing parallel MCMC sampling in Bayesian inference, a weighted average intuition

Although the Markov Chain Monte Carlo (MCMC) is very popular in parameter inference, the alleviation of the burden of calculation is crucial due to the limit of processors, memory, and disk bottleneck. This is especially true in terms of handling big data. In recent years, researchers have developed a parallel MCMC algorithm, in which full data are partitioned into subdatasets. Samples are drawn from the subdatasets independently at different machines without communication. In the extant literature, all machines are deemed to be identical. However, due to the heterogeneity of the data put into different machines, and the random nature of MCMC, the assumption of “identical machines” is questionable. Here we propose a Powered Embarrassing Parallel MCMC (PEPMCMC) algorithm, in which the full data posterior density is the product of the sub-posterior densities (posterior densities of different subdatasets) raised by some constraint powers. This is proven to be equivalent to a weighted averaging procedure. In our work, the powers are determined based on a maximum likelihood criterion, which leads to finding a maximum likelihood point within the convex hull of the estimates from different machines. We prove the asymptotic exactness and apply it to several cases to verify its strength in comparison with the unparallel and unpowered parallel algorithms. Furthermore, the connection between normal kernel density and parametric density estimations under certain conditions is investigated.

Orthogonal parallel MCMC methods for sampling and optimization

Monte Carlo (MC) methods are widely used for Bayesian inference and optimization in statistics, signal processing and machine learning. A well-known class of MC methods are Markov Chain Monte Carlo (MCMC) algorithms. In order to foster better exploration of the state space, specially in high-dimensional applications, several schemes employing multiple parallel MCMC chains have been recently introduced. In this work, we describe a novel parallel interacting MCMC scheme, called orthogonal MCMC (O-MCMC), where a set of “vertical” parallel MCMC chains share information using some “horizontal” MCMC techniques working on the entire population of current states. More specifically, the vertical chains are led by random-walk proposals, whereas the horizontal MCMC techniques employ independent proposals, thus allowing an efficient combination of global exploration and local approximation. The interaction is contained in these horizontal iterations. Within the analysis of different implementations of O-MCMC, novel schemes in order to reduce the overall computational cost of parallel Multiple Try Metropolis (MTM) chains are also presented. Furthermore, a modified version of O-MCMC for optimization is provided by considering parallel Simulated Annealing (SA) algorithms. Numerical results show the advantages of the proposed sampling scheme in terms of efficiency in the estimation, as well as robustness in terms of independence with respect to initial values and the choice of the parameters.

Parallel MCMC with generalized elliptical slice sampling

Probabilistic models are conceptually powerful tools for finding structure in data, but their practical effectiveness is often limited by our ability to perform inference in them. Exact inference is frequently intractable, so approximate inference is often performed using Markov chain Monte Carlo (MCMC). To achieve the best possible results from MCMC, we want to effciently simulate many steps of a rapidly mixing Markov chain which leaves the target distribution invariant. Of particular interest in this regard is how to take advantage of multi-core computing to speed up MCMC-based inference, both to improve mixing and to distribute the computational load. In this paper, we present a parallelizable Markov chain Monte Carlo algorithm for effciently sampling from continuous probability distributions that can take advantage of hundreds of cores. This method shares information between parallel Markov chains to build a scale-location mixture of Gaussians approximation to the density function of the target distribution. We combine this approximation with a recently developed method known as elliptical slice sampling to create a Markov chain with no step-size parameters that can mix rapidly without requiring gradient or curvature computations.

Threshold variable selection of asymmetric stochastic volatility models

A threshold stochastic volatility (SV) model is used for capturing time-varying volatilities and nonlinearity. Two adaptive Markov chain Monte Carlo (MCMC) methods of model selection are designed for the selection of threshold variables for this family of SV models. The first method is the direct estimation which approximates the model posterior probabilities of competing models. Using parallel MCMC sampling to estimate these probabilities, the best threshold variable is selected with the highest posterior model probability. The second method is to use the deviance information criterion to compare among these competing models and select the best one. Simulation results lead us to conclude that for large samples the posterior model probability approximation method can give an accurate approximation of the posterior probability in Bayesian model selection. The method delivers a powerful and sharp model selection tool. An empirical study of five Asian stock markets provides strong support for the threshold variable which is formulated as a weighted average of important variables.

SwiftLink: parallel MCMC linkage analysis using multicore CPU and GPU

Linkage analysis remains an important tool in elucidating the genetic component of disease and has become even more important with the advent of whole exome sequencing, enabling the user to focus on only those genomic regions co-segregating with Mendelian traits. Unfortunately, methods to perform multipoint linkage analysis scale poorly with either the number of markers or with the size of the pedigree. Large pedigrees with many markers can only be evaluated with Markov chain Monte Carlo (MCMC) methods that are slow to converge and, as no attempts have been made to exploit parallelism, massively underuse available processing power. Here, we describe SWIFTLINK, a novel application that performs MCMC linkage analysis by spreading the computational burden between multiple processor cores and a graphics processing unit (GPU) simultaneously. SWIFTLINK was designed around the concept of explicitly matching the characteristics of an algorithm with the underlying computer architecture to maximize performance. We implement our approach using existing Gibbs samplers redesigned for parallel hardware. We applied SWIFTLINK to a real-world dataset, performing parametric multipoint linkage analysis on a highly consanguineous pedigree with EAST syndrome, containing 28 members, where a subset of individuals were genotyped with single nucleotide polymorphisms (SNPs). In our experiments with a four core CPU and GPU, SWIFTLINK achieves a 8.5× speed-up over the single-threaded version and a 109× speed-up over the popular linkage analysis program SIMWALK. SWIFTLINK is available at https://github.com/ajm/swiftlink. All source code is licensed under GPLv3.