Multimodal Posterior Distributions Research Articles

Copula Approximate Bayesian Computation Using Distribution Random Forests

Ongoing modern computational advancements continue to make it easier to collect increasingly large and complex datasets, which can often only be realistically analyzed using models defined by intractable likelihood functions. This Stats invited feature article introduces and provides an extensive simulation study of a new approximate Bayesian computation (ABC) framework for estimating the posterior distribution and the maximum likelihood estimate (MLE) of the parameters of models defined by intractable likelihoods, that unifies and extends previous ABC methods proposed separately. This framework, copulaABCdrf, aims to accurately estimate and describe the possibly skewed and high-dimensional posterior distribution by a novel multivariate copula-based meta-t distribution based on univariate marginal posterior distributions that can be accurately estimated by distribution random forests (drf), while performing automatic summary statistics (covariates) selection, based on robustly estimated copula dependence parameters. The copulaABCdrf framework also provides a novel multivariate mode estimator to perform MLE and posterior mode estimation and an optional step to perform model selection from a given set of models using posterior probabilities estimated by drf. The posterior distribution estimation accuracy of the ABC framework is illustrated and compared with previous standard ABC methods through several simulation studies involving low- and high-dimensional models with computable posterior distributions, which are either unimodal, skewed, or multimodal; and exponential random graph and mechanistic network models, each defined by an intractable likelihood from which it is costly to simulate large network datasets. This paper also proposes and studies a new solution to the simulation cost problem in ABC involving the posterior estimation of parameters from datasets simulated from the given model that are smaller compared to the potentially large size of the dataset being analyzed. This proposal is motivated by the fact that, for many models defined by intractable likelihoods, such as the network models when they are applied to analyze massive networks, the repeated simulation of large datasets (networks) for posterior-based parameter estimation can be too computationally costly and vastly slow down or prohibit the use of standard ABC methods. The copulaABCdrf framework and standard ABC methods are further illustrated through analyses of large real-life networks of sizes ranging between 28,000 and 65.6 million nodes (between 3 million and 1.8 billion edges), including a large multilayer network with weighted directed edges. The results of the simulation studies show that, in settings where the true posterior distribution is not highly multimodal, copulaABCdrf usually produced similar point estimates from the posterior distribution for low-dimensional parametric models as previous ABC methods, but the copula-based method can produce more accurate estimates from the posterior distribution for high-dimensional models, and, in both dimensionality cases, usually produced more accurate estimates of univariate marginal posterior distributions of parameters. Also, posterior estimation accuracy was usually improved when pre-selecting the important summary statistics using drf compared to ABC employing no pre-selection of the subset of important summaries. For all ABC methods studied, accurate estimation of a highly multimodal posterior distribution was challenging. In light of the results of all the simulation studies, this article concludes by discussing how the copulaABCdrf framework can be improved for future research.

Toward Robust Corrections for Stellar Contamination in JWST Exoplanet Transmission Spectra

Transmission spectroscopy is still the preferred characterization technique for exoplanet atmospheres, although it presents unique challenges that translate into characterization bottlenecks when robust mitigation strategies are missing. Stellar contamination is one such challenge that can overpower the planetary signal by up to an order of magnitude, and thus not accounting for it can lead to significant biases in the derived atmospheric properties. Yet this accounting may not be straightforward, as important discrepancies exist between state-of-the-art stellar models and measured spectra and between models themselves. Here we explore the extent to which stellar models can be used to reliably correct for stellar contamination and yield a planet’s uncontaminated transmission spectrum. We find that discrepancies between stellar models can significantly contribute to the noise budget of JWST transmission spectra of planets around stars with heterogeneous photospheres, the true number of unique photospheric spectral components and their properties can only be accurately retrieved when the stellar models have sufficient fidelity, and under such optimistic circumstances the contribution of stellar contamination to the noise budget of a transmission spectrum is considerably below that of the photon noise for the standard transit observation setup. Therefore, we advocate for further development of model spectra of stars and their active regions in a data-driven manner, empirical approaches for deriving spectra of photospheric components using the observatories with which the atmospheric explorations are carried out, and analysis techniques accounting for multimodal posterior distributions for photospheric parameters of interest, which will be increasingly revealed by precise JWST measurements.

The asteroseismic imprints of mass transfer

We present new simulations investigating the impact of mass transfer on the asteroseismic signals of slowly pulsating B stars. We used MESA to simulate the evolution of a binary star system and GYRE to compute the asteroseismic properties of the accretor star. We show that, compared to a single star of the same final mass, a star that has undergone accretion (of non-enriched material) has a significantly different internal structure, which is evident in both the hydrogen abundance profile and the Brunt-Väisälä frequency profile. These differences result in significant changes in the observed period spacing patterns, implying that one may use this as a diagnostic to test whether a star’s core has been rejuvenated as a result of accretion. We show that it is essential to consider the full multimodal posterior distributions when fitting stellar properties of mass-gainers to avoid drawing misleading conclusions. Even with these considerations, stellar ages will be significantly underestimated when assuming single star evolution for a mass-gainer. We find that future detectors with improved uncertainties would rule out single star models with the correct mass and central hydrogen fraction. Our proof of principle analysis demonstrates the need to further investigate the impact of binary interactions on stellar asteroseismic signals for a wide range of parameters, such as the initial mass, the amount of mass transferred, and the age of the accretor star at the onset of mass transfer.

Sequential Monte Carlo with cross-validated neural networks for complexity of hyperbolic black hole solutions in 4D

This paper investigates the self-similar solutions of the Einstein-axion-dilaton configuration from type IIB string theory and the global SL(2,R) symmetry. We consider the Continuous Self Similarity (CSS), where the scale transformation is controlled by an SL(2, R) boost or hyperbolic translation. The solutions stay invariant under the combination of space-time dilation with internal SL(2,R) transformations. We develop a new formalism based on Sequential Monte Carlo (SMC) and artificial neural networks (NNs) to estimate the self-similar solutions to the equations of motion in the hyperbolic class in four dimensions. Due to the complex and highly nonlinear patterns, researchers typically have to use various constraints and numerical approximation methods to estimate the equations of motion; thus, they have to overlook the measurement errors in parameter estimation. Through a Bayesian framework, we incorporate measurement errors into our models to find the solutions to the hyperbolic equations of motion. It is well known that the hyperbolic class suffers from multiple solutions where the critical collapse functions have overlap domains for these solutions. To deal with this complexity, for the first time in literature on the axion-dilaton system, we propose the SMC approach to obtain the multi-modal posterior distributions. Through a probabilistic perspective, we confirm the deterministic α\\documentclass[12pt]{minimal} \\usepackage{amsmath} \\usepackage{wasysym} \\usepackage{amsfonts} \\usepackage{amssymb} \\usepackage{amsbsy} \\usepackage{mathrsfs} \\usepackage{upgreek} \\setlength{\\oddsidemargin}{-69pt} \\begin{document}$$\\alpha $$\\end{document} and β\\documentclass[12pt]{minimal} \\usepackage{amsmath} \\usepackage{wasysym} \\usepackage{amsfonts} \\usepackage{amssymb} \\usepackage{amsbsy} \\usepackage{mathrsfs} \\usepackage{upgreek} \\setlength{\\oddsidemargin}{-69pt} \\begin{document}$$\\beta $$\\end{document} solutions available in the literature and determine all possible solutions that may occur due to measurement errors. We finally proposed the penalized Leave-One-Out Cross-validation (LOOCV) to combine the Bayesian NN-based estimates optimally. The approach enables us to determine the optimum weights while dealing with the co-linearity issue in the NN-based estimates and better predict the critical functions corresponding to multiple solutions of the equations of motion.

Flexible Variational Bayes Based on a Copula of a Mixture

Variational Bayes methods approximate the posterior density by a family of tractable distributions whose parameters are estimated by optimization. Variational approximation is useful when exact inference is intractable or very costly. Our article develops a flexible variational approximation based on a copula of a mixture, which is implemented by combining boosting, natural gradient, and a variance reduction method. The efficacy of the approach is illustrated by using simulated and real datasets to approximate multimodal, skewed and heavy-tailed posterior distributions, including an application to Bayesian deep feedforward neural network regression models. Supplementary materials, including appendices and computer code for this article, are available online.

Efficient bayes inference in neural networks through adaptive importance sampling

Bayesian neural networks (BNNs) have received an increased interest in the last years. In BNNs, a complete posterior distribution of the unknown weight and bias parameters of the network is produced during the training stage. This probabilistic estimation offers several advantages with respect to point-wise estimates, in particular, the ability to provide uncertainty quantification when predicting new data. This feature inherent to the Bayesian paradigm, is useful in countless machine learning applications. It is particularly appealing in areas where decision-making has a crucial impact, such as medical healthcare or autonomous driving. The main challenge of BNNs is the computational cost of the training procedure since Bayesian techniques often face a severe curse of dimensionality. Adaptive importance sampling (AIS) is one of the most prominent Monte Carlo methodologies benefiting from sounded convergence guarantees and ease for adaptation. This work aims to show that AIS constitutes a successful approach for designing BNNs. More precisely, we propose a novel algorithm named PMCnet that includes an efficient adaptation mechanism, exploiting geometric information on the complex (often multimodal) posterior distribution. Numerical results illustrate the excellent performance and the improved exploration capabilities of the proposed method for both shallow and deep neural networks.

Pattern-adaptive generative adversarial network with sparse data for traffic state estimation

Accurate traffic state estimation is vital basis for traffic control and management applications. However, owing to the multi-modality of traffic state patterns, the estimation methods are hard to adapt to the state variation over the urban roads and time periods. Especially, the sparse sampling of traffic state makes the traffic pattern recognition easy to bias and increases uncertainty in the estimation. To address the problem of multi-modal traffic state estimation under sparse data, we propose a pattern-adaptive generative adversarial network, named PA-GAN. In the PA-GAN, the Bayesian Inference is introduced to place multi-modal posterior distributions over the network parameters. Therefore, to targeted state estimation for each pattern, the posterior sampling will adaptively activate the corresponding parameters of each traffic pattern according to the context features. Then, the PA-GAN learns traffic patterns from sparse sampling data by the traffic state generator and the discriminator, in which an error-feedback mechanism uses multi-level traffic features to correct the estimation under an encoder–decoder framework. To evaluate the proposed PA-GAN, we use two real-world datasets to conduct comprehensive case studies about multi-modal traffic patterns and sparse sampling. The experimental results demonstrate that the PA-GAN can outperform other estimation methods and that the Bayesian Inference can improve the adaptability ability of the learning network to various traffic states.

Cyclical Variational Bayes Monte Carlo for efficient multi-modal posterior distributions evaluation

Multi-modal distributions of some physics-based model parameters are often encountered in engineering due to different situations such as a change in some environmental conditions, and the presence of some types of damage and non-linearity. In statistical model updating, for locally identifiable parameters, it can be anticipated that multi-modal posterior distributions would be found. The full characterization of these multi-modal distributions is important as methodologies for structural condition monitoring in structures are frequently based in the comparison of the damaged and healthy models of the structure. The characterization of posterior multi-modal distributions using state-of-the-art sampling techniques would require a large number of simulations of expensive-to-run physics-based models. Therefore, when a limited number of simulations can be run, as it often occurs in engineering, the traditional sampling techniques would not be able to capture accurately the multi-modal distributions. This could potentially lead to large numerical errors when assessing the performance of an engineering structure under uncertainty.Therefore, an approach is proposed for drastically reducing the number of models runs while yielding accurate estimates of highly multi-modal posterior distributions. This approach introduces a cyclical annealing schedule into the Variational Bayes Monte Carlo (VBMC) method to improve the algorithm’s phase of exploration and the finding of high probability areas in the multi-modal posteriors throughout the different cycles.Three numerical and one experimental investigations are used to compare the proposed cyclical VBMC with the standard VBMC algorithm, the monotonic VBMC and the Transitional Ensemble Markov Chain Monte Carlo (TEMCMC). It is shown that the standard VBMC fails in capturing multi-modal posteriors as it is unable to escape already found regions of high posterior density. In the presence of highly multi-modal posteriors, the proposed cyclical VBMC algorithm outperforms all the other approaches in terms of accuracy of the resulting posterior, and number of model runs required.

Dark lenses through the dust: parallax microlensing events in the VVV

ABSTRACT We use near-infrared photometry and astrometry from the VISTA Variables in the Via Lactea (VVV) survey to analyse microlensing events containing annual microlensing parallax information. These events are located in highly extincted and low-latitude regions of the Galactic bulge typically off-limits to optical microlensing surveys. We fit a catalogue of 1959 events previously found in the VVV and extract 21 microlensing parallax candidates. The fitting is done using nested sampling to automatically characterize the multimodal and degenerate posterior distributions of the annual microlensing parallax signal. We compute the probability density in lens mass-distance using the source proper motion and a Galactic model of disc and bulge deflectors. By comparing the expected flux from a main sequence lens to the baseline magnitude and blending parameter, we identify four candidates which have probability >50 per cent that the lens is dark. The strongest candidate corresponds to a nearby (≈0.78 kpc), medium-mass ($1.46^{+1.13}_{-0.71} \ M_{\odot }$) dark remnant as lens. In the next strongest, the lens is located at heliocentric distance ≈5.3 kpc. It is a dark remnant with a mass of $1.63^{+1.15}_{-0.70} \ M_{\odot }$. Both of those candidates are most likely neutron stars, though possibly high-mass white dwarfs. The last two events may also be caused by dark remnants, though we are unable to rule out other possibilities because of limitations in the data. We are also demonstrating future possibilities of studying similar events with the Roman Space Telescopeby modelling a mock dataset of Roman photometry and astrometry for an event resembling our strongest candidate.

Model Based Screening Embedded Bayesian Variable Selection for Ultra-high Dimensional Settings

We develop a Bayesian variable selection method, called SVEN, based on a hierarchical Gaussian linear model with priors placed on the regression coefficients as well as on the model space. Sparsity is achieved by using degenerate spike priors on inactive variables, whereas Gaussian slab priors are placed on the coefficients for the important predictors making the posterior probability of a model available in explicit form (up to a normalizing constant). Embedding a unique model based screening and using fast Cholesky updates, SVEN produces a highly scalable computational framework to explore gigantic model spaces, rapidly identify the regions of high posterior probabilities and make fast inference and prediction. A temperature schedule is used to further mitigate multimodal posterior distributions. The temperature value is guided by our model selection consistency results which hold even when the norm of mean effects solely due to the unimportant variables diverges. An appealing byproduct of SVEN is the construction of novel model weight adjusted prediction intervals. The performance of SVEN is demonstrated through a number of simulation experiments and a real data example from a genome wide association study with over half a million markers. Supplementary materials for this article are available online.

A Detailed Analysis of GW190521 with Phenomenological Waveform Models

Abstract In this paper we present an extensive analysis of the GW190521 gravitational wave event with the current (fourth) generation of phenomenological waveform models for binary black hole coalescences. GW190521 stands out from other events since only a few wave cycles are observable. This leads to a number of challenges, one being that such short signals are prone to not resolving approximate waveform degeneracies, which may result in multimodal posterior distributions. The family of waveform models we use includes a new fast time-domain model (IMRPhenomTPHM), which allows us to extensively test different priors and robustness with respect to variations in the waveform model, including the content of spherical harmonic modes. We clarify some issues raised in a recent paper, Nitz & Capano, associated with possible support for a high-mass-ratio source, but confirm their finding of a multimodal posterior distribution, albeit with important differences in the statistical significance of the peaks. In particular, we find that the support for both masses being outside the pair instability supernova mass gap, and the support for an intermediate-mass-ratio binary are drastically reduced with respect to what Nitz & Capano found. We also provide updated probabilities for associating GW190521 to the potential electromagnetic counterpart from the Zwicky Transient Facility (ZTF) Graham et al.

Randomized maximum likelihood based posterior sampling

Minimization of a stochastic cost function is commonly used for approximate sampling in high-dimensional Bayesian inverse problems with Gaussian prior distributions and multimodal posterior distributions. The density of the samples generated by minimization is not the desired target density, unless the observation operator is linear, but the distribution of samples is useful as a proposal density for importance sampling or for Markov chain Monte Carlo methods. In this paper, we focus on applications to sampling from multimodal posterior distributions in high dimensions. We first show that sampling from multimodal distributions is improved by computing all critical points instead of only minimizers of the objective function. For applications to high-dimensional geoscience inverse problems, we demonstrate an efficient approximate weighting that uses a low-rank Gauss-Newton approximation of the determinant of the Jacobian. The method is applied to two toy problems with known posterior distributions and a Darcy flow problem with multiple modes in the posterior.

HypoSVI: Hypocentre inversion with Stein variational inference and physics informed neural networks

SUMMARY We introduce a scheme for probabilistic hypocentre inversion with Stein variational inference. Our approach uses a differentiable forward model in the form of a physics informed neural network, which we train to solve the Eikonal equation. This allows for rapid approximation of the posterior by iteratively optimizing a collection of particles against a kernelized Stein discrepancy. We show that the method is well-equipped to handle highly multimodal posterior distributions, which are common in hypocentral inverse problems. A suite of experiments is performed to examine the influence of the various hyperparameters. Once trained, the method is valid for any seismic network geometry within the study area without the need to build traveltime tables. We show that the computational demands scale efficiently with the number of differential times, making it ideal for large-N sensing technologies like Distributed Acoustic Sensing. The techniques outlined in this manuscript have considerable implications beyond just ray tracing procedures, with the work flow applicable to other fields with computationally expensive inversion procedures such as full waveform inversion.

Multimodal Bayesian registration of noisy functions using Hamiltonian Monte Carlo

Functional data registration is a necessary processing step for many applications. The observed data can be inherently noisy, often due to measurement error or natural process uncertainty; which most functional alignment methods cannot handle. A pair of functions can also have multiple optimal alignment solutions, which is not addressed in current literature. In this paper, a flexible Bayesian approach to functional alignment is presented, which appropriately accounts for noise in the data without any pre-smoothing required. Additionally, by running parallel MCMC chains, the method can account for multiple optimal alignments via the multi-modal posterior distribution of the warping functions. To most efficiently sample the warping functions, the approach relies on a modification of the standard Hamiltonian Monte Carlo to be well-defined on the infinite-dimensional Hilbert space. This flexible Bayesian alignment method is applied to both simulated data and real data sets to show its efficiency in handling noisy functions and successfully accounting for multiple optimal alignments in the posterior; characterizing the uncertainty surrounding the warping functions.

Bayesian signal processing: A practical guide to particle filtering design and performance

When uncertain acoustic processes can no longer be characterized by Gaussian or for that matter unimodal (single peak) distributions along with the fact that the underlying phenomenology is nonstationary (time-varying) and nonlinear, then more general Bayesian processors must be applied to solve the underlying signal enhancement/extraction problem. A particle filter provides a solution to this multimodal (multiple peaks) posterior distribution estimation problem in noisy acoustic environments. A particle filter is a sequential Markov chain Monte Carlo processor capable of providing reasonable performance for data evolving from a multimodal distribution by estimating a nonparametric representation of the posterior distribution from which a multitude of meaningful statistics can be retrieved. However, the question of evaluating its performance can be challenging even for the simplest of processes. For instance, it is well-known that Kalman filter optimality can be obtained only when the resulting error residuals (innovations) are zero-mean and white. It is not that simple for the particle filter. Once characterized, the performance of the particle filter must be analyzed for it to be of practical value. Here a set of design and analysis criteria is discussed and applied to demonstrate their ability to quantify particle filtering performance.

Bayesian approach to inverse scattering with topological priors

We propose a Bayesian inference framework to estimate uncertainties in inverse scattering problems. Given the observed data, the forward model and their uncertainties, we find the posterior distribution over a finite parameter field representing the objects. To construct the prior distribution we use a topological sensitivity analysis. We demonstrate the approach on the Bayesian solution of 2D inverse problems in light and acoustic holography with synthetic data. Statistical information on objects such as their center location, diameter size, orientation, as well as material properties, are extracted by sampling the posterior distribution. Assuming the number of objects known, comparison of the results obtained by Markov Chain Monte Carlo (MCMC) sampling and by sampling a Gaussian distribution found by linearization about the maximum a posteriori estimate show reasonable agreement. The latter procedure has low computational cost, which makes it an interesting tool for uncertainty studies in 3D. However, MCMC sampling provides a more complete picture of the posterior distribution and yields multi-modal posterior distributions for problems with larger measurement noise. When the number of objects is unknown, we devise a stochastic model selection framework.

Stochastic approximation Hamiltonian Monte Carlo

Recently, the Hamilton Monte Carlo (HMC) has become widespread as one of the more reliable approaches to efficient sample generation processes. However, HMC is difficult to sample in a multimodal posterior distribution because the HMC chain cannot cross energy barrier between modes due to the energy conservation property. In this paper, we propose a Stochastic Approximate Hamilton Monte Carlo (SAHMC) algorithm for generating samples from multimodal density under the Hamiltonian Monte Carlo (HMC) framework. SAHMC can adaptively lower the energy barrier to move the Hamiltonian trajectory more frequently and more easily between modes. Our simulation studies show that the potential for SAHMC to explore a multimodal target distribution is more efficient than HMC-based implementations.

Parallel generalized elliptical slice sampling with adaptive regional pseudo-priors

MCMC algorithm is well-known for having difficulty exploring distant modes when the target distribution is multi-modal. The reason is that a proposal state is likely to be rejected when traversing across low density regions. Focusing on this issue, we proposed parallel generalized elliptical slice sampling algorithm with adaptive regional pseudo-prior (RGESS). Different from the work of Fagan et al. [2016. Elliptical slice sampling with expectation propagation. In: UAI] and Nishihara et al. [Parallel mcmc with generalized elliptical slice sampling. J Mach Learn Res. 2014;15:2087–2112], different pseudo-priors are used at different regions to conduct the generalized elliptical slice sampling (GESS) algorithm. The rejection rate is modified to guarantee detailed balance condition. We also employ adaptive transition kernel and parallel computing to accelerate sampling speed. Experimental results on one synthetic and one real-world dataset show that the proposed algorithm has the following advantages: with the same starting points, the proposed algorithm spend less time to find the modes of the target distribution; after finding the modes, the proposed algorithm has less interactions between modes due to parameters adaption, which leads to lower rejection rate; when estimating the parameters of multi-modal posterior distributions, the samples generated by proposed algorithm can find different modes better.

Surrogate-assisted parallel tempering for Bayesian neural learning

Due to the need for robust uncertainty quantification, Bayesian neural learning has gained attention in the era of deep learning and big data. Markov Chain Monte-Carlo (MCMC) methods typically implement Bayesian inference which faces several challenges given a large number of parameters, complex and multimodal posterior distributions, and computational complexity of large neural network models. Parallel tempering MCMC addresses some of these limitations given that they can sample multimodal posterior distributions and utilize high-performance computing. However, certain challenges remain given large neural network models and big data. Surrogate-assisted optimization features the estimation of an objective function for models which are computationally expensive. In this paper, we address the inefficiency of parallel tempering MCMC for large-scale problems by combining parallel computing features with surrogate assisted likelihood estimation that describes the plausibility of a model parameter value, given specific observed data. Hence, we present surrogate-assisted parallel tempering for Bayesian neural learning for simple to computationally expensive models. Our results demonstrate that the methodology significantly lowers the computational cost while maintaining quality in decision making with Bayesian neural networks. The method has applications for a Bayesian inversion and uncertainty quantification for a broad range of numerical models.

Bayesreef: A Bayesian inference framework for modelling reef growth in response to environmental change and biological dynamics

Estimating the impact of environmental processes on vertical reef development in geological time is a very challenging task. pyReef-Core is a deterministic carbonate stratigraphic forward model designed to simulate the key biological and environmental processes that determine vertical reef accretion and assemblage changes in fossil reef drill cores. We present a Bayesian framework called Bayesreef for the estimation and uncertainty quantification of parameters in pyReef-Core that represent environmental conditions affecting the growth of coral assemblages in geological timescales. Weencounter multimodal posterior distributions and investigate the challenges of sampling using Markov chain Monte-Carlo (MCMC) methods, which includes parallel tempering MCMC. We use a synthetic reef-core to investigate fundamental issues and then apply the methodology to a selected reef-core from the Great Barrier Reef in Australia. The results show that Bayesreef accurately estimates and provides uncertainty quantification of the selected parameters that represent environment and ecological conditions in pyReef-Core. Bayesreef provides insights into the complex posterior distributions of the parameters in pyReef-Core, which provides the groundwork for future research in this area.