Black Box Variational Inference Research Articles

Bernstein flows for flexible posteriors in variational Bayes

Black-box variational inference (BBVI) is a technique to approximate the posterior of Bayesian models by optimization. Similar to MCMC, the user only needs to specify the model; then, the inference procedure is done automatically. In contrast to MCMC, BBVI scales to many observations, is faster for some applications, and can take advantage of highly optimized deep learning frameworks since it can be formulated as a minimization task. In the case of complex posteriors, however, other state-of-the-art BBVI approaches often yield unsatisfactory posterior approximations. This paper presents Bernstein flow variational inference (BF-VI), a robust and easy-to-use method flexible enough to approximate complex multivariate posteriors. BF-VI combines ideas from normalizing flows and Bernstein polynomial-based transformation models. In benchmark experiments, we compare BF-VI solutions with exact posteriors, MCMC solutions, and state-of-the-art BBVI methods, including normalizing flow-based BBVI. We show for low-dimensional models that BF-VI accurately approximates the true posterior; in higher-dimensional models, BF-VI compares favorably against other BBVI methods. Further, using BF-VI, we develop a Bayesian model for the semi-structured melanoma challenge data, combining a CNN model part for image data with an interpretable model part for tabular data, and demonstrate, for the first time, the use of BBVI in semi-structured models.

Machine learning and model driven bayesian uncertainty quantification in suspended nonstructural systems

This paper presents a novel framework for the uncertainty quantification of inverse problems often encountered in suspended nonstructural systems. This framework adopts machine learning- and model-driven stochastic Gaussian process model calibration to quantify the uncertainty via a new blackbox variational inference that accounts for geometric complexity through Bayesian inference. The soundness of the proposed framework is validated by examining one of the largest full-scale shaking table tests of suspended nonstructural systems and accompanying simulated (numerical) data. Our findings indicate that the proposed framework is computationally sound and scalable and yields optimal generalizability.

An Overdispersed Black-Box Variational Bayesian-Kalman Filter with Inaccurate Noise Second-Order Statistics.

Aimed at the problems in which the performance of filters derived from a hypothetical model will decline or the cost of time of the filters derived from a posterior model will increase when prior knowledge and second-order statistics of noise are uncertain, a new filter is proposed. In this paper, a Bayesian robust Kalman filter based on posterior noise statistics (KFPNS) is derived, and the recursive equations of this filter are very similar to that of the classical algorithm. Note that the posterior noise distributions are approximated by overdispersed black-box variational inference (O-BBVI). More precisely, we introduce an overdispersed distribution to push more probability density to the tails of variational distribution and incorporated the idea of importance sampling into two strategies of control variates and Rao–Blackwellization in order to reduce the variance of estimators. As a result, the convergence process will speed up. From the simulations, we can observe that the proposed filter has good performance for the model with uncertain noise. Moreover, we verify the proposed algorithm by using a practical multiple-input multiple-output (MIMO) radar system.

Variational inference using approximate likelihood under the coalescent with recombination.

Coalescent methods are proven and powerful tools for population genetics, phylogenetics, epidemiology, and other fields. A promising avenue for the analysis of large genomic alignments, which are increasingly common, is coalescent hidden Markov model (coalHMM) methods, but these methods have lacked general usability and flexibility. We introduce a novel method for automatically learning a coalHMM and inferring the posterior distributions of evolutionary parameters using black-box variational inference, with the transition rates between local genealogies derived empirically by simulation. This derivation enables our method to work directly with three or four taxa and through a divide-and-conquer approach with more taxa. Using a simulated data set resembling a human–chimp–gorilla scenario, we show that our method has comparable or better accuracy to previous coalHMM methods. Both species divergence times and population sizes were accurately inferred. The method also infers local genealogies, and we report on their accuracy. Furthermore, we discuss a potential direction for scaling the method to larger data sets through a divide-and-conquer approach. This accuracy means our method is useful now, and by deriving transition rates by simulation, it is flexible enough to enable future implementations of various population models.

A Novel Anti-Jamming Technique for INS/GNSS Integration Based on Black Box Variational Inference

In this paper, a novel anti-jamming technique based on black box variational inference for INS/GNSS integration with time-varying measurement noise covariance matrices is presented. We proved that the time-varying measurement noise is more similar to the Gaussian distribution with time-varying mean value than to the Inv-Gamma or Inv-Wishart distribution found by Kullback–Leibler divergence. Therefore, we assumed the prior distribution of measurement noise covariance matrices as Gaussian, and calculated the Gaussian parameters by the black box variational inference method. Finally, we obtained the measurement noise covariance matrices by using the Gaussian parameters. The experimental results illustrate that the proposed algorithm performs better in resisting time-varying measurement noise than the existing Variational Bayesian adaptive filter.

Monte Carlo co-ordinate ascent variational inference

In variational inference (VI), coordinate-ascent and gradient-based approaches are two major types of algorithms for approximating difficult-to-compute probability densities. In real-world implementations of complex models, Monte Carlo methods are widely used to estimate expectations in coordinate-ascent approaches and gradients in derivative-driven ones. We discuss a Monte Carlo co-ordinate ascent VI (MC-CAVI) algorithm that makes use of Markov chain Monte Carlo (MCMC) methods in the calculation of expectations required within co-ordinate ascent VI (CAVI). We show that, under regularity conditions, an MC-CAVI recursion will get arbitrarily close to a maximiser of the evidence lower bound with any given high probability. In numerical examples, the performance of MC-CAVI algorithm is compared with that of MCMC and—as a representative of derivative-based VI methods—of Black Box VI (BBVI). We discuss and demonstrate MC-CAVI’s suitability for models with hard constraints in simulated and real examples. We compare MC-CAVI’s performance with that of MCMC in an important complex model used in nuclear magnetic resonance spectroscopy data analysis—BBVI is nearly impossible to be employed in this setting due to the hard constraints involved in the model.

Evaluating probabilistic programming and fast variational Bayesian inference in phylogenetics.

Recent advances in statistical machine learning techniques have led to the creation of probabilistic programming frameworks. These frameworks enable probabilistic models to be rapidly prototyped and fit to data using scalable approximation methods such as variational inference. In this work, we explore the use of the Stan language for probabilistic programming in application to phylogenetic models. We show that many commonly used phylogenetic models including the general time reversible substitution model, rate heterogeneity among sites, and a range of coalescent models can be implemented using a probabilistic programming language. The posterior probability distributions obtained via the black box variational inference engine in Stan were compared to those obtained with reference implementations of Markov chain Monte Carlo (MCMC) for phylogenetic inference. We find that black box variational inference in Stan is less accurate than MCMC methods for phylogenetic models, but requires far less compute time. Finally, we evaluate a custom implementation of mean-field variational inference on the Jukes–Cantor substitution model and show that a specialized implementation of variational inference can be two orders of magnitude faster and more accurate than a general purpose probabilistic implementation.

Black box variational inference to adaptive kalman filter with unknown process noise covariance matrix

Adaptive Kalman filter (AKF) is concerned with jointly estimating the system state and the unknown parameters of the state-space models. In this paper, we treat the model uncertainty of the process noise covariance matrix (PNCM) from black box variational inference (BBVI) perspective. In order to lay the foundation for research, we prove that the probabilistic model for online Bayesian inference of the system state and PNCM is non-conjugate, so the traditional coordinate-ascent variational inference (CAVI) cannot deal with this problem. To fill this gap, we propose an AKF in the presence of unknown PNCM based on the BBVI method (which is recently introduced to conduct the approximate Bayesian inference for the non-conjugate probabilistic model). Firstly, we introduce a structured posterior model of the system state and PNCM, by which the posterior distributions of the system state and the PNCM can be calculated efficiently. Then, the BBVI online inference for the posterior distribution of the PNCM is derived. In what follows, we use the intrinsically Bayesian robust KF (IBR-KF) to calculate the state posterior distribution. In addition, a special case, when the structure of the PNCM is known, is explored. Finally, numerical examples are provided to demonstrate the effectiveness of the proposed filters.

Tightening bounds for variational inference by revisiting perturbation theory**This original manuscript is based on the article presented at NIPS 2017: Bamler R, Zhang C, Opper M and Mandt S 2017 Perturbative black box variational inference Advances in Neural Information Processing Systems 30 ed I Guyon, U V Luxburg, S Bengio, H Wallach, R Fergus, S Vishwanathan and R Garnett (Red

Variational inference has become one of the most widely used methods in latent variable modeling. In its basic form, variational inference employs a fully factorized variational distribution and minimizes its Kullback–Leibler divergence to the posterior. As the minimization can only be carried out approximately, this approximation induces a bias. In this paper, we revisit perturbation theory as a powerful way of improving the variational approximation. Perturbation theory relies on a form of Taylor expansion of the log marginal likelihood, vaguely in terms of the log ratio of the true posterior and its variational approximation. While first order terms give the classical variational bound, higher-order terms yield corrections that tighten it. However, traditional perturbation theory does not provide a lower bound, making it inapt for stochastic optimization. In this paper, we derive a similar yet alternative way of deriving corrections to the evidence lower bound that resemble perturbation theory, but that result in a valid bound. We show in experiments on Gaussian processes and variational autoencoders that the new bounds are more mass covering, and that the resulting posterior covariances are closer to the true posterior and lead to higher likelihoods on held-out data.

Estimating Uncertainty of Work Zone Capacity using Neural Network Models

This study aims to develop a neural network model to predict work zone capacity including various uncertainties stemming from traffic and operational conditions. The neural network model is formulated in terms of the number of total lanes, number of open lanes, heavy vehicle percentage, work intensity, and work duration. The data used in this paper are obtained from previous studies published in open literature. To capture the uncertainty of work zone capacity, this paper provides two recent methods that enable neural network models to generate prediction intervals which are determined by mean work zone capacity and prediction standard error. The research first builds a Bayesian neural network model with the application of black-box variational inference (BBVI) technique. The second model is based on a regular artificial neural network with an application of the recently proposed Monte-Carlo dropout technique. Both of the neural network models construct prediction intervals under various confidence levels and provide the coverage rates of the actual work zone capacities. The statistical accuracy (MAPE, MAE, MSE, and RMSE) of the models is then compared with traditional estimation methods in predicted mean work zone capacity. BBVI produces better statistical results than the other three models. Both of the models provide predicted work zone capacity distribution and prediction intervals, whereas traditional models only provide a single estimate.