Gaussian Variational Approximation Research Articles

Variational Gaussian approximation for the magnetic Schrödinger equation *

In the present paper we consider the semiclassical magnetic Schrödinger equation, which describes the dynamics of particles under the influence of a magnetic field. The solution of the time-dependent Schrödinger equation is approximated by a single Gaussian wave packet via the time-dependent Dirac–Frenkel variational principle. For the approximation we derive ordinary differential equations of motion for the parameters of the variational solution. Moreover, we prove L 2-error bounds and observable error bounds for the approximating Gaussian wave packet.

R-VGAL: a sequential variational Bayes algorithm for generalised linear mixed models

Models with random effects, such as generalised linear mixed models (GLMMs), are often used for analysing clustered data. Parameter inference with these models is difficult because of the presence of cluster-specific random effects, which must be integrated out when evaluating the likelihood function. Here, we propose a sequential variational Bayes algorithm, called Recursive Variational Gaussian Approximation for Latent variable models (R-VGAL), for estimating parameters in GLMMs. The R-VGAL algorithm operates on the data sequentially, requires only a single pass through the data, and can provide parameter updates as new data are collected without the need of re-processing the previous data. At each update, the R-VGAL algorithm requires the gradient and Hessian of a “partial” log-likelihood function evaluated at the new observation, which are generally not available in closed form for GLMMs. To circumvent this issue, we propose using an importance-sampling-based approach for estimating the gradient and Hessian via Fisher’s and Louis’ identities. We find that R-VGAL can be unstable when traversing the first few data points, but that this issue can be mitigated by introducing a damping factor in the initial steps of the algorithm. Through illustrations on both simulated and real datasets, we show that R-VGAL provides good approximations to posterior distributions, that it can be made robust through damping, and that it is computationally efficient.

A mixture of logistic skew-normal multinomial models

The logistic normal multinomial distribution is gaining interest in modelling microbiome data. It utilizes a hierarchical structure such that the observed counts conditional on the compositions are assumed to be multinomial random variables and the log-ratio transformed compositions are assumed to be from a Gaussian distribution. While multinomial distribution accounts for the compositional nature of the data, and a Gaussian prior offers flexibility in the structure of covariance matrices, the log-ratio transformed compositions of the microbiome data can be highly skewed, especially at a lower taxonomic level. Thus, a Gaussian distribution may not be an ideal prior for the log-ratio transformed compositions. A novel mixture of logistic skew-normal multinomial (LSNM) distribution is proposed in which a multivariate skew-normal distribution is utilized as a prior for the log-ratio transformed compositions. A variational Gaussian approximation in conjunction with the EM algorithm is utilized for parameter estimation.

Generalized functional linear model with a point process predictor.

Point process data have become increasingly popular these days. For example, many of the data captured in electronic health records (EHR) are in the format of point process data. It is of great interest to study the association between a point process predictor and a scalar response using generalized functional linear regression models. Various generalized functional linear regression models have been developed under different settings in the past decades. However, existing methods can only deal with functional or longitudinal predictors, not point process predictors. In this article, we propose a novel generalized functional linear regression model for a point process predictor. Our proposed model is based on the joint modeling framework, where we adopt a log-Gaussian Cox process model for the point process predictor and a generalized linear regression model for the outcome. We also develop a new algorithm for fast model estimation based on the Gaussian variational approximation method. We conduct extensive simulation studies to evaluate the performance of our proposed method and compare it to competing methods. The performance of our proposed method is further demonstrated on an EHR dataset of patients admitted into the intensive care units of the Beth Israel Deaconess Medical Center between 2001 and 2008.

Clustering microbiome data using mixtures of logistic normal multinomial models

Discrete data such as counts of microbiome taxa resulting from next-generation sequencing are routinely encountered in bioinformatics. Taxa count data in microbiome studies are typically high-dimensional, over-dispersed, and can only reveal relative abundance therefore being treated as compositional. Analyzing compositional data presents many challenges because they are restricted to a simplex. In a logistic normal multinomial model, the relative abundance is mapped from a simplex to a latent variable that exists on the real Euclidean space using the additive log-ratio transformation. While a logistic normal multinomial approach brings flexibility for modeling the data, it comes with a heavy computational cost as the parameter estimation typically relies on Bayesian techniques. In this paper, we develop a novel mixture of logistic normal multinomial models for clustering microbiome data. Additionally, we utilize an efficient framework for parameter estimation using variational Gaussian approximations (VGA). Adopting a variational Gaussian approximation for the posterior of the latent variable reduces the computational overhead substantially. The proposed method is illustrated on simulated and real datasets.

High-order geometric integrators for the variational Gaussian approximation.

Among the single-trajectory Gaussian-based methods for solving the time-dependent Schrödinger equation, the variational Gaussian approximation is the most accurate one. In contrast to Heller's original thawed Gaussian approximation, it is symplectic, conserves energy exactly, and may partially account for tunneling. However, the variational method is also much more expensive. To improve its efficiency, we symmetrically compose the second-order symplectic integrator of Faou and Lubich and obtain geometric integrators that can achieve an arbitrary even order of convergence in the time step. We demonstrate that the high-order integrators can speed up convergence drastically compared to the second-order algorithm and, in contrast to the popular fourth-order Runge-Kutta method, are time-reversible and conserve the normand the symplectic structure exactly, regardless of the time step. To show that the method is not restricted to low-dimensional systems, we perform most of the analysis on a non-separable twenty-dimensional model of coupled Morse oscillators. We also show that the variational method may capture tunneling and, in general, improves accuracy over the non-variational thawed Gaussian approximation.

Gaussian Variational Approximations for High-dimensional State Space Models

We consider a Gaussian variational approximation of the posterior density in high-dimensional state space models. The number of parameters in the covariance matrix of the variational approximation grows as the square of the number of model parameters, so it is necessary to find simple yet effective parametrisations of the covariance structure when the number of model parameters is large. We approximate the joint posterior density of the state vectors by a dynamic factor model, having Markovian time dependence and a factor covariance structure for the states. This gives a reduced description of the dependence structure for the states, as well as a temporal conditional independence structure similar to that in the true posterior. We illustrate the methodology on two examples. The first is a spatio-temporal model for the spread of the Eurasian collared-dove across North America. Our approach compares favorably to a recently proposed ensemble Kalman filter method for approximate inference in high-dimensional hierarchical spatio-temporal models. Our second example is a Wishart-based multivariate stochastic volatility model for financial returns, which is outside the class of models the ensemble Kalman filter method can handle.

The limited-memory recursive variational Gaussian approximation (L-RVGA)

We consider the problem of computing a Gaussian approximation to the posterior distribution of a parameter given a large number N of observations and a Gaussian prior when the dimension of the parameter d is also large. To address this problem we build on a recently introduced recursive algorithm for variational Gaussian approximation of the posterior, called recursive variational Gaussian approximation (RVGA), which is a single pass algorithm, free of parameter tuning. In this paper, we consider the case where the parameter dimension d is high, and we propose a novel version of RVGA that scales linearly in the dimension d (as well as in the number of observations N), and which only requires linear storage capacity in d. This is afforded by using a novel recursive expectation maximization (EM) algorithm applied for factor analysis introduced herein, to approximate at each step the covariance matrix of the Gaussian distribution conveying the uncertainty in the parameter. The approach is successfully illustrated on the problems of high dimensional least-squares and logistic regression, and generalized to a large class of nonlinear models.

Dynamics of spin-tensor-momentum coupled Bose-Einstein condensates

We investigate the dynamics of the plane wave state in one-dimensional spin-tensor-momentum coupled Bose-Einstein condensate. By using the Gaussian variational approximation, we first derive the equations of motion for the variational parameters, including the center-of-mass coordinate, momentum, amplitude, width, chirp, and relative phase. These variational parameters are coupled together nonlinearly by the spin-tensor-momentum coupling, Raman coupling, and the spin-dependent atomic interaction. By minimizing the energy with respect to the variational parameters, we find that the ground state is a biaxial nematic state, the momentum of the ground state decreases monotonically with the increase of the strength of the Raman coupling, and the parity of real part of the ground-state wave function is opposite to that of the imaginary part. The linear stability analysis shows that the ground state is dynamically stable under a perturbation, and exhibits three different oscillation excitation modes, the frequencies of which are related to the strength of the Raman coupling, the aspect ratio of the harmonic trap, and the strength of the atomic interaction. By solving the equations of motion for the variational parameters, we find that the system displays periodical oscillation in the dynamical evolution. These variational results are also confirmed by the direct numerical simulations of the Gross-Pitaevskii equations, and these findings reveal the unique properties given by the spin-tensor-momentum coupling.

Explicit Gaussian Variational Approximation for the Poisson Lognormal Mixed Model

In recent years, the Poisson lognormal mixed model has been frequently used in modeling count data because it can accommodate both the over-dispersion of the data and the existence of within-subject correlation. Since the likelihood function of this model is expressed in terms of an intractable integral, estimating the parameters and obtaining inference for the parameters are challenging problems. Some approximation procedures have been proposed in the literature; however, they are computationally intensive. Moreover, the existing studies of approximate parameter inference using the Gaussian variational approximation method are usually restricted to models with only one predictor. In this paper, we consider the Poisson lognormal mixed model with more than one predictor. By extending the Gaussian variational approximation method, we derive explicit forms for the estimators of the parameters and examine their properties, including the asymptotic distributions of the estimators of the parameters. Accurate inference for the parameters is also obtained. A real-life example demonstrates the applicability of the proposed method, and simulation studies illustrate the accuracy of the proposed method.

Gaussian trajectory description of fragmentation in an isolated spinor condensate

Spin-1 Bose gases quenched to spin degeneracy exhibit fragmentation: the appearance of a condensate in more than one single-particle state. Due to its highly entangled nature, the dynamics leading to this collective state are beyond the scope of a Gaussian variational approximation of the many-body wave function. Here, we improve the performance of the Gaussian variational ansatz by considering dissipation into a fictitious environment, effectively suppressing entanglement within individual quantum trajectories at the expense of introducing a classical mixture of states. We find that this quantum trajectory approach captures the dynamical formation of a fragmented condensate and analyze how much dissipation should be added to the experiment in order to keep a single realization in a nonfragmented state.

The recursive variational Gaussian approximation (R-VGA)

We consider the problem of computing a Gaussian approximation to the posterior distribution of a parameter given N observations and a Gaussian prior. Owing to the need of processing large sample sizes N, a variety of approximate tractable methods revolving around online learning have flourished over the past decades. In the present work, we propose to use variational inference to compute a Gaussian approximation to the posterior through a single pass over the data. Our algorithm is a recursive version of variational Gaussian approximation we have called recursive variational Gaussian approximation. We start from the prior, and for each observation, we compute the nearest Gaussian approximation in the sense of Kullback–Leibler divergence to the posterior given this observation. In turn, this approximation is considered as the new prior when incorporating the next observation. This recursive version based on a sequence of optimal Gaussian approximations leads to a novel implicit update scheme which resembles the online Newton algorithm and which is shown to boil down to the Kalman filter for Bayesian linear regression. In the context of Bayesian logistic regression, the implicit scheme may be solved, and the algorithm is shown to perform better than the extended Kalman filter, while being less computationally demanding than its sampling counterparts.

Flexible and Efficient Inference with Particles for the Variational Gaussian Approximation.

Variational inference is a powerful framework, used to approximate intractable posteriors through variational distributions. The de facto standard is to rely on Gaussian variational families, which come with numerous advantages: they are easy to sample from, simple to parametrize, and many expectations are known in closed-form or readily computed by quadrature. In this paper, we view the Gaussian variational approximation problem through the lens of gradient flows. We introduce a flexible and efficient algorithm based on a linear flow leading to a particle-based approximation. We prove that, with a sufficient number of particles, our algorithm converges linearly to the exact solution for Gaussian targets, and a low-rank approximation otherwise. In addition to the theoretical analysis, we show, on a set of synthetic and real-world high-dimensional problems, that our algorithm outperforms existing methods with Gaussian targets while performing on a par with non-Gaussian targets.

Manifold Optimization-Assisted Gaussian Variational Approximation

Gaussian variational approximation is a popular methodology to approximate posterior distributions in Bayesian inference, especially in high-dimensional and large data settings. To control the computational cost, while being able to capture the correlations among the variables, the low rank plus diagonal structure was introduced in the previous literature for the Gaussian covariance matrix. For a specific Bayesian learning task, the uniqueness of the solution is usually ensured by imposing stringent constraints on the parameterized covariance matrix, which could break down during the optimization process. In this article, we consider two special covariance structures by applying the Stiefel manifold and Grassmann manifold constraints, to address the optimization difficulty in such factorization architectures. To speed up the updating process with minimum hyperparameter-tuning efforts, we design two new schemes of Riemannian stochastic gradient descent methods and compare them with other existing methods of optimizing on manifolds. In addition to fixing the identification issue, results from both simulation and empirical experiments prove the ability of the proposed methods of obtaining competitive accuracy and comparable converge speed in both high-dimensional and large-scale learning tasks. Supplementary materials for this article are available online.

Use of Model Reparametrization to Improve Variational Bayes

AbstractWe propose using model reparametrization to improve variational Bayes inference for hierarchical models whose variables can be classified as global (shared across observations) or local (observation-specific). Posterior dependence between local and global variables is minimized by applying an invertible affine transformation on the local variables. The functional form of this transformation is deduced by approximating the posterior distribution of each local variable conditional on the global variables by a Gaussian density via a second order Taylor expansion. Variational Bayes inference for the reparametrized model is then obtained using stochastic approximation. Our approach can be readily extended to large datasets via a divide and recombine strategy. Using generalized linear mixed models, we demonstrate that reparametrized variational Bayes (RVB) provides improvements in both accuracy and convergence rate compared to state of the art Gaussian variational approximation methods.

Conditionally structured variational Gaussian approximation with importance weights

We develop flexible methods of deriving variational inference for models with complex latent variable structure. By splitting the variables in these models into “global” parameters and “local” latent variables, we define a class of variational approximations that exploit this partitioning and go beyond Gaussian variational approximation. This approximation is motivated by the fact that in many hierarchical models, there are global variance parameters which determine the scale of local latent variables in their posterior conditional on the global parameters. We also consider parsimonious parametrizations by using conditional independence structure and improved estimation of the log marginal likelihood and variational density using importance weights. These methods are shown to improve significantly on Gaussian variational approximation methods for a similar computational cost. Application of the methodology is illustrated using generalized linear mixed models and state space models.

Bayesian Deep Net GLM and GLMM

Deep feedforward neural networks (DFNNs) are a powerful tool for functional approximation. We describe flexible versions of generalized linear and generalized linear mixed models incorporating basis functions formed by a DFNN. The consideration of neural networks with random effects is not widely used in the literature, perhaps because of the computational challenges of incorporating subject specific parameters into already complex models. Efficient computational methods for high-dimensional Bayesian inference are developed using Gaussian variational approximation, with a parsimonious but flexible factor parameterization of the covariance matrix. We implement natural gradient methods for the optimization, exploiting the factor structure of the variational covariance matrix in computation of the natural gradient. Our flexible DFNN models and Bayesian inference approach lead to a regression and classification method that has a high prediction accuracy, and is able to quantify the prediction uncertainty in a principled and convenient way. We also describe how to perform variable selection in our deep learning method. The proposed methods are illustrated in a wide range of simulated and real-data examples, and compare favorably to a state of the art flexible regression and classification method in the statistical literature, the Bayesian additive regression trees (BART) method. User-friendly software packages in Matlab, R, and Python implementing the proposed methods are available at https://github.com/VBayesLab.

Magnetocaloric effect and Grüneisen parameter of quantum magnets with a spin gap

We obtained, within a path-integral formalism (mean-field variational Gaussian approximation), analytical expressions for thermodynamic quantities such as magnetization, heat capacity, and the magnetic Gr\"uneisen parameter ${\mathrm{\ensuremath{\Gamma}}}_{H}$ of the system of triplons in spin gapped quantum magnets. Near the critical temperature ${\mathrm{\ensuremath{\Gamma}}}_{H}$ is discontinuous and changes its sign upon the Bose-Einstein condensation (BEC) of triplons. We predict that in the limit of low temperature $T$ and near the critical magnetic field ${H}_{c},\phantom{\rule{0.16em}{0ex}}{\mathrm{\ensuremath{\Gamma}}}_{H}$ diverges as ${\mathrm{\ensuremath{\Gamma}}}_{H}\ensuremath{\sim}1/{T}^{2}$, while it scales as ${\mathrm{\ensuremath{\Gamma}}}_{H}\ensuremath{\sim}1/(H\ensuremath{-}{H}_{c})$ as the magnetic field approaches ${H}_{c}$.

Gaussian Variational Approximation With a Factor Covariance Structure

ABSTRACTVariational approximations have the potential to scale Bayesian computations to large datasets and highly parameterized models. Gaussian approximations are popular, but can be computationally burdensome when an unrestricted covariance matrix is employed and the dimension of the model parameter is high. To circumvent this problem, we consider a factor covariance structure as a parsimonious representation. General stochastic gradient ascent methods are described for efficient implementation, with gradient estimates obtained using the so-called “reparameterization trick.” The end result is a flexible and efficient approach to high-dimensional Gaussian variational approximation. We illustrate using robust P-spline regression and logistic regression models. For the latter, we consider eight real datasets, including datasets with many more covariates than observations, and another with mixed effects. In all cases, our variational method provides fast and accurate estimates. Supplementary material for this article is available online.

Variational Gaussian approximation for Poisson data

The Poisson model is frequently employed to describe count data, but in a Bayesian context it leads to an analytically intractable posterior probability distribution. In this work, we analyze a variational Gaussian approximation to the posterior distribution arising from the Poisson model with a Gaussian prior. This is achieved by seeking an optimal Gaussian distribution minimizing the Kullback–Leibler divergence from the posterior distribution to the approximation, or equivalently maximizing the lower bound for the model evidence. We derive an explicit expression for the lower bound, and show the existence and uniqueness of the optimal Gaussian approximation. The lower bound functional can be viewed as a variant of classical Tikhonov regularization that penalizes also the covariance. Then we develop an efficient alternating direction maximization algorithm for solving the optimization problem, and analyze its convergence. We discuss strategies for reducing the computational complexity via low rank structure of the forward operator and the sparsity of the covariance. Further, as an application of the lower bound, we discuss hierarchical Bayesian modeling for selecting the hyperparameter in the prior distribution, and propose a monotonically convergent algorithm for determining the hyperparameter. We present extensive numerical experiments to illustrate the Gaussian approximation and the algorithms.