Variational Approximation Research Articles

Variational inference for acceleration of SN Ia photometric distance estimation with BayeSN

Abstract Type Ia supernovae (SNe Ia) are standarizable candles whose observed light curves can be used to infer their distances, which can in turn be used in cosmological analyses. As the quantity of observed SNe Ia grows with current and upcoming surveys, increasingly scalable analyses are necessary to take full advantage of these new datasets for precise estimation of cosmological parameters. Bayesian inference methods enable fitting SN Ia light curves with robust uncertainty quantification, but traditional posterior sampling using Markov Chain Monte Carlo (MCMC) is computationally expensive. We present an implementation of variational inference (VI) to accelerate the fitting of SN Ia light curves using the BayeSN hierarchical Bayesian model for time-varying SN Ia spectral energy distributions (SEDs). We demonstrate and evaluate its performance on both simulated light curves and data from the Foundation Supernova Survey with two different forms of surrogate posterior – a multivariate normal and a custom multivariate zero-lower-truncated normal distribution – and compare them with the Laplace Approximation and full MCMC analysis. To validate of our variational approximation, we calculate the pareto-smoothed importance sampling (PSIS) diagnostic, and perform variational simulation-based calibration (VSBC). The VI approximation achieves similar results to MCMC but with an order-of-magnitude speedup for the inference of the photometric distance moduli. Overall, we show that VI is a promising method for scalable parameter inference that enables analysis of larger datasets for precision cosmology.

Bayesian Empirical Likelihood Regression for Semiparametric Estimation of Optimal Dynamic Treatment Regimes.

We propose a semiparametric approach to Bayesian modeling of dynamic treatment regimes that is built on a Bayesian likelihood-based regression estimation framework. Methods based on this framework exhibit a probabilistic coherence property that leads to accurate estimation of the optimal dynamic treatment regime. Unlike most Bayesian estimation methods, our proposed method avoids strong distributional assumptions for the intermediate and final outcomes by utilizing empirical likelihoods. Our proposed method allows for either linear, or more flexible forms of mean functions for the stagewise outcomes. A variational Bayes approximation is used for computation to avoid common pitfalls associated with Markov Chain Monte Carlo approaches coupled with empirical likelihood. Through simulations and analysis of the STAR*D sequential randomized trial data, our proposed method demonstrates superior accuracy over Q-learning and parametric Bayesian likelihood-based regression estimation, particularly when the parametric assumptions of regression error distributions may be potentially violated.

The energy-frequency diagram of the (1+1)-dimensional Φ4 oscillon

Two different methods are used to study the existence and stability of the (1+1)-dimensional Φ4 oscillon. The variational technique approximates it by a periodic function with a set of adiabatically changing parameters. An alternative approach treats oscillons as standing waves in a finite-size box; these are sought as solutions of a boundary-value problem on a two-dimensional domain. The numerical analysis reveals that the standing wave’s energy-frequency diagram is fragmented into disjoint segments with ωn+1< ω < ωn, where ωn = ω0/(n + 1), n = 0, 1, 2, ., and ω0 is the endpoint of the continuous spectrum (mass threshold of the model). The variational approximation involving the first, zeroth and second harmonic components provides an accurate description of the oscillon with the frequency in (ω1, ω0), but breaks down as ω falls out of that interval.

Variational Inference based on a Subclass of Closed Skew Normals

Gaussian distributions are widely used in Bayesian variational inference to approximate intractable posterior densities, but the ability to accommodate skewness can improve approximation accuracy significantly, when data or prior information is scarce. We study the properties of a subclass of closed skew normals constructed using affine transformation of independent standardized univariate skew normals as the variational density, and illustrate how it provides increased flexibility and accuracy in approximating the joint posterior in various applications, by overcoming limitations in existing skew normal variational approximations. The evidence lower bound is optimized using stochastic gradient ascent, where analytic natural gradient updates are derived. We also demonstrate how problems in maximum likelihood estimation of skew normal parameters occur similarly in stochastic variational inference, and can be resolved using the centered parameterization. Supplementary materials for this article are available online.

Probabilistic Quantile Factor Analysis

This article extends quantile factor analysis to a probabilistic variant that incorporates regularization and computationally efficient variational approximations. We establish through synthetic and real data experiments that the proposed estimator can, in many cases, achieve better accuracy than a recently proposed loss-based estimator. We contribute to the factor analysis literature by extracting new indexes of low, medium, and high economic policy uncertainty, as well as loose, median, and tight financial conditions. We show that the high uncertainty and tight financial conditions indexes have superior predictive ability for various measures of economic activity. In a high-dimensional exercise involving about 1000 daily financial series, we find that quantile factors also provide superior out-of-sample information compared to mean or median factors.

Model‐based clustering in simple hypergraphs through a stochastic blockmodel

AbstractWe propose a model to address the overlooked problem of node clustering in simple hypergraphs. Simple hypergraphs are suitable when a node may not appear multiple times in the same hyperedge, such as in co‐authorship datasets. Our model generalizes the stochastic blockmodel for graphs and assumes the existence of latent node groups and hyperedges are conditionally independent given these groups. We first establish the generic identifiability of the model parameters. We then develop a variational approximation Expectation‐Maximization algorithm for parameter inference and node clustering, and derive a statistical criterion for model selection. To illustrate the performance of our R package HyperSBM, we compare it with other node clustering methods using synthetic data generated from the model, as well as from a line clustering experiment and a co‐authorship dataset.

Guaranteed matrix recovery using weighted nuclear norm plus weighted total variation minimization

This work presents a general framework regarding the recovery of matrices equipped with hybrid low-rank and local-smooth properties from just a few measurements consisting of linear combinations of the matrix entries. Concretely, we consider the problem of robust low-rank matrix recovery using Weighted Nuclear Norm plus Weight Total Variation (WNNWTV) minimization. First of all, based on a new restricted isometry property, we prove that the WNNWTV method possesses an error bound consisting of a low-rank approximation term, a total variation approximation term, and an observation error term. It should be noted that although there are many models considering both properties, there are very few recoverable error theories about such models. Specifically, the theoretical error bound provides an automatic mechanism to reducing regularization parameters with no need for cross-validation while keeping almost the same selection result with commonly used cross-validation technique. Subsequently, the proposed method is reformulated into a regularized unconstrained problem, and we study its optimization aspects in detail based on the Alternating Direction Method of Multipliers (ADMM). Extensive experiments on synthetic data and two applications, i.e. hyperspectral image recovery and dynamic magnetic resonance imaging recovery verified our theories and proposed algorithms.

Variational Bayesian Approximation (VBA): Implementation and Comparison of Different Optimization Algorithms.

In any Bayesian computations, the first step is to derive the joint distribution of all the unknown variables given the observed data. Then, we have to do the computations. There are four general methods for performing computations: Joint MAP optimization; Posterior expectation computations that require integration methods; Sampling-based methods, such as MCMC, slice sampling, nested sampling, etc., for generating samples and numerically computing expectations; and finally, Variational Bayesian Approximation (VBA). In this last method, which is the focus of this paper, the objective is to search for an approximation for the joint posterior with a simpler one that allows for analytical computations. The main tool in VBA is to use the Kullback-Leibler Divergence (KLD) as a criterion to obtain that approximation. Even if, theoretically, this can be conducted formally, for practical reasons, we consider the case where the joint distribution is in the exponential family, and so is its approximation. In this case, the KLD becomes a function of the usual parameters or the natural parameters of the exponential family, where the problem becomes parametric optimization. Thus, we compare four optimization algorithms: general alternate functional optimization; parametric gradient-based with the normal and natural parameters; and the natural gradient algorithm. We then study their relative performances on three examples to demonstrate the implementation of each algorithm and their efficiency performance.

Convergence of critical points for a phase-field approximation of 1D cohesive fracture energies

Variational models for cohesive fracture are based on the idea that the fracture energy is released gradually as the crack opening grows. Recently, [21] proposed a variational approximation via Γ\\documentclass[12pt]{minimal} \\usepackage{amsmath} \\usepackage{wasysym} \\usepackage{amsfonts} \\usepackage{amssymb} \\usepackage{amsbsy} \\usepackage{mathrsfs} \\usepackage{upgreek} \\setlength{\\oddsidemargin}{-69pt} \\begin{document}$$\\Gamma $$\\end{document}-convergence of a class of cohesive fracture energies by phase-field energies of Ambrosio-Tortorelli type, which may be also used as regularization for numerical simulations. In this paper we address the question of the asymptotic behaviour of critical points of the phase-field energies in the one-dimensional setting: we show that they converge to a selected class of critical points of the limit functional. Conversely, each critical point in this class can be approximated by a family of critical points of the phase-field functionals.

Dynamical system identification, model selection, and model uncertainty quantification by Bayesian inference.

This study presents a Bayesian maximum a posteriori (MAP) framework for dynamical system identification from time-series data. This is shown to be equivalent to a generalized Tikhonov regularization, providing a rational justification for the choice of the residual and regularization terms, respectively, from the negative logarithms of the likelihood and prior distributions. In addition to the estimation of model coefficients, the Bayesian interpretation gives access to the full apparatus for Bayesian inference, including the ranking of models, the quantification of model uncertainties, and the estimation of unknown (nuisance) hyperparameters. Two Bayesian algorithms, joint MAP and variational Bayesian approximation, are compared to the least absolute shrinkage and selection operator (LASSO), ridge regression, and the sparse identification of nonlinear dynamics (SINDy) algorithms for sparse regression by application to several dynamical systems with added Gaussian or Laplace noise. For multivariate Gaussian likelihood and prior distributions, the Bayesian formulation gives Gaussian posterior and evidence distributions, in which the numerator terms can be expressed in terms of the Mahalanobis distance or "Gaussian norm" ||y-y^||M-12=(y-y^)⊤M-1(y-y^), where y is a vector variable, y^ is its estimator, and M is the covariance matrix. The posterior Gaussian norm is shown to provide a robust metric for quantitative model selection for the different systems and noise models examined.

Mixtures of logistic normal multinomial regression models for microbiome data

In the realm of bioinformatics, we frequently encounter discrete data, particularly microbiome taxa count data obtained through 16S rRNA sequencing. These microbiome datasets are commonly characterized by their high dimensionality and the ability to provide insights solely into relative abundance, necessitating their classification as compositional data. Analyzing such data presents challenges due to their confinement within a simplex. Additionally, microbiome taxa counts are subject to influence by various biological and environmental factors like age, gender, and diet. Thus, we have developed a novel approach involving regression-based mixtures of logistic normal multinomial models for clustering microbiome data. These models effectively categorize samples into more homogeneous subpopulations, enabling the exploration of relationships between bacterial abundance and biological or environmental covariates within each identified group. To enhance the accuracy and efficiency of parameter estimation, we employ a robust framework based on variational Gaussian approximations (VGA). Our proposed method's effectiveness is demonstrated through its application to simulated and real datasets.

Theoretical solution of vortex splitting due to islands effect on the vortex

Islands, acting as transitional areas between land and sea, significantly influence their surrounding environments. Studying the changes in vortex structure resulting from the interaction between islands and mesoscale vortices is crucial for understanding the dynamic characteristics and ecological processes of the marine environment. Previous theoretical studies have shown that in a domain without boundaries, due to the conservation of angular momentum, a vortex cannot split by itself. This paper establishes the conditions for the splitting of an anticyclonic vortex when it collides with two square islands. By linking the initial and final states of islands effect on the vortex and utilizing conserved quantities such as angular momentum and mass, along with the slow variation approximation, a nonlinear theoretical solution is constructed. The analysis shows that when the interaction between the vortex and the two islands leads to vortex splitting, the length of the islands, the vortex radius, and the distance between the islands must satisfy a certain condition O(LR)∼O(wR)∼O(1). These results provide support for subsequent analyses of the impact of various parameters on vortex structure when the North Brazil Current (NBC) ring encounters the Lesser Antilles in the tropical western Atlantic.

Bessel vortices in spin-1 Bose–Einstein condensates with Zeeman splitting and spin–orbit coupling

Abstract We investigate the ground states of spin–orbit coupled spin-1 Bose–Einstein condensates in the presence of Zeeman splitting. By introducing the generalized momentum operator, the linear version of the system is solved exactly, yielding a set of Bessel vortices. Additionally, based on linear solution and using variational approximation, the solutions for the full nonlinear system and their ground state phase diagrams are derived, including the vortex states with quantum numbers m = 0, 1, as well as mixed states. In this work, mixed states in spin-1 spin–orbit coupling (SOC) BEC are interpreted for the first time as weighted superpositions of three vortex states. Furthermore, the results also indicate that under strong Zeeman splitting, the system cannot form localized states. The variational solutions align well with numerical simulations, showing stable evolution and meeting the criteria for long-term observation in experiments.

Hierarchical mixture of discriminative Generalized Dirichlet classifiers

This paper presents a discriminative classifier for compositional data. This classifier is based on the posterior distribution of the Generalized Dirichlet which is the discriminative counterpart of Generalized Dirichlet mixture model. Moreover, following the mixture of experts paradigm, we proposed a hierarchical mixture of this classifier. In order to learn the models parameters, we use a variational approximation by deriving an upper-bound for the Generalized Dirichlet mixture. To the best of our knownledge, this is the first time this bound is proposed in the literature. Experimental results are presented for spam detection and color space identification.

Ultra-Efficient MCMC for Bayesian Longitudinal Functional Data Analysis

Functional mixed models are widely useful for regression analysis with dependent functional data, including longitudinal functional data with scalar predictors. However, existing algorithms for Bayesian inference with these models only provide either scalable computing or accurate approximations to the posterior distribution, but not both. We introduce a new MCMC sampling strategy for highly efficient and fully Bayesian regression with longitudinal functional data. Using a novel blocking structure paired with an orthogonalized basis reparameterization, our algorithm jointly samples the fixed effects regression functions together with all subject- and replicate-specific random effects functions. Crucially, the joint sampler optimizes sampling efficiency for these key parameters while preserving computational scalability. Perhaps surprisingly, our new MCMC sampling algorithm even surpasses state-of-the-art algorithms for frequentist estimation and variational Bayes approximations for functional mixed models—while also providing accurate posterior uncertainty quantification—and is orders of magnitude faster than existing Gibbs samplers. Simulation studies show improved point estimation and interval coverage in nearly all simulation settings over competing approaches. We apply our method to a large physical activity dataset to study how various demographic and health factors associate with intraday activity. Supplementary materials for this article are available online.

Variational Gaussian approximation for the magnetic Schrödinger equation *

In the present paper we consider the semiclassical magnetic Schrödinger equation, which describes the dynamics of particles under the influence of a magnetic field. The solution of the time-dependent Schrödinger equation is approximated by a single Gaussian wave packet via the time-dependent Dirac–Frenkel variational principle. For the approximation we derive ordinary differential equations of motion for the parameters of the variational solution. Moreover, we prove L 2-error bounds and observable error bounds for the approximating Gaussian wave packet.

Mixed effect estimation in deep compartment models: Variational methods outperform first-order approximations.

This work focusses on extending the deep compartment model (DCM) framework to the estimation of mixed-effects. By introducing random effects, model predictions can be personalized based on drug measurements, enabling the testing of different treatment schedules on an individual basis. The performance of classical first-order (FO and FOCE) and machine learning based variational inference (VI) algorithms were compared in a simulation study. In VI, posterior distributions of the random variables are approximated using variational distributions whose parameters can be directly optimized. We found that variational approximations estimated using the path derivative gradient estimator version of VI were highly accurate. Models fit on the simulated data set using the FO and VI objective functions gave similar results, with accurate predictions of both the population parameters and covariate effects. Contrastingly, models fit using FOCE depicted erratic behaviour during optimization, and resulting parameter estimates were inaccurate. Finally, we compared the performance of the methods on two real-world data sets of haemophilia A patients who received standard half-life factor VIII concentrates during prophylactic and perioperative settings. Again, models fit using FO and VI depicted similar results, although some models fit using FO presented divergent results. Again, models fit using FOCE were unstable. In conclusion, we show that mixed-effects estimation using the DCM is feasible. VI performs conditional estimation, which might lead to more accurate results in more complex models compared to the FO method.

Motorized chain models of the ideal chromosome

An array of motor proteins consumes chemical energy in setting up the architectures of chromosomes. Here, we explore how the structure of ideal polymer chains is influenced by two classes of motors. The first class which we call "swimming motors" acts to propel the chromatin fiber through three-dimensional space. They represent a caricature of motors such as RNA polymerases. Previously, they have often been described by adding a persistent flow onto Brownian diffusion of the chain. The second class of motors, which we call "grappling motors" caricatures the loop extrusion processes in which segments of chromatin fibers some distance apart are brought together. We analyze these models using a self-consistent variational phonon approximation to a many-body Master equation incorporating motor activities. We show that whether the swimming motors lead to contraction or expansion depends on the susceptibility of the motors, that is, how their activity depends on the forces they must exert. Grappling motors in contrast to swimming motors lead to long-ranged correlations that resemble those first suggested for fractal globules and that are consistent with the effective interactions inferred by energy landscape analyses of Hi-C data on the interphase chromosome.

On the Temporal Tweezing of Cavity Solitons

Motivated by the work of Jang et al., Nat Commun 6:7370 (2015), where the authors experimentally tweeze cavity solitons in a passive loop of optical fiber, we study the amenability to tweezing of cavity solitons as the properties of a localized tweezer are varied. The system is modeled by the Lugiato-Lefever equation, a variant of the complex Ginzburg-Landau equation. We produce an effective, localized, trapping tweezer potential by assuming a Gaussian phase-modulation of the holding beam. The potential for tweezing is then assessed as the total (temporal) displacement and speed of the tweezer are varied, and corresponding phase diagrams are presented. As the relative speed of the tweezer is increased we find two possible dynamical scenarios: successful tweezing and release of the cavity soliton. We also deploy a non-conservative variational approximation (NCVA) based on a Lagrangian description which reduces the original dissipative partial differential equation to a set of coupled ordinary differential equations for the cavity soliton parameters. We illustrate the ability of the NCVA to accurately predict the separatrix between successful and failed tweezing. This showcases the versatility of the NCVA to provide a low-dimensional description of the experimental realization of the temporal tweezing.

Passive underwater tracking with unknown measurement noise statistics using variational Bayesian approximation

This paper considers a bearings-only tracking problem with unknown measurement noise statistics. It is assumed that the measurement noise follows a Gaussian probability density function where the mean and the covariance of the noise are unknown. Here, an adaptive nonlinear filtering technique is proposed, where the joint distribution of the measurement noise mean and its covariance are considered to follow a normal inverse Wishart (NIW) distribution. Using the variational Bayesian (VB) approximation, joint distribution of the target state, the measurement noise mean and covariance is factorized as the product of their individual probability density function (pdf). Minimizing the Kullback-Leibler divergence (KLD) between the factorized and true joint pdfs, probability distributions of the noise mean, covariance and the target states are evaluated. The estimation of states with the proposed VB based method is compared with the maximum a posteriori (MAP) and the maximum likelihood estimation (MLE) based adaptive filtering. Deterministic sigma points are used to realize the filtering algorithms. The proposed adaptive filter with VB approximation is found to be more accurate compared to their corresponding MAP-MLE based counterparts.