Variational Inference Framework Research Articles

Extended variational inference for gamma mixture model in positive vectors modeling

Bayesian estimation of finite Gamma mixture model (GaMM) has attracted considerable attention recently due to its capability of modeling positive data. With conventional variational inference (VI) frameworks, we cannot derive an analytically tractable solution for the variational posterior, since the expectation of the joint distribution of all the random variables cannot be estimated in a closed form. Therefore, numerical techniques are commonly utilized to simulate the posterior distribution. However, the optimization process of these methods can be prohibitively slow for practical applications. In order to obtain closed-form solutions, some lower-bound approximations are then introduced into the evidence lower bound (ELBO), following the recently proposed extended variational inference (EVI) framework. The problem in numerical simulation can be overcome. In this paper, we address the Bayesian estimation of the finite Gamma mixture model (GaMM) under the EVI framework in a flexible way. Moreover, the optimal mixture component number can be automatically determined based on the observed data and the over-fitting problem related to the conventional expectation–maximization (EM) is overcome. We demonstrate the excellent performance of the proposed method with synthesized data and real data evaluations. In the real data evaluation, we compare the proposed method on object detection and image categorization tasks with referred methods and find statistically significant improvement on accuracies and runtime.

Joint Models for Event Prediction From Time Series and Survival Data

We present a nonparametric prognostic framework for individualized event prediction based on joint modeling of both time series and time-to-event data. Our approach exploits a multivariate Gaussian convolution process (MGCP) to model the evolution of time series signals and a Cox model to map time-to-event data with time series data modeled through the MGCP. Taking advantage of the unique structure imposed by convolved processes, we provide a variational inference framework to simultaneously estimate parameters in the joint MGCP-Cox model. This significantly reduces computational complexity and safeguards against model overfitting. Experiments on synthetic and real world data show that the proposed framework outperforms state-of-the art approaches built on two-stage inference and strong parametric assumptions. Technical details are available in the supplementary materials.

Model Refinement Learning and an Example on Channel Estimation With Universal Noise Model

Model-based method and data-based method are two basic approaches for the design of wireless communication systems. Model-based methods suffer from inaccurate modeling assumptions due to excessively complex environment. Recently, data-based methods have achieved remarkable performances in the communication system design without the knowledge of accurate model but encounter some challenges such as, lack of available labelled training data and explainability. In this paper, we propose a novel hybrid idea to integrate the strengths of both data and model-based methods, named model refinement learning , which is training affordable, theoretically interpretable and self-adapting. To show the idea more concretely, a novel channel estimation algorithm is proposed in the multiple-input single-output (MISO) system in the case where the noise model is unknown. In particular, we utilize a universal mixture of Gaussian (MoG) model, which can adaptively adjust the involved parameters to fit the true noise distribution by using observed data. We propose a novel variational inference framework to achieve automatical noise model refinement and design the corresponding online channel estimator. To reduce the online algorithm overhead, we propose a decoupled variational Bayesian method to achieve linear computational complexity. Simulations show that our proposed method outperforms both the model-based and data-based counterparts.

Variational Inference With Parameter Learning Applied to Vehicle Trajectory Estimation

We present parameter learning in a Gaussian variational inference setting using only noisy measurements (i.e., no groundtruth). This is demonstrated in the context of vehicle trajectory estimation, although the method we propose is general. The paper extends the Exactly Sparse Gaussian Variational Inference (ESGVI) framework, which has previously been used for large-scale nonlinear batch state estimation. Our contribution is to additionally learn parameters of our system models (which may be difficult to choose in practice) within the ESGVI framework. In this paper, we learn the covariances for the motion and sensor models used within vehicle trajectory estimation. Specifically, we learn the parameters of a white-noise-on-acceleration motion model and the parameters of an Inverse-Wishart prior over measurement covariances for our sensor model. We demonstrate our technique using a 36~km dataset consisting of a car using lidar to localize against a high-definition map; we learn the parameters on a training section of the data and then show that we achieve high-quality state estimates on a test section, even in the presence of outliers. Lastly, we show that our framework can be used to solve pose graph optimization even with many false loop closures.

Scalable Variational Bayesian Kernel Selection for Sparse Gaussian Process Regression

This paper presents a variational Bayesian kernel selection (VBKS) algorithm for sparse Gaussian process regression (SGPR) models. In contrast to existing GP kernel selection algorithms that aim to select only one kernel with the highest model evidence, our VBKS algorithm considers the kernel as a random variable and learns its belief from data such that the uncertainty of the kernel can be interpreted and exploited to avoid overconfident GP predictions. To achieve this, we represent the probabilistic kernel as an additional variational variable in a variational inference (VI) framework for SGPR models where its posterior belief is learned together with that of the other variational variables (i.e., inducing variables and kernel hyperparameters). In particular, we transform the discrete kernel belief into a continuous parametric distribution via reparameterization in order to apply VI. Though it is computationally challenging to jointly optimize a large number of hyperparameters due to many kernels being evaluated simultaneously by our VBKS algorithm, we show that the variational lower bound of the log-marginal likelihood can be decomposed into an additive form such that each additive term depends only on a disjoint subset of the variational variables and can thus be optimized independently. Stochastic optimization is then used to maximize the variational lower bound by iteratively improving the variational approximation of the exact posterior belief via stochastic gradient ascent, which incurs constant time per iteration and hence scales to big data. We empirically evaluate the performance of our VBKS algorithm on synthetic and massive real-world datasets.

Point set registration with mixture framework and variational inference

We propose a new point set registration method based on mixture framework and variational inference. A three-phase registration strategy (TRS) is proposed to automatically process point set registration problem in different cases. A Gaussian variational mixture model (GVMM) with isotropic and anisotropic components under the variational inference framework is designed to weaken the effect of outliers. The Dirichlet distribution is applied to govern the mixture proportion of Gaussian components and then distinguishes missing points. We test the performance of our method in contour registration, Graffiti images, retinal images, remote sensing images and 3D human motion, and compare with six state-of-the-art methods. Our method shows favorable performances in most scenarios.

LCBM: A Multi-View Probabilistic Model for Multi-Label Classification.

Multi-label classification is an important research topic in machine learning, for which exploiting label dependencies is an effective modeling principle. Recently, probabilistic models have shown great potential in discovering dependencies among labels. In this paper, motivated by the recent success of multi-view learning to improve the generalization performance, we propose a novel multi-view probabilistic model named latent conditional Bernoulli mixture (LCBM) for multi-label classification. LCBM is a generative model taking features from different views as inputs, and conditional on the latent subspace shared by the views a Bernoulli mixture model is adopted to build label dependencies. Inside each component of the mixture, the labels have a weak correlation which facilitates computational convenience. The mean field variational inference framework is used to carry out approximate posterior inference in the probabilistic model, where we propose a Gaussian mixture variational autoencoder (GMVAE) for effective posterior approximation. We further develop a scalable stochastic training algorithm for efficiently optimizing the model parameters and variational parameters, and derive an efficient prediction procedure based on greedy search. Experimental results on multiple benchmark datasets show that our approach outperforms other state-of-the-art methods under various metrics.

Bayes-Optimal MMSE Detector for Massive MIMO Relaying With Low-Precision ADCs/DACs

In this study, we focus on the detector design for a massive multiple-input multiple-output amplify-and-forward relaying system with low-precision analog-to-digital converters (ADCs) and digital-to-analog converters (DACs). A general relaying model with direct and relay links is considered. Several studies have been performed on the basis of simple but performance-limited linear transceivers. The optimal detector for the aforementioned setting remains unknown. To bridge this gap, we develop a minimum mean squared error detector based on the framework of variational Bayes inference. The proposed detector works similarly to the iterative turbo update, and the extrinsic information of the direct and relay links are updated through maximum ratio combining. Furthermore, our algorithm uses a partially parallel update schedule that improves the convergence compared with an intuitive method that is based on a fully parallel schedule. We then present the state evolution (SE) as an analytical framework to investigate the mean squared error and symbol error rate of the detector. By specifying the dynamics of the SE, we reveal that low-precision ADCs/DACs, that is, those using 2-4-bit quantization, realize a good tradeoff between performance and cost. Moreover, low-precision ADCs/DACs provide numerous insights into the relaying system and its design, such as the quantization step size.

Latent Question Interpretation Through Variational Adaptation

Most artificial neural network models for question-answering rely on complex attention mechanisms. These techniques demonstrate high performance on existing datasets; however, they are limited in their ability to capture natural language variability, and to generate diverse relevant answers. To address this limitation, we propose a model that learns multiple interpretations of a given question. This diversity is ensured by our interpretation policy module which automatically adapts the parameters of a question-answering model with respect to a discrete latent variable. This variable follows the distribution of interpretations learned by the interpretation policy through a semi-supervised variational inference framework. To boost the performance further, the resulting policy is fine-tuned using the rewards from the answer accuracy with a policy gradient. We demonstrate the relevance and efficiency of our model through a large panel of experiments. Qualitative results, in particular, underline the ability of the proposed architecture to discover multiple interpretations of a question. When tested using the Stanford Question Answering Dataset 1.1, our model outperforms the baseline methods in finding multiple and diverse answers. To assess our strategy from a human standpoint, we also conduct a large-scale user study. This study highlights the ability of our network to produce diverse and coherent answers compared to existing approaches. Our Pytorch implementation is available as open source. 1 1 github.com/parshakova/APIP .

Unsupervised Change Detection of SAR Images Based on Variational Multivariate Gaussian Mixture Model and Shannon Entropy

In this letter, we propose an unsupervised change detection method for synthetic aperture radar (SAR) images based on variational multivariate Gaussian mixture model (MGMM) and Shannon entropy. First, the difference features are generated from the Gabor wavelet transform of two SAR images. In variational inference framework, the variational MGMM is first introduced to implement accurate modeling for the data distribution of difference features and to output responsibilities. Subsequently, spatial information is explored on the responsibilities to yield the contextual responsibilities for improving the accuracy and reliability of change detection. Then, a posteriori probabilities of the changed and unchanged classes are derived from the contextual responsibilities , and Shannon entropy, being directly related to the classification error rate, is proposed to determine the optimal index integer. Finally, the binary change mask is achieved by separating the pixels into the changed and unchanged classes. The experiments on three pairs of SAR images for describing urban sprawl and water bodies demonstrate the effectiveness of the proposed method.

Positive Data Modeling Using a Mixture of Mixtures of Inverted Beta Distributions

Finite mixture models based on the symmetric Gaussian distribution have been applied broadly in data analysis. However, not all the data in real-world applications can be safely supposed to have a symmetric Gaussian form. This paper presents a new mixture model that includes the inverted Beta mixture model (IBMM) as a special case to analyze the positive non-Gaussian data. The advantage of the proposed model is that the number of the model parameters is variable and infinite. Consequently, the proposed model is adaptable to the size of the data. On the basis of the recently proposed extended variational inference (EVI) framework, we develop a closed-form solution to approximate the posterior distributions. The performance and the effectiveness of the proposed model are demonstrated with the real data generated from two challenging applications, namely, image classification and object detection.

Conditionally Conjugate Mean-Field Variational Bayes for Logistic Models

Variational Bayes (VB) is a common strategy for approximate Bayesian inference, but simple methods are only available for specific classes of models including, in particular, representations having conditionally conjugate constructions within an exponential family. Models with logit components are an apparently notable exception to this class, due to the absence of conjugacy among the logistic likelihood and the Gaussian priors for the coefficients in the linear predictor. To facilitate approximate inference within this widely used class of models, Jaakkola and Jordan (Stat. Comput. 10 (2000) 25–37) proposed a simple variational approach which relies on a family of tangent quadratic lower bounds of the logistic log-likelihood, thus restoring conjugacy between these approximate bounds and the Gaussian priors. This strategy is still implemented successfully, but few attempts have been made to formally understand the reasons underlying its excellent performance. Following a review on VB for logistic models, we cover this gap by providing a formal connection between the above bound and a recent Pólya-gamma data augmentation for logistic regression. Such a result places the computational methods associated with the aforementioned bounds within the framework of variational inference for conditionally conjugate exponential family models, thereby allowing recent advances for this class to be inherited also by the methods relying on Jaakkola and Jordan (Stat. Comput. 10 (2000) 25–37).

Quality variable prediction for chemical processes based on semisupervised Dirichlet process mixture of Gaussians

Data driven soft sensors have found widespread applications in chemical processes for predicting those important yet difficult-to-measure quality variables. In the vast majority of chemical processes, relationships among primary and secondary variables are nonlinear, and process data inherently contain uncertainties and present strongly non-Gaussian characteristics. In addition, labeled samples are often scarce due to certain technical or economical difficulties. These process and data characteristics impose challenges on high-accuracy soft sensors. To deal with these issues, this paper proposes a soft sensing approach referred to as the semisupervised Dirichlet process mixture of Gaussians (SsDPMG). In the SsDPMG, a fully Bayesian model structure is first designed to enable semisupervised tasks that are suitable for regression applications. Subsequently, a Bayesian learning procedure for the SsDPMG is developed based on variational inference framework, where information contained in both labeled and unlabeled samples are extracted. Case studies are carried out on one numerical example and two real-life chemical processes to evaluate the performance of the proposed approach. The results demonstrate that the SsDPMG is an effective soft sensing approach with promising application foreground.

Stochastic variational hierarchical mixture of sparse Gaussian processes for regression

In this article, we propose a scalable Gaussian process (GP) regression method that combines the advantages of both global and local GP approximations through a two-layer hierarchical model using a variational inference framework. The upper layer consists of a global sparse GP to coarsely model the entire data set, whereas the lower layer comprises a mixture of sparse GP experts which exploit local information to learn a fine-grained model. A two-step variational inference algorithm is developed to learn the global GP, the GP experts and the gating network simultaneously. Stochastic optimization can be employed to allow the application of the model to large-scale problems. Experiments on a wide range of benchmark data sets demonstrate the flexibility, scalability and predictive power of the proposed method.

Variational Bayesian Learning for Dirichlet Process Mixture of Inverted Dirichlet Distributions in Non-Gaussian Image Feature Modeling.

In this paper, we develop a novel variational Bayesian learning method for the Dirichlet process (DP) mixture of the inverted Dirichlet distributions, which has been shown to be very flexible for modeling vectors with positive elements. The recently proposed extended variational inference (EVI) framework is adopted to derive an analytically tractable solution. The convergency of the proposed algorithm is theoretically guaranteed by introducing single lower bound approximation to the original objective function in the EVI framework. In principle, the proposed model can be viewed as an infinite inverted Dirichlet mixture model that allows the automatic determination of the number of mixture components from data. Therefore, the problem of predetermining the optimal number of mixing components has been overcome. Moreover, the problems of overfitting and underfitting are avoided by the Bayesian estimation approach. Compared with several recently proposed DP-related methods and conventional applied methods, the good performance and effectiveness of the proposed method have been demonstrated with both synthesized data and real data evaluations.

Variational Bayesian Inference for Finite Inverted Dirichlet Mixture Model and Its Application to Object Detection

As a variant of Finite mixture model (FMM), finite Inverted Dirichlet mixture model (IDMM) can not avoid the conventional challenges, such as how to select the appropriate number of mixture components based on the observed data. Towards easing these issues, we propose a variational inference framework for learning IDMM which has been proved to be an efficient tool for modeling vectors with positive elements. Compared with the conventional Expectation maximization (EM) algorithm commonly used for learning FMM, the proposed approach prevents over-fitting well. Furthermore, it is able to do automatic determination of the number of mixture components and parameters estimation, simultaneously. Experimental results on both synthetic and real data of object detection confirm significant improvements on flexibility and efficiency being achieved.

Variational Bayesian Inference for Infinite Dirichlet Mixture Towards Accurate Data Categorization

In this paper, we focus on a variational Bayesian learning approach to infinite Dirichlet mixture model (VarInDMM) which inherits the confirmed effectiveness of modeling proportional data from infinite Dirichlet mixture model. Based on the Dirichlet process mixture model, VarInDMM has an interpretation as a mixture model with a countably infinite number of components, and it is able to determine the optimal value of this number according to the observed data. By introducing an extended variational inference framework, we further obtain an analytically tractable solution to estimate the posterior distributions of the parameters for the mixture model. Experimental results on both synthetic and real data demonstrate its good performance on object categorization and text categorization.

Uncertainties in Parameters Estimated with Neural Networks: Application to Strong Gravitational Lensing

Abstract In Hezaveh et al. we showed that deep learning can be used for model parameter estimation and trained convolutional neural networks to determine the parameters of strong gravitational-lensing systems. Here we demonstrate a method for obtaining the uncertainties of these parameters. We review the framework of variational inference to obtain approximate posteriors of Bayesian neural networks and apply it to a network trained to estimate the parameters of the Singular Isothermal Ellipsoid plus external shear and total flux magnification. We show that the method can capture the uncertainties due to different levels of noise in the input data, as well as training and architecture-related errors made by the network. To evaluate the accuracy of the resulting uncertainties, we calculate the coverage probabilities of marginalized distributions for each lensing parameter. By tuning a single variational parameter, the dropout rate, we obtain coverage probabilities approximately equal to the confidence levels for which they were calculated, resulting in accurate and precise uncertainty estimates. Our results suggest that the application of approximate Bayesian neural networks to astrophysical modeling problems can be a fast alternative to Monte Carlo Markov Chains, allowing orders of magnitude improvement in speed.

Variational Bayesian inference for a Dirichlet process mixture of beta distributions and application

Abstract Finite beta mixture model (BMM) has been shown to be very flexible and powerful for bounded support data modeling. However, BMM cannot automatically select the proper number of the mixture components based on the observed data, which is important and has a deterministic effect on the modeling accuracy. In this paper, we aim at tackling this problem by infinite Beta mixture model (InBMM). It is based on the Dirichlet process (DP) mixture with the assumption that the number of the mixture components is infinite in advance and can be automatically determined according to the observed data. Further, a variational InBMM using single lower-bound approximation (VBInBMM) is proposed which applies the stick-breaking representation of the DP and is learned by an extended variational inference framework. Numerical experiments on both synthetic and real data, generated from two challenging application namely image categorization and object detection, demonstrate good performance obtained by the proposed method.

Bayesian K-SVD Using Fast Variational Inference.

Recent work in signal processing in general and image processing in particular deals with sparse representation related problems. Two such problems are of paramount importance: an overriding need for designing a well-suited overcomplete dictionary containing a redundant set of atoms-i.e., basis signals-and how to find a sparse representation of a given signal with respect to the chosen dictionary. Dictionary learning techniques, among which we find the popular K-singular value decomposition algorithm, tackle these problems by adapting a dictionary to a set of training data. A common drawback of such techniques is the need for parameter-tuning. In order to overcome this limitation, we propose a fully-automated Bayesian method that considers the uncertainty of the estimates and produces a sparse representation of the data without prior information on the number of non-zeros in each representation vector. We follow a Bayesian approach that uses a three-tiered hierarchical prior to enforce sparsity on the representations and develop an efficient variational inference framework that reduces computational complexity. Furthermore, we describe a greedy approach that speeds up the whole process. Finally, we present experimental results that show superior performance on two different applications with real images: denoising and inpainting.