Variational Bayesian Approximation Research Articles

Variational Bayesian Methods For Multimedia Problems

In this paper we present an introduction to Variational Bayesian (VB) methods in the context of probabilistic graphical models, and discuss their application in multimedia related problems. VB is a family of deterministic probability distribution approximation procedures that offer distinct advantages over alternative approaches based on stochastic sampling and those providing only point estimates. VB inference is flexible to be applied in different practical problems, yet is broad enough to subsume as its special cases several alternative inference approaches including Maximum A Posteriori (MAP) and the Expectation-Maximization (EM) algorithm. In this paper we also show the connections between VB and other posterior approximation methods such as the marginalization-based Loopy Belief Propagation (LBP) and the Expectation Propagation (EP) algorithms. Specifically, both VB and EP are variational methods that minimize functionals based on the Kullback-Leibler (KL) divergence. LBP, traditionally developed using graphical models, can also be viewed as a VB inference procedure. We present several multimedia related applications illustrating the use and effectiveness of the VB algorithms discussed herein. We hope that by reading this tutorial the readers will obtain a general understanding of Bayesian methods and establish connections among popular algorithms used in practice.

Efficient variational Bayesian approximation method based on subspace optimization.

Variational Bayesian approximations have been widely used in fully Bayesian inference for approximating an intractable posterior distribution by a separable one. Nevertheless, the classical variational Bayesian approximation (VBA) method suffers from slow convergence to the approximate solution when tackling large dimensional problems. To address this problem, we propose in this paper a more efficient VBA method. Actually, variational Bayesian issue can be seen as a functional optimization problem. The proposed method is based on the adaptation of subspace optimization methods in Hilbert spaces to the involved function space, in order to solve this optimization problem in an iterative way. The aim is to determine an optimal direction at each iteration in order to get a more efficient method. We highlight the efficiency of our new VBA method and demonstrate its application to image processing by considering an ill-posed linear inverse problem using a total variation prior. Comparisons with state of the art variational Bayesian methods through a numerical example show a notable improvement in computation time.

Inexact Bayesian point pattern matching for linear transformations

We introduce a novel Bayesian inexact point pattern matching model that assumes that a linear transformation relates the two sets of points. The matching problem is inexact due to the lack of one-to-one correspondence between the point sets and the presence of noise. The algorithm is itself inexact; we use variational Bayesian approximation to estimate the posterior distributions in the face of a problematic evidence term. The method turns out to be similar in structure to the iterative closest point algorithm.

A Measure-Theoretic Variational Bayesian Algorithm for Large Dimensional Problems

In this paper we provide an algorithm adapted to variational Bayesian approximation. The main contribution is to transpose a classical iterative algorithm of optimization in the metric space of measures involved in Bayesian methodology. Once given the convergence properties of this algorithm, we consider its application to large dimensional inverse problems, especially for unsupervised reconstruction. The interest of our method is enhanced by its application to large dimensional linear inverse problems involving sparse objects. Finally, we provide simulation results. First we show the good numerical performances of our method compared to classical ones on a small example. Then we deal with a large dimensional dictionary learning problem.

Improved variational Bayes inference for transcript expression estimation

RNA-seq studies allow for the quantification of transcript expression by aligning millions of short reads to a reference genome. However, transcripts share much of their sequence, so that many reads map to more than one place and their origin remains uncertain. This problem can be dealt using mixtures of distributions and transcript expression reduces to estimating the weights of the mixture. In this paper, variational Bayesian (VB) techniques are used in order to approximate the posterior distribution of transcript expression. VB has previously been shown to be more computationally efficient for this problem than Markov chain Monte Carlo. VB methodology can precisely estimate the posterior means, but leads to variance underestimation. For this reason, a novel approach is introduced which integrates the latent allocation variables out of the VB approximation. It is shown that this modification leads to a better marginal likelihood bound and improved estimate of the posterior variance. A set of simulation studies and application to real RNA-seq datasets highlight the improved performance of the proposed method.

Adaptive probability hypothesis density filter based on variational Bayesian approximation for multi‐target tracking

Probability hypothesis density (PHD) filter has been demonstrated a promising algorithm for tracking an unknown number of targets in real time. However, this method can only be used in the multi-target tracking systems with known measurement noise variances; otherwise, the tracking performance will decline greatly. To solve this problem, an adaptive PHD filter algorithm is proposed based on the variational Bayesian approximation technique to recursively estimate the joint PHDs of the multi-target states and the time-varying measurement noise variances. First, the variational calculus method is employed to derive the multi-target estimate recursions, and then the Gaussian and the inverse Gamma distributions are introduced to approximate the joint posterior PHD, and achieve a closed-form solution. Simulation results show that the proposed algorithm can effectively estimate the unknown measurement noise variances and has a good performance of multi-target tracking with a strong robustness.

Adaptive noise variance identification for probability hypothesis density‐based multi‐target filter by variational Bayesian approximations

A new extended probability hypothesis density (PHD) filter is proposed for joint estimation of the time-varying number of targets and their states without the measurement noise variance. The extended PHD filter can adaptively learn the unknown noise parameters at each scan time by using the received measurements. With the decomposition of the posterior intensity separated into Gaussian and Inverse-Gamma components, the closed-form solutions to the extended PHD filter are derived by using the variational Bayesian approximations, which have been proved as a simple, analytically tractable method to approximate the posterior intensity of multi-target states and time-varying noise variances. Simulation results show that the proposed filter can accommodate the unknown measurement variances effectively, and improve the estimation accuracy of both the number of targets and their states.

Adaptive filtering for jump Markov systems with unknown noise covariance

The paper proposed an adaptive filter for jump Markov systems with unknown measurement noise covariance. The filter is derived by treating covariance as a random matrix and an inverse-Wishart distribution is adopted as the conjugate prior. The variational Bayesian approximation method is employed to derive mode-conditioned estimates and mode-likelihood functions in the framework of interacting multiple model. A numerical example is provided to illustrate the performance of the proposed filter.

Variational Bayesian sparse additive matrix factorization

Principal component analysis (PCA) approximates a data matrix with a low-rank one by imposing sparsity on its singular values. Its robust variant can cope with spiky noise by introducing an element-wise sparse term. In this paper, we extend such sparse matrix learning methods, and propose a novel framework called sparse additive matrix factoriza- tion (SAMF). SAMF systematically induces various types of sparsity by a Bayesian regu- larization effect, called model-induced regularization. Although group LASSO also allows us to design arbitrary types of sparsity on a matrix, SAMF, which is based on the Bayesian framework, provides inference without any requirement for manual parameter tuning. We propose an efficient iterative algorithm called the mean update (MU) for the variational Bayesian approximation to SAMF, which gives the global optimal solution for a large sub- set of parameters in each step. We demonstrate the usefulness of our method on benchmark datasets and a foreground/background video separation problem.

An improved multi-target tracking algorithm based on CBMeMBer filter and variational Bayesian approximation

Random finite set (RFS) filters have been demonstrating a promising algorithm for tracking an unknown number of targets in real time. However, these methods can only be used in the multi-target tracking systems with known measurement noise variances; otherwise, their tracking performances will decline greatly. To solve this problem, an improved multi-target tracking algorithm is proposed based on the cardinality-balanced multi-target multi-Bernoulli (CBMeMBer) filter and the variational Bayesian (VB) approximation technique to recursively estimate the joint posterior distributions of the multi-target states and the time-varying measurement noise variances. First, the variational calculus method is employed to derive the multi-target estimate recursions, and then the Gaussian and inverse Gamma mixture distributions are introduced to approximate the joint posterior density, and achieve a Gaussian closed-form solution. Simulation results show that the proposed algorithm can effectively estimate the unknown measurement noise variances and has a good performance of multi-target tracking with a strong robustness.

Global analytic solution of fully-observed variational Bayesian matrix factorization

The variational Bayesian (VB) approximation is known to be a promising approach to Bayesian estimation, when the rigorous calculation of the Bayes posterior is intractable. The VB approximation has been successfully applied to matrix factorization (MF), offering automatic dimensionality selection for principal component analysis. Generally, finding the VB solution is a nonconvex problem, and most methods rely on a local search algorithm derived through a standard procedure for the VB approximation. In this paper, we show that a better option is available for fully-observed VBMF--the global solution can be analytically computed. More specifically, the global solution is a reweighted SVD of the observed matrix, and each weight can be obtained by solving a quartic equation with its coefficients being functions of the observed singular value. We further show that the global optimal solution of empirical VBMF (where hyperparameters are also learned from data) can also be analytically computed. We illustrate the usefulness of our results through experiments in multi-variate analysis.

Variational Bayesian Approximation methods for inverse problems

Variational Bayesian Approximation (VBA) methods are recent tools for effective Bayesian computations. In this paper, these tools are used for inverse problems where the prior models include hidden variables and where where the estimation of the hyper parameters has also to be addressed. In particular two specific prior models (Student-t and mixture of Gaussian models) are considered and details of the algorithms are given.

Bayesian Robust PCA of Incomplete Data

We present a probabilistic model for robust factor analysis and principal component analysis in which the observation noise is modeled by Student-t distributions in order to reduce the negative effect of outliers. The Student-t distributions are modeled independently for each data dimensions, which is different from previous works using multivariate Student-t distributions. We compare methods using the proposed noise distribution, the multivariate Student-t and the Laplace distribution. Intractability of evaluating the posterior probability density is solved by using variational Bayesian approximation methods. We demonstrate that the assumed noise model can yield accurate reconstructions because corrupted elements of a bad quality sample can be reconstructed using the other elements of the same data vector. Experiments on an artificial dataset and a weather dataset show that the dimensional independency and the flexibility of the proposed Student-t noise model can make it superior in some applications.

A Variational Bayesian Framework for Clustering with Multiple Graphs

Mining patterns in graphs has become an important issue in real applications, such as bioinformatics and web mining. We address a graph clustering problem where a cluster is a set of densely connected nodes, under a practical setting that 1) the input is multiple graphs which share a set of nodes but have different edges and 2) a true cluster cannot be found in all given graphs. For this problem, we propose a probabilistic generative model and a robust learning scheme based on variational Bayesian estimation. A key feature of our probabilistic framework is that not only nodes but also given graphs can be clustered at the same time, allowing our model to capture clusters found in only part of all given graphs. We empirically evaluated the effectiveness of the proposed framework on not only a variety of synthetic graphs but also real gene networks, demonstrating that our proposed approach can improve the clustering performance of competing methods in both synthetic and real data.

State estimation for jump Markov linear systems by variational Bayesian approximation

This article studies the problem of state estimation for jump Markov linear systems with unknown measurement noise variance parameters. By using the concept of conjugate prior distributions for noise statistics, a novel estimator is developed by applying the basic interacting multiple model (IMM) approach and the Kalman filtering theory. The main difficulties encountered are the exponentially growing terms in the interacting stage of the IMM and the coupled state and noise variance in the likelihood functions. They are overcome by employing the merging scheme via matching the first two moments and the variational Bayesian approximation technique, respectively. Simulation results are presented to verify the effectiveness of the proposed filter via a manoeuvring target tracking example.

Multivariate Autoregressive Modeling of Hand Kinematics for Laparoscopic Skills Assessment of Surgical Trainees

Virtual reality (VR) simulators aim to enhance surgical education by allowing trainees to optimize their skills without patient risk. To achieve this quality, an objective analysis of surgical dexterity is crucial. The application of hidden Markov models (HMMs) has offered important insights in the evaluation of surgical skills (e.g., task decomposition), but there are still issues that need standardization, especially when constructing the hand motion vocabulary. In this paper, we investigate an alternative approach based on multivariate autoregressive (MAR) models. Kinematic signals from orientation sensors attached to the instruments of a VR simulator were used to study the laparoscopic skills of surgical residents. Two different tasks were performed: knot tying and needle driving. A variational Bayesian (VB) approximation was employed to calculate the MAR coefficients, which after data reduction were fed to a classifier. The MAR weights also provided the opportunity to study the hand motion connections. Specificity (Spec) and sensitivity (Sens) analysis was used to evaluate and compare the classification performance between MAR models and HMMs. Our results demonstrate the strength of the proposed approach in recognizing surgical maneuvers of residents with limited experience in laparoscopic suturing. The MAR approach yielded the best performance (Sens/Spec: 86%-96%), significantly outperforming the well-established approach of statistical similarity between different HMMs (Sens/Spec: 64%-87%). Subjects at the end of residency training demonstrated more and greater hand motion couplings compared to beginners. The methodological aspects of the proposed approach may be easily embedded in the assessment module of modern laparoscopic simulators.

A tutorial on variational Bayesian inference

This tutorial describes the mean-field variational Bayesian approximation to inference in graphical models, using modern machine learning terminology rather than statistical physics concepts. It begins by seeking to find an approximate mean-field distribution close to the target joint in the KL-divergence sense. It then derives local node updates and reviews the recent Variational Message Passing framework.

Multi-Sensor Centralized Fusion without Measurement Noise Covariance by Variational Bayesian Approximation

The work presented here solves the multi-sensor centralized fusion problem in the linear Gaussian model without the measurement noise variance. We generalize the variational Bayesian approximation based adaptive Kalman filter (VB_AKF) from the single sensor filtering to a multi-sensor fusion system, and propose two new centralized fusion algorithms, i.e., VB_AKF-based augmented centralized fusion algorithm and VB_AKF-based sequential centralized fusion algorithm, to deal with the case that the measurement noise variance is unknown. The simulation results show the effectiveness of the proposed algorithms.

Robust Autoregression: Student-t Innovations Using Variational Bayes

Autoregression (AR) is a tool commonly used to understand and predict time series data. Traditionally the excitation noise is modelled as a Gaussian. However, real-world data may not be Gaussian in nature, and it is known that Gaussian models are adversely affected by the presence of outliers. We introduce a Bayesian AR model in which the excitation noise is assumed to be Student-t distributed. Variational Bayesian approximations to the posterior distributions of the model parameters are used to overcome the intractable integrations inherent in the Bayesian model. Independent automatic relevance determination (ARD) priors over each of the AR coefficients are used to estimate the model order. Using synthetic data, we show that the Student-t model performs well against both Gaussian and leptokurtic data, in terms of parameter estimation (including the model order) and is much more robust to outliers than either Gaussian or finite mixtures of Gaussian models. We apply the model to strongly leptokurtic EEG signals and show that the Student-t model makes more accurate one-step-ahead predictions than the Gaussian model and provides more consistent estimates of the AR coefficients over simultaneously recorded EEG channels.

Multivariate dynamical systems models for estimating causal interactions in fMRI

Analysis of dynamical interactions between distributed brain areas is of fundamental importance for understanding cognitive information processing. However, estimating dynamic causal interactions between brain regions using functional magnetic resonance imaging (fMRI) poses several unique challenges. For one, fMRI measures Blood Oxygenation Level Dependent (BOLD) signals, rather than the underlying latent neuronal activity. Second, regional variations in the hemodynamic response function (HRF) can significantly influence estimation of causal interactions between them. Third, causal interactions between brain regions can change with experimental context over time. To overcome these problems, we developed a novel state-space Multivariate Dynamical Systems (MDS) model to estimate intrinsic and experimentally-induced modulatory causal interactions between multiple brain regions. A probabilistic graphical framework is then used to estimate the parameters of MDS as applied to fMRI data. We show that MDS accurately takes into account regional variations in the HRF and estimates dynamic causal interactions at the level of latent signals. We develop and compare two estimation procedures using maximum likelihood estimation (MLE) and variational Bayesian (VB) approaches for inferring model parameters. Using extensive computer simulations, we demonstrate that, compared to Granger causal analysis (GCA), MDS exhibits superior performance for a wide range of signal to noise ratios (SNRs), sample length and network size. Our simulations also suggest that GCA fails to uncover causal interactions when there is a conflict between the direction of intrinsic and modulatory influences. Furthermore, we show that MDS estimation using VB methods is more robust and performs significantly better at low SNRs and shorter time series than MDS with MLE. Our study suggests that VB estimation of MDS provides a robust method for estimating and interpreting causal network interactions in fMRI data.