Variational Bayesian Framework Research Articles

Regularized Variational Bayesian Learning of Echo State Networks with Delay&Sum Readout

In this work, a variational Bayesian framework for efficient training of echo state networks (ESNs) with automatic regularization and delay&sum (D&S) readout adaptation is proposed. The algorithm uses a classical batch learning of ESNs. By treating the network echo states as fixed basis functions parameterized with delay parameters, we propose a variational Bayesian ESN training scheme. The variational approach allows for a seamless combination of sparse Bayesian learning ideas and a variational Bayesian space-alternating generalized expectation-maximization (VB-SAGE) algorithm for estimating parameters of superimposed signals. While the former method realizes automatic regularization of ESNs, which also determines which echo states and input signals are relevant for "explaining" the desired signal, the latter method provides a basis for joint estimation of D&S readout parameters. The proposed training algorithm can naturally be extended to ESNs with fixed filter neurons. It also generalizes the recently proposed expectation-maximization-based D&S readout adaptation method. The proposed algorithm was tested on synthetic data prediction tasks as well as on dynamic handwritten character recognition.

Optical diffraction tomography within a variational Bayesian framework

Optical diffraction tomography techniques aim to find an image of an unknown object (e.g. a map of its refraction index) using measurements of the scattered field that results from its interaction with a known interrogating wave. We address this issue as a nonlinear inverse scattering problem. The forward model is based upon domain integral representations of the electric field whose discrete counterparts are obtained by means of a method of moments. The inverse problem is tackled in a Bayesian estimation framework involving a hierarchical prior model that accounts for the piece-wise homogeneity of the object. A joint unsupervised estimation approach is adopted to estimate the induced currents, the contrast and all the other parameters introduced in the prior model. As an analytic expression for the joint maximum a posteriori (MAP) and posterior mean (PM) estimators is hard to obtain, a tractable approximation of the latter is proposed. This approximation is based upon a variational Bayesian technique and consists in the best separable distribution that approximates the true posterior distribution in the Kullback–Leibler sense. This leads to an implicit parametric optimization scheme which is solved iteratively.

Bayesian symmetrical EEG/fMRI fusion with spatially adaptive priors

In this paper, we propose a novel symmetrical EEG/fMRI fusion method which combines EEG and fMRI by means of a common generative model. We use a total variation (TV) prior to model the spatial distribution of the cortical current responses and hemodynamic response functions, and utilize spatially adaptive temporal priors to model their temporal shapes. The spatial adaptivity of the prior model allows for adaptation to the local characteristics of the estimated responses and leads to high estimation performance for the cortical current distribution and the hemodynamic response functions. We utilize a Bayesian formulation with a variational Bayesian framework and obtain a fully automatic fusion algorithm. Simulations with synthetic data and experiments with real data from a multimodal study on face perception demonstrate the performance of the proposed method.

Adaptive iterative detectors for phase-uncertain channels via variational bounding

The problem of iterative detection/decoding of data symbols transmitted over an additive white Gaussian noise (AWGN) channel in the presence of phase uncertainty is addressed in this paper. By modelling the phase uncertainty either as an unknown deterministic variable/process or random variable/ process with a known a priori probability density function, a number of non-Bayesian and Bayesian detection algorithms with various amount of suboptimality have been proposed in the literature to solve the problem. In this paper, a new set of suboptimal iterative detection algorithms is obtained by utilizing the variational bounding technique. Especially, applying the generic variational Bayesian (VB) framework, efficient iterative joint estimation and detection/decoding schemes are derived for the constant phase model as well as for the dynamic phase model. In addition, the relation of the VB-based approach to the optimal noncoherent receiver as well as to the classical approach via the expectation-maximization (EM) algorithm is provided. Performance of the proposed detectors in the presence of a strong dynamic phase noise is compared to the performance of the existing detectors. Furthermore, an incremental scheduling of the VB (or EM) algorithm is shown to reduce the overall complexity of the receiver.

Patch-Based Video Processing: A Variational Bayesian Approach

In this paper, we present a patch-based variational Bayesian framework for video processing and demonstrate its potential in denoising, inpainting and deinterlacing. Unlike previous methods based on explicit motion estimation, we propose to embed motion-related information into the relationship among video patches and develop a nonlocal sparsity-based prior for typical video sequences. Specifically, we first extend block matching (nearest neighbor search) into patch clustering (k-nearest-neighbor search), which represents motion in an implicit and distributed fashion. Then we show how to exploit the sparsity constraint by sorting and packing similar patches, which can be better understood from a manifold perspective. Under the Bayesian framework, we treat both patch clustering result and unobservable data as latent variables and solve the inference problem via variational EM algorithms. A weighted averaging strategy of fusing diverse inference results from overlapped patches is also developed. The effectiveness of patch-based video models is demonstrated by extensive experimental results on a wide range of video materials.

Second-Order Latent-Space Variational Bayes for Approximate Bayesian Inference

In this letter, we consider a variational approximate Bayesian inference framework, latent-space variational Bayes (LSVB), in the general context of conjugate-exponential family models with latent variables. In the LSVB approach, we integrate out model parameters in an exact way and then perform the variational inference over only the latent variables. It can be shown that LSVB can achieve better estimates of the model evidence as well as the distribution over the latent variables than the popular variational Bayesian expectation-maximization (VBEM). However, the distribution over the latent variables in LSVB has to be approximated in practice. As an approximate implementation of LSVB, we propose a second-order LSVB (SoLSVB) method. In particular, VBEM can be derived as a special case of a first-order approximation in LSVB (Sung). SoLSVB can capture higher order statistics neglected in VBEM and can therefore achieve a better approximation. Examples of Gaussian mixture models are used to illustrate the comparison between our method and VBEM, demonstrating the improvement.

A neurologically plausible implementation of statistical inference applied to random dot motion

We propose a model of basic motion perception consisting of a hierarchical non-linear state space model (NSSM) developed within a variational Bayesian (VB) framework. Each level of the hierarchy is a `cause' that generates a prior distribution on the level below via a generative function. The temporal dynamics and generative functions between layers of the hierarchy are implemented as neural networks with non-linear activation functions. Optimization of model parameters and causes proceed concurrently as a combination of fixed-point rules and gradient decent schemes. To make the optimization problem tractable, the standard mean-field and Laplace approximations are employed. The precise factoring used in the mean-field approximation is designed to meet a balance between tractability, neurological plausibility and modeling power. In this approach inference and learning proceed concurrently, in an online and unsupervised fashion. Other work has produced similar implementations of NSSMs that have been successful in predicting low-dimensional temporal signals, but with highly restrictive assumptions on the form of the posterior distributions and with learning done in batches instead of online [1]. Competing work has relaxed many of these implementation assumptions and achieved prediction of high dimensional input but at the cost of discarding VB techniques in favor of inefficient discrete state-space methods [2]. Moreover, such techniques have been demonstrated only with pre-learned weights and simplistic statistical models. Our model is primarily tested on realistic high-dimensional input generated by randomly moving dots over a detection grid. The results from a spiking neuron implementation of the model based on the Neural Engineering Framework (NEF) are compared directly to single cell recordings in random dot motion perception and decision-making tasks.

Symmetrical event-related EEG/fMRI information fusion in a variational Bayesian framework

In this work, we propose a symmetrical multimodal EEG/fMRI information fusion approach dedicated to the identification of event-related bioelectric and hemodynamic responses. Unlike existing, asymmetrical EEG/fMRI data fusion algorithms, we build a joint EEG/fMRI generative model that explicitly accounts for local coupling/uncoupling of bioelectric and hemodynamic activities, which are supposed to share a common substrate. Under a dedicated assumption of spatio-temporal separability, the spatial profile of the common EEG/fMRI sources is introduced as an unknown hierarchical prior on both markers of cerebral activity. Thereby, a devoted Variational Bayesian (VB) learning scheme is derived to infer common EEG/fMRI sources from a joint EEG/fMRI dataset. This yields an estimate of the common spatial profile, which is built as a trade-off between information extracted from EEG and fMRI datasets. Furthermore, the spatial structure of the EEG/fMRI coupling/uncoupling is learned exclusively from the data. The proposed data generative model and devoted VBEM learning scheme thus provide an un-supervised well-balanced approach for the fusion of EEG/fMRI information. We first demonstrate our approach on synthetic data. Results show that, in contrast to classical EEG/fMRI fusion approach, the method proved efficient and robust regardless of the EEG/fMRI discordance level. We apply the method on EEG/fMRI recordings from a patient with epilepsy, in order to identify brain areas involved during the generation of epileptic spikes. The results are validated using intracranial EEG measurements.

Hierarchical Bayesian estimates of distributed MEG sources: Theoretical aspects and comparison of variational and MCMC methods

Magnetoencephalography (MEG) provides millisecond-scale temporal resolution for noninvasive mapping of human brain functions, but the problem of reconstructing the underlying source currents from the extracranial data has no unique solution. Several distributed source estimation methods based on different prior assumptions have been suggested for the resolution of this inverse problem. Recently, a hierarchical Bayesian generalization of the traditional minimum norm estimate (MNE) was proposed, in which the variance of distributed current at each cortical location is considered as a random variable and estimated from the data using the variational Bayesian (VB) framework. Here, we introduce an alternative scheme for performing Bayesian inference in the context of this hierarchical model by using Markov chain Monte Carlo (MCMC) strategies. In principle, the MCMC method is capable of numerically representing the true posterior distribution of the currents whereas the VB approach is inherently approximative. We point out some potential problems related to hyperprior selection in the previous work and study some possible solutions. A hyperprior sensitivity analysis is then performed, and the structure of the posterior distribution as revealed by the MCMC method is investigated. We show that the structure of the true posterior is rather complex with multiple modes corresponding to different possible solutions to the source reconstruction problem. We compare the results from the VB algorithm to those obtained from the MCMC simulation under different hyperparameter settings. The difficulties in using a unimodal variational distribution as a proxy for a truly multimodal distribution are also discussed. Simulated MEG data with realistic sensor and source geometries are used in performing the analyses.

A variational Bayesian mixture modelling framework for cluster analysis of gene-expression data

Accurate subcategorization of tumour types through gene-expression profiling requires analytical techniques that estimate the number of categories or clusters rigorously and reliably. Parametric mixture modelling provides a natural setting to address this problem. We compare a criterion for model selection that is derived from a variational Bayesian framework with a popular alternative based on the Bayesian information criterion. Using simulated data, we show that the variational Bayesian method is more accurate in finding the true number of clusters in situations that are relevant to current and future microarray studies. We also compare the two criteria using freely available tumour microarray datasets and show that the variational Bayesian method is more sensitive to capturing biologically relevant structure.

Variational Bayesian inference for fMRI time series

We describe a Bayesian estimation and inference procedure for fMRI time series based on the use of General Linear Models with Autoregressive (AR) error processes. We make use of the Variational Bayesian (VB) framework which approximates the true posterior density with a factorised density. The fidelity of this approximation is verified via Gibbs sampling. The VB approach provides a natural extension to previous Bayesian analyses which have used Empirical Bayes. VB has the advantage of taking into account the variability of hyperparameter estimates with little additional computational effort. Further, VB allows for automatic selection of the order of the AR process. Results are shown on simulated data and on data from an event-related fMRI experiment.

Bayesian model search for mixture models based on optimizing variational bounds

When learning a mixture model, we suffer from the local optima and model structure determination problems. In this paper, we present a method for simultaneously solving these problems based on the variational Bayesian (VB) framework. First, in the VB framework, we derive an objective function that can simultaneously optimize both model parameter distributions and model structure. Next, focusing on mixture models, we present a deterministic algorithm to approximately optimize the objective function by using the idea of the split and merge operations which we previously proposed within the maximum likelihood framework. Then, we apply the method to mixture of expers (MoE) models to experimentally show that the proposed method can find the optimal number of experts of a MoE while avoiding local maxima.