Variational Bayes Approach Research Articles

Testing Homogeneity for Normal Mixture Models: Variational Bayes Approach

The test of homogeneity for normal mixtures has been conducted in diverse research areas, but constructing a theory of the test of homogeneity is challenging because the parameter set for the null hypothesis corresponds to singular points in the parameter space. In this paper, we examine this problem from a new perspective and offer a theory of hypothesis testing for homogeneity based on a variational Bayes framework. In the conventional theory, the constant order term of the free energy has remained unknown, however, we clarify its asymptotic behavior because it is necessary for constructing a hypothesis test. Numerical experiments shows the validity of our theoretical results.

Variational Online Learning of Neural Dynamics.

New technologies for recording the activity of large neural populations during complex behavior provide exciting opportunities for investigating the neural computations that underlie perception, cognition, and decision-making. Non-linear state space models provide an interpretable signal processing framework by combining an intuitive dynamical system with a probabilistic observation model, which can provide insights into neural dynamics, neural computation, and development of neural prosthetics and treatment through feedback control. This brings with it the challenge of learning both latent neural state and the underlying dynamical system because neither are known for neural systems a priori. We developed a flexible online learning framework for latent non-linear state dynamics and filtered latent states. Using the stochastic gradient variational Bayes approach, our method jointly optimizes the parameters of the non-linear dynamical system, the observation model, and the black-box recognition model. Unlike previous approaches, our framework can incorporate non-trivial distributions of observation noise and has constant time and space complexity. These features make our approach amenable to real-time applications and the potential to automate analysis and experimental design in ways that testably track and modify behavior using stimuli designed to influence learning.

Online Spatiotemporal Popularity Learning via Variational Bayes for Cooperative Caching

Herein, we focus on an end-to-end design of a proactive cooperative caching strategy for a multi-cell network. The design is challenging as it involves two interrelated problems: the ability to predict future content popularity and to meet network operation characteristics. To this end, we first formulate a cooperative content caching in order to optimize the aggregated network cost for delivering contents to users. An efficient proactive caching policy requires an accurate prediction of time-varying content popularity. Content popularity has temporal and spatial dependencies and therefore, we develop a probabilistic dynamical model for content popularity prediction by exploiting its spatiotemporal correlations. To achieve an accurate tracking and prediction of content popularity evolution, the proposed dynamical model is non-linear and incorporates non-Gaussian distributions. We use Variational Bayes (VB) approach for estimating the model parameters. The VB provides mathematical tractability. We then develop an online VB method that works with streaming data where content request arrives sequentially. Using extensive simulations study on a real-world dataset, we show that our online VB based dynamical model provides improved performance compared to conventional content caching policies.

A Simple Variational Bayes Detector for Orthogonal Time Frequency Space (OTFS) Modulation

The emerging orthogonal time frequency space (OTFS) modulation technique has shown its superiority to the current orthogonal frequency division multiplexing (OFDM) scheme, in terms of its capabilities of exploiting full time-frequency diversity and coping with channel dynamics. The optimal maximum a posteriori (MAP) detection is capable of eliminating the negative impacts of the inter-symbol interference in the delay-Doppler (DD) domain at the expense of a prohibitively high complexity. To reduce the receiver complexity for OTFS scheme, this paper proposes a variational Bayes (VB) approach as an approximation of the optimal MAP detection. Compared to the widely used message passing algorithm, we prove that the proposed iterative algorithm is guaranteed to converge to the global optimum of the approximated MAP detector regardless the resulting factor graph is loopy or not. Simulation results validate the fast convergence of the proposed VB receiver and also show its promising performance gain compared to the conventional message passing algorithm.

BVAR-Connect: A Variational Bayes Approach to Multi-Subject Vector Autoregressive Models for Inference on Brain Connectivity Networks.

In this paper we propose BVAR-connect, a variational inference approach to a Bayesian multi-subject vector autoregressive (VAR) model for inference on effective brain connectivity based on resting-state functional MRI data. The modeling framework uses a Bayesian variable selection approach that flexibly integrates multi-modal data, in particular structural diffusion tensor imaging (DTI) data, into the prior construction. The variational inference approach we develop allows scalability of the methods and results in the ability to estimate subject- and group-level brain connectivity networks over whole-brain parcellations of the data. We provide a brief description of a user-friendly MATLAB GUI released for public use. We assess performance on simulated data, where we show that the proposed inference method can achieve comparable accuracy to the sampling-based Markov Chain Monte Carlo approach but at a much lower computational cost. We also address the case of subject groups with imbalanced sample sizes. Finally, we illustrate the methods on resting-state functional MRI and structural DTI data on children with a history of traumatic injury.

A Variational Approximations-DIC Rubric for Parameter Estimation and Mixture Model Selection Within a Family Setting

Mixture model-based clustering has become an increasingly popular data analysis technique since its introduction over fifty years ago, and is now commonly utilized within a family setting. Families of mixture models arise when the component parameters, usually the component covariance (or scale) matrices, are decomposed and a number of constraints are imposed. Within the family setting, model selection involves choosing the member of the family, i.e., the appropriate covariance structure, in addition to the number of mixture components. To date, the Bayesian information criterion (BIC) has proved most effective for model selection, and the expectation-maximization (EM) algorithm is usually used for parameter estimation. In fact, this EM-BIC rubric has virtually monopolized the literature on families of mixture models. Deviating from this rubric, variational Bayes approximations are developed for parameter estimation and the deviance information criteria (DIC) for model selection. The variational Bayes approach provides an alternate framework for parameter estimation by constructing a tight lower bound on the complex marginal likelihood and maximizing this lower bound by minimizing the associated Kullback-Leibler divergence. The framework introduced, which we refer to as VB-DIC, is applied to the most commonly used family of Gaussian mixture models, and real and simulated data are used to compared with the EM-BIC rubric.

Subset Selection in High-Dimensional Genomic Data using Hybrid Variational Bayes and Bootstrap priors

In this study, the Variational Bayes (VB) approach was hybridized with the bootstrap prior procedure to improve the accuracy of subset selection as well as optimizing the algorithm time in modelling high-dimensional genomic data with inherent sparse structure. The new hybrid VB approach is shown to yields a minimal sufficient statistic which under mild regularity conditions converges to the true sparse structure. Simulation and real-life high-dimensional genomic data experiments revealed comparable empirical performance with other competing frequentist and Bayesian methods. In addition, a new fast algorithm that illustrates the procedure was developed and implemented in the environment of R statistical software as package “VBbootprior”.

Neural variational collaborative filtering with side information for top-K recommendation

Collaborative filtering (CF) is one of the most widely applied models for recommender systems. Despite its success, CF-based methods suffer from rating sparsity and cold-start problem, which leads to poor quality of recommendations. Previous studies have gave great attention to construct hybrid methods, by incorporating side information and user rating. Variational autoencoder (VAE) has been confirmed to be highly effective in CF task, due to its Bayesian nature and non-linearity. However, rating sparsity remains a great challenge to most VAE models, which leads to poor latent user/item representations. In addition, most existing VAE-based methods model either latent user factors or latent item factors, resulting in the incapacity to recommend items to a new user or suggest a new item to existing users. To address these problems, we design a novel deep hybrid framework for top-k recommendation, neural variational collaborative filtering (NVCF), and propose three NVCF-based instantiation. In generative process, the side information of user and item is incorporated to alleviate rating sparsity, for learning better latent user/item representations. In inference process, a Stochastic Gradient Variational Bayes approach is employed to approximate the unmanageable distributions of latent user/item factors. Experiments performed on four public datasets have indicated our methods significantly outperform the state-of-the-art hybrid CF models and VAE-based methods.

A Variational Bayes Approach to Adaptive Radio Tomography

Radio tomographic imaging (RTI) is an emerging technology for localization of physical objects in a geographical area covered by wireless networks. With attenuation measurements collected at spatially distributed sensors, RTI capitalizes on spatial loss fields (SLFs) measuring the absorption of radio frequency waves at spatial locations along the propagation path. These SLFs can be utilized for interference management in wireless communication networks, environmental monitoring, and survivor localization after natural disasters such as earthquakes. Key to the success of RTI is to accurately model shadowing as the weighted line integral of the SLF. To learn the SLF exhibiting statistical heterogeneity induced by spatially diverse environments, the present work develops a Bayesian framework entailing a piecewise homogeneous SLF with an underlying hidden Markov random field model. Utilizing variational Bayes techniques, the novel approach yields efficient field estimators at affordable complexity. A data-adaptive sensor selection strategy is also introduced to collect informative measurements for effective reconstruction of the SLF. Numerical tests using synthetic and real datasets demonstrate the capabilities of the proposed approach to radio tomography and channel-gain estimation.

GEMINI: a variational Bayesian approach to identify genetic interactions from combinatorial CRISPR screens

Systems for CRISPR-based combinatorial perturbation of two or more genes are emerging as powerful tools for uncovering genetic interactions. However, systematic identification of these relationships is complicated by sample, reagent, and biological variability. We develop a variational Bayes approach (GEMINI) that jointly analyzes all samples and reagents to identify genetic interactions in pairwise knockout screens. The improved accuracy and scalability of GEMINI enables the systematic analysis of combinatorial CRISPR knockout screens, regardless of design and dimension. GEMINI is available as an open source R package on GitHub at https://github.com/sellerslab/gemini.

Estimation of Viterbi path in Bayesian hidden Markov models

The article studies different methods for estimating the Viterbi path in the Bayesian framework. The Viterbi path is an estimate of the underlying state path in hidden Markov models (HMMs), which has a maximum joint posterior probability. Hence it is also called the maximum a posteriori (MAP) path. For an HMM with given parameters, the Viterbi path can be easily found with the Viterbi algorithm. In the Bayesian framework the Viterbi algorithm is not applicable and several iterative methods can be used instead. We introduce a new EM-type algorithm for finding the MAP path and compare it with various other methods for finding the MAP path, including the variational Bayes approach and MCMC methods. Examples with simulated data are used to compare the performance of the methods. The main focus is on non-stochastic iterative methods and our results show that the best of those methods work as well or better than the best MCMC methods. Our results demonstrate that when the primary goal is segmentation, then it is more reasonable to perform segmentation directly by considering the transition and emission parameters as nuisance parameters.

Collaborative Variational Deep Learning for Healthcare Recommendation

Healthcare recommender system (HRS) has shown the great potential of targeting medical experts or patients, and plays a key role in improving an individual’s health by providing insightful recommendations. The HRSs generate recommendations based on a successful and widely applied method known as collaborative filtering (CF). Despite its success, the CF suffers from data sparsity and cold-start problem, which results in the poor quality of recommendations. In particular, it is a great challenge to seeking information relevant to patients’ condition, and understanding the medical terms and relationships between them in HRSs. To address these problems, we design a novel collaborative variational deep learning model (CVDL) to exploit multi-sourced information for providing appropriate healthcare recommendations in primary care service. CVDL employs additional variational autoencoder (VAE) to learn deep latent representations for item contents (the description of primary care doctors) in latent space, instead of observation space through an inference network. Meanwhile, the CVDL extracts latent user (patient) features by incorporating user profile in a VAE neural network. Therefore, the CVDL can learn better implicit relationships between items and users from item content, user profile, and rating matrix. In addition, a Stochastic Gradient Variational Bayes (SGVB) approach is proposed to calculate the maximum posterior estimates for learning model parameters. The experiments conducted on three datasets have indicated that our method significantly outperforms the state-of-the-art hybrid CF methods.

Flexible online multivariate regression with variational Bayes and the matrix-variate Dirichlet process

Flexible density regression methods, in which the whole distribution of a response vector changes with the covariates, are very useful in some applications. A recently developed technique of this kind uses the matrix-variate Dirichlet process as a prior for a mixing distribution on a coefficient in a multivariate linear regression model. The method is attractive for the convenient way that it allows borrowing strength across different component regressions and for its computational simplicity and tractability. The purpose of the present article is to develop fast online variational Bayes approaches to fitting this model, and to investigate how they perform compared to MCMC and batch variational methods in a number of scenarios.

Robust Linear Filter with Parameter Estimation Under Student-t Measurement Distribution

In this paper, we propose an approximate Kalman filter of measurements following Student’s t-distribution by using the variational Bayes approach. This approach can decompose the estimation of multivariate parameters into a univariate estimation. The recursive formula for the approximate posterior densities of parameters and states is derived in detail. Then, the asymptotic Bayesian Cramer–Rao lower bounds are derived for the proposed filter. Numerical simulations verify both the performance of the proposed filter and the variance lower bounds under time-varying noise. The efficiency of the proposed filter is also demonstrated in a real application, namely an integrated strapdown inertial navigation system/Doppler velocity log shipborne test for navigation.

A Study of the discrepancy function when modeling a packed bed using Variational Bayes and Gaussian processes

Estimating properties using maximum likelihood (MLE) gives point values and a rough idea of the uncertainty in the parameters if their distributions are approximately normal. More accurate results require a Monte Carlo approach, often using the Markov Chain Monte Carlo (MCMC) method, which for complex models is unrealistic because of the computational expense. The Variational Bayes approach gives results comparable to MCMC with only a few evaluation of the model. When the model is imprecise or expensive to evaluate, Gaussian processes provide one means of analyzing the noise to determine the uncertainties.

Deconvolution and Restoration of Optical Endomicroscopy Images

Optical endomicroscopy (OEM) is an emerging technology platform with preclinical and clinical imaging applications. Pulmonary OEM via fibre bundles has the potential to provide in vivo, in situ molecular signatures of disease such as infection and inflammation. However, enhancing the quality of data acquired by this technique for better visualization and subsequent analysis remains a challenging problem. Cross coupling between fiber cores and sparse sampling by imaging fiber bundles are the main reasons for image degradation, and poor detection performance (i.e., inflammation, bacteria, etc.). In this work, we address the problem of deconvolution and restoration of OEM data. We propose a hierarchical Bayesian model to solve this problem and compare three estimation algorithms to exploit the resulting joint posterior distribution. The first method is based on Markov chain Monte Carlo (MCMC) methods, however, it exhibits a relatively long computational time. The second and third algorithms deal with this issue and are based on a variational Bayes (VB) approach and an alternating direction method of multipliers (ADMM) algorithm respectively. Results on both synthetic and real datasets illustrate the effectiveness of the proposed methods for restoration of OEM images.

Remote Sensing Image Classification With Large-Scale Gaussian Processes

Current remote sensing image classification problems have to deal with an unprecedented amount of heterogeneous and complex data sources. Upcoming missions will soon provide large data streams that will make land cover/use classification difficult. Machine learning classifiers can help at this, and many methods are currently available. A popular kernel classifier is the Gaussian process classifier (GPC), since it approaches the classification problem with a solid probabilistic treatment, thus yielding confidence intervals for the predictions as well as very competitive results to state-of-the-art neural networks and support vector machines. However, its computational cost is prohibitive for large scale applications, and constitutes the main obstacle precluding wide adoption. This paper tackles this problem by introducing two novel efficient methodologies for Gaussian Process (GP) classification. We first include the standard random Fourier features approximation into GPC, which largely decreases its computational cost and permits large scale remote sensing image classification. In addition, we propose a model which avoids randomly sampling a number of Fourier frequencies, and alternatively learns the optimal ones within a variational Bayes approach. The performance of the proposed methods is illustrated in complex problems of cloud detection from multispectral imagery and infrared sounding data. Excellent empirical results support the proposal in both computational cost and accuracy.

Genomes as documents of evolutionary history: a probabilistic macrosynteny model for the reconstruction of ancestral genomes.

MotivationIt has been argued that whole-genome duplication (WGD) exerted a profound influence on the course of evolution. For the purpose of fully understanding the impact of WGD, several formal algorithms have been developed for reconstructing pre-WGD gene order in yeast and plant. However, to the best of our knowledge, those algorithms have never been successfully applied to WGD events in teleost and vertebrate, impeded by extensive gene shuffling and gene losses.ResultsHere, we present a probabilistic model of macrosynteny (i.e. conserved linkage or chromosome-scale distribution of orthologs), develop a variational Bayes algorithm for inferring the structure of pre-WGD genomes, and study estimation accuracy by simulation. Then, by applying the method to the teleost WGD, we demonstrate effectiveness of the algorithm in a situation where gene-order reconstruction algorithms perform relatively poorly due to a high rate of rearrangement and extensive gene losses. Our high-resolution reconstruction reveals previously overlooked small-scale rearrangements, necessitating a revision to previous views on genome structure evolution in teleost and vertebrate.ConclusionsWe have reconstructed the structure of a pre-WGD genome by employing a variational Bayes approach that was originally developed for inferring topics from millions of text documents. Interestingly, comparison of the macrosynteny and topic model algorithms suggests that macrosynteny can be regarded as documents on ancestral genome structure. From this perspective, the present study would seem to provide a textbook example of the prevalent metaphor that genomes are documents of evolutionary history.Availability and implementationThe analysis data are available for download at http://www.gen.tcd.ie/molevol/supp_data/MacrosyntenyTGD.zip, and the software written in Java is available upon request.Supplementary information Supplementary data are available at Bioinformatics online.

Approximate inference of nonparametric Hammerstein models

We propose a method for nonparametric identification of Hammerstein models with Gaussian-process models for the impulse response of the linear block and for the input nonlinearity. Interpreting the Gaussian-processes as prior distributions, we can estimate the unknowns using the posterior means given the data. To estimate the hyperparameters we set up an iterative scheme, reminiscent of the expectation-maximization method, where the posterior expectation of the complete likelihood is iteratively maximized. In the Hammerstein case, the posterior density is intractable because, in general, it does not admit a closed form expression. In this work, we propose two approximation approaches to estimate the posterior mean. In the first, we make a particle approximation of the posterior using Markov Chain Monte Carlo. In the second, we use a variational Bayes approach with a mean-field hypothesis. We validate the proposed methods on synthetic datasets of Hammerstein systems.

Variational Bayes identification of acyclic dynamic networks

We study the problem of identifying dynamic networks that do not present loops. We model the impulse responses of the modules in the network as zero-mean independent Gaussian processes. The covariance matrices of the processes can be used to encode prior information, such as stability and smoothness, about the impulse responses of the modules. To estimate the modules, we approximate the joint posterior distribution of the impulse responses using a variational Bayes approach. In particular, using a mean-field approximation, we assume a factorization of the posterior where each factor corresponds to a single module. We estimate the kernel hyperparameters and the measurement noise variances by combining variational Bayes with the expectation-maximization method. We evaluate the performance of the identification procedure in a simulation experiment, where we compare to other kernel-based approaches.