Variational Bayes Approach Research Articles

Variational Bayes latent class analysis for EHR-based phenotyping with large real-world data

IntroductionBayesian approaches to patient phenotyping in clinical observational studies have been limited by the computational challenges associated with applying the Markov Chain Monte Carlo (MCMC) approach to real-world data. Approximate Bayesian inference via optimization of the variational evidence lower bound, variational Bayes (VB), has been successfully demonstrated for other applications.MethodsWe investigate the performance and characteristics of currently available VB and MCMC software to explore the practicability of available approaches and provide guidance for clinical practitioners. Two case studies are used to fully explore the methods covering a variety of real-world data. First, we use the publicly available Pima Indian diabetes data to comprehensively compare VB implementations of logistic regression. Second, a large real-world data set, Optum™ EHR with approximately one million diabetes patients extended the analysis to large, highly unbalanced data containing discrete and continuous variables. A Bayesian patient phenotyping composite model incorporating latent class analysis (LCA) and regression was implemented with the second case study.ResultsWe find that several data characteristics common in clinical data, such as sparsity, significantly affect the posterior accuracy of automatic VB methods compared with conditionally conjugate mean-field methods. We find that for both models, automatic VB approaches require more effort and technical knowledge to set up for accurate posterior estimation and are very sensitive to stopping time compared with closed-form VB methods.DiscussionOur results indicate that the patient phenotyping composite Bayes model is more easily usable for real-world studies if Monte Carlo is replaced with VB. It can potentially become a uniquely useful tool for decision support, especially for rare diseases where gold-standard biomarker data are sparse but prior knowledge can be used to assist model diagnosis and may suggest when biomarker tests are warranted.

A stable deep adversarial learning approach for geological facies generation

The simulation of geological facies in an unobservable volume is essential in various geoscience applications. Given the complexity of the problem, deep generative learning is a promising approach to overcome the limitations of traditional geostatistical simulation models, in particular their lack of physical realism. This research aims to investigate the application of generative adversarial networks and deep variational inference for conditionally simulating channelized reservoir in underground volumes. In this paper, we review the generative deep learning approaches, in particular the adversarial ones and the stabilization techniques that aim to facilitate their training. We also study the problem of conditioning deep learning models to observations through a variational Bayes approach, comparing a conditional neural network model to a Gaussian mixture model. The proposed approach is tested on 2D and 3D simulations generated by the stochastic process-based model Flumy. Morphological metrics are utilized to compare our proposed method with earlier iterations of generative adversarial networks. The results indicate that by utilizing recent stabilization techniques, generative adversarial networks can efficiently sample complex target data distributions.

Sparse Bayesian Deep Learning for Cross Domain Medical Image Reconstruction

Cross domain medical image reconstruction aims to address the issue that deep learning models trained solely on one source dataset might not generalize effectively to unseen target datasets from different hospitals. Some recent methods achieve satisfactory reconstruction performance, but often at the expense of extensive parameters and time consumption. To strike a balance between cross domain image reconstruction quality and model computational efficiency, we propose a lightweight sparse Bayesian deep learning method. Notably, we apply a fixed-form variational Bayes (FFVB) approach to quantify pixel-wise uncertainty priors derived from degradation distribution of the source domain. Furthermore, by integrating the uncertainty prior into the posterior sampled through stochastic gradient Langevin dynamics (SGLD), we develop a training strategy that dynamically generates and optimizes the prior distribution on the network weights for each unseen domain. This strategy enhances generalizability and ensures robust reconstruction performance. When evaluated on medical image reconstruction tasks, our proposed approach demonstrates impressive performance across various previously unseen domains.

Variational Inference for Large Bayesian Vector Autoregressions

We propose a novel variational Bayes approach to estimate high-dimensional Vector Autoregressive (VAR) models with hierarchical shrinkage priors. Our approach does not rely on a conventional structural representation of the parameter space for posterior inference. Instead, we elicit hierarchical shrinkage priors directly on the matrix of regression coefficients so that (a) the prior structure maps into posterior inference on the reduced-form transition matrix and (b) posterior estimates are more robust to variables permutation. An extensive simulation study provides evidence that our approach compares favorably against existing linear and nonlinear Markov chain Monte Carlo and variational Bayes methods. We investigate the statistical and economic value of the forecasts from our variational inference approach for a mean-variance investor allocating her wealth to different industry portfolios. The results show that more accurate estimates translate into substantial out-of-sample gains across hierarchical shrinkage priors and model dimensions.

Variational inference for semiparametric Bayesian novelty detection in large datasets

After being trained on a fully-labeled training set, where the observations are grouped into a certain number of known classes, novelty detection methods aim to classify the instances of an unlabeled test set while allowing for the presence of previously unseen classes. These models are valuable in many areas, ranging from social network and food adulteration analyses to biology, where an evolving population may be present. In this paper, we focus on a two-stage Bayesian semiparametric novelty detector, also known as Brand, recently introduced in the literature. Leveraging on a model-based mixture representation, Brand allows clustering the test observations into known training terms or a single novelty term. Furthermore, the novelty term is modeled with a Dirichlet Process mixture model to flexibly capture any departure from the known patterns. Brand was originally estimated using MCMC schemes, which are prohibitively costly when applied to high-dimensional data. To scale up Brand applicability to large datasets, we propose to resort to a variational Bayes approach, providing an efficient algorithm for posterior approximation. We demonstrate a significant gain in efficiency and excellent classification performance with thorough simulation studies. Finally, to showcase its applicability, we perform a novelty detection analysis using the openly-available Statlog dataset, a large collection of satellite imaging spectra, to search for novel soil types.

Layer adaptive node selection in Bayesian neural networks: Statistical guarantees and implementation details

Sparse deep neural networks have proven to be efficient for predictive model building in large-scale studies. Although several works have studied theoretical and numerical properties of sparse neural architectures, they have primarily focused on the edge selection. Sparsity through edge selection might be intuitively appealing; however, it does not necessarily reduce the structural complexity of a network. Instead pruning excessive nodes leads to a structurally sparse network with significant computational speedup during inference. To this end, we propose a Bayesian sparse solution using spike-and-slab Gaussian priors to allow for automatic node selection during training. The use of spike-and-slab prior alleviates the need of an ad-hoc thresholding rule for pruning. In addition, we adopt a variational Bayes approach to circumvent the computational challenges of traditional Markov Chain Monte Carlo (MCMC) implementation. In the context of node selection, we establish the fundamental result of variational posterior consistency together with the characterization of prior parameters. In contrast to the previous works, our theoretical development relaxes the assumptions of the equal number of nodes and uniform bounds on all network weights, thereby accommodating sparse networks with layer-dependent node structures or coefficient bounds. With a layer-wise characterization of prior inclusion probabilities, we discuss the optimal contraction rates of the variational posterior. We empirically demonstrate that our proposed approach outperforms the edge selection method in computational complexity with similar or better predictive performance. Our experimental evidence further substantiates that our theoretical work facilitates layer-wise optimal node recovery.

An Approximated Collapsed Variational Bayes Approach to Variable Selection in Linear Regression

In this work, we propose a novel approximated collapsed variational Bayes approach to model selection in linear regression. The approximated collapsed variational Bayes algorithm offers improvements over mean field variational Bayes by marginalizing over a subset of parameters and using mean field variational Bayes over the remaining parameters in an analogous fashion to collapsed Gibbs sampling. We have shown that the proposed algorithm, under typical regularity assumptions, (a) includes variables in the true underlying model at an exponential rate in the sample size, or (b) excludes the variables at least at the first order rate in the sample size if the variables are not in the true model. Simulation studies show that the performance of the proposed method is close to that of a particular Markov chain Monte Carlo sampler and a path search based variational Bayes algorithm, but requires an order of magnitude less time. The proposed method is also highly competitive with penalized methods, expectation propagation, stepwise AIC/BIC, BMS, and EMVS under various settings. Supplementary materials for the article are available online.

Fast Bayesian inference on spectral analysis of multivariate stationary time series

Spectral analysis discovers trends, periodic and other characteristics of a time series by representing these features in the frequency domain. However, when multivariate time series are considered, and the number of components increases, the size of the spectral density matrix grows quadratically, making estimation and inference rather challenging. The proposed novel Bayesian framework considers a Whittle likelihood-based spectral modeling approach and imposes a discounted regularized horseshoe prior on the coefficients that define a spline representation of each of the components of a Cholesky factorization of the inverse spectral density matrix. The proposed prior structure leads to a model that provides higher posterior accuracy when compared to alternative currently available approaches. To achieve fast inference that takes advantage of the massive power of modern hardware (e.g., GPU), a stochastic gradient variational Bayes approach is proposed for the highly parallelizable posterior inference that provides computational flexibility for modeling high-dimensional time series. The accurate empirical performance of the proposed method is illustrated via extensive simulation studies and the analysis of two datasets: a wind speed data from 6 locations in California, and a 61-channel electroencephalogram data recorded on two contrasting subjects under specific experimental conditions.

Fitting Structural Equation Models via Variational Approximations

Structural equation models are commonly used to capture the relationship between sets of observed and unobservable variables. Traditionally these models are fitted using frequentist approaches, but recently researchers and practitioners have developed increasing interest in Bayesian inference. In Bayesian settings, inference for these models is typically performed via Markov chain Monte Carlo methods, which may be computationally intensive for models with a large number of manifest variables or complex structures. Variational approximations can be a fast alternative; however, they have not been adequately explored for this class of models. We develop a mean field variational Bayes approach for fitting elemental structural equation models and demonstrate how bootstrap can considerably improve the variational approximation quality. We show that this variational approximation method can provide reliable inference while being significantly faster than Markov chain Monte Carlo methods.

Approximation of Zero-Inflated Poisson Credibility Premium via Variational Bayes Approach

While both zero-inflation and the unobserved heterogeneity in risks are prevalent issues in modeling insurance claim counts, determination of Bayesian credibility premium of the claim counts with these features are often demanding due to high computational costs associated with a use of MCMC. This article explores a way to approximate credibility premium for claims frequency that follows a zero-inflated Poisson distribution via variational Bayes approach. Unlike many existing industry benchmarks, the proposed method enables insurance companies to capture both zero-inflation and unobserved heterogeneity of policyholders simultaneously with modest computation costs. A simulation study and an empirical analysis using the LGPIF dataset were conducted and it turned out that the proposed method outperforms many industry benchmarks in terms of prediction performances and computation time. Such results support the applicability of the proposed method in the posterior ratemaking practices.

Learn From Unpaired Data for Image Restoration: A Variational Bayes Approach.

Collecting paired training data is difficult in practice, but the unpaired samples broadly exist. Current approaches aim at generating synthesized training data from unpaired samples by exploring the relationship between the corrupted and clean data. This work proposes LUD-VAE, a deep generative method to learn the joint probability density function from data sampled from marginal distributions. Our approach is based on a carefully designed probabilistic graphical model in which the clean and corrupted data domains are conditionally independent. Using variational inference, we maximize the evidence lower bound (ELBO) to estimate the joint probability density function. Furthermore, we show that the ELBO is computable without paired samples under the inference invariant assumption. This property provides the mathematical rationale of our approach in the unpaired setting. Finally, we apply our method to real-world image denoising, super-resolution, and low-light image enhancement tasks and train the models using the synthetic data generated by the LUD-VAE. Experimental results validate the advantages of our method over other approaches.

VAE-KRnet and Its Applications to Variational Bayes

In this work, we have proposed a generative model, called VAE-KRnet, for density estimation or approximation, which combines the canonical variational autoencoder (VAE) with our recently developed flow-based generative model, called KRnet. VAE is used as a dimension reduction technique to capture the latent space, and KRnet is used to model the distribution of the latent variable. Using a linear model between the data and the latent variable, we show that VAE-KRnet can be more effective and robust than the canonical VAE. VAE-KRnet can be used as a density model to approximate either data distribution or an arbitrary probability density function (PDF) known up to a constant. VAE-KRnet is flexible in terms of dimensionality. When the number of dimensions is relatively small, KRnet can effectively approximate the distribution in terms of the original random variable. For high-dimensional cases, we may use VAE-KRnet to incorporate dimension reduction. One important application of VAE-KRnet is the variational Bayes for the approximation of the posterior distribution. The variational Bayes approaches are usually based on the minimization of the Kullback-Leibler (KL) divergence between the model and the posterior. For high-dimensional distributions, it is very challenging to construct an accurate density model due to the curse of dimensionality, where extra assumptions are often introduced for efficiency. For instance, the classical mean-field approach assumes mutual independence between dimensions, which often yields an underestimated variance due to oversimplification. To alleviate this issue, we include into the loss the maximization of the mutual information between the latent random variable and the original random variable, which helps keep more information from the region of low density such that the estimation of variance is improved.

Variational Autoencoder-Based Hybrid Recommendation With Poisson Factorization for Modeling Implicit Feedback

Hybrid recommendation, which is based on collaborative filtering and supplemented with auxiliary content information, is being actively researched due to its ability to overcome the cold-start problem. Many proposed hybrid methods make recommendations using Gaussian distribution-based collaborative filtering even though they handle variables that tend to be non-Gaussian, such as the number of interactions. We present a method that uses a hybrid recommendation framework based on collaborative filtering that models the number of interactions as a Poisson-distributed and variational autoencoder-based content information generation process that shares latent variables with collaborative filtering. As a prior for the shared latent variables, we use a gamma distribution, which is a conjugate prior of a Poisson distribution. An implicit-derivative-based reparameterization trick enables the use of a gamma distribution in a variational autoencoder. The latent variables in the generative model are inferred using the stochastic gradient variational Bayes approach, taking the number of interactions corresponding to users and items and content information as input. In accordance with the inference, unobserved interactions between users and items are predicted for recommendation. The use of a neural-network-based generative model for content information enables the framework to handle various types of content information. Experimental results show that the proposed method utilizes content information effectively for predicting the number of interactions and that it should aid in overcoming the cold-start problem.

Robust adaptive multi‐target tracking with unknown measurement and process noise covariance matrices

A robust adaptive probability hypothesis density (PHD) filter is proposed to address the degradation of PHD performance due to an unknown process noise and measurement noise covariance matrix. Therefore, the inverse Wishart distribution is introduced to model the prior distribution of process noise and measurement noise. Meanwhile, the multi-target posterior intensity is approximated as a mixture of the inverse Wishart distribution and Gaussian distribution. The closed solution of the robust PHD filter is derived by the variational Bayes approach. Simulation results show that the proposed algorithm outperforms the Gaussian-mixed PHD filter and the variational Bayesian PHD filter in terms of target number estimation accuracy and optimal sub-pattern assignment distance.

Evaluating uncertainties in electrochemical impedance spectra of solid oxide fuel cells

Electrochemical impedance spectroscopy (EIS) is a widely used tool for characterization of fuel cells and other electrochemical conversion systems. When applied to the on-line monitoring in the context of in-field applications, the disturbances, drifts and sensor noise may cause severe distortions in the evaluated spectra, especially in the low-frequency part. Failure to ignore the random effects can result in misinterpreted spectra and, consequently, in misleading diagnostic reasoning. This fact has not been often addressed in the research so far. In this paper, we propose an approach to the quantification of the spectral uncertainty, which relies on evaluating the uncertainty of the equivalent circuit model (ECM). We apply the computationally efficient variational Bayes (VB) method and compare the quality of the results with those obtained with the Markov chain Monte Carlo (MCMC) algorithm. Namely, MCMC algorithm returns accurate distributions of the estimated model parameters, while VB approach provides the approximate distributions. By using simulated and real data we show that approximate results provided by VB approach, although slightly over-optimistic, are still close to the more realistic MCMC estimates. A great advantage of the VB method for online monitoring is low computational load, which is several orders of magnitude lower compared to MCMC. The performance of VB algorithm is demonstrated on a case of ECM parameters estimation in a 6 cell solid oxide fuel cell (SOFC) stack. The complete numerical implementation for recreating the results can be found at https://repo.ijs.si/lznidaric/variational-bayes-supplementary-material.

Identification of Wiener–Hammerstein models based on variational bayesian approach in the presence of process noise

This paper addresses the identification of Wiener–Hammerstein (WH) models in the presence of process and measurement noises, which has not been well studied yet in the existing works. To achieve an unbiased estimation, the model parameters are obtained by maximizing the likelihood function, which is solved in the expectation-maximization framework. Due to the difficulty of computing the posterior distributions of the latent variables of WH models, variational Bayes (VB) is used here, and a method for approximating the posterior distributions based on Monte Carlo integral is proposed in VB framework. To the best of our knowledge, it is the first time to use VB approach for WH model identification. Two simulation examples demonstrate the effectiveness of the proposed method. Moreover, the proposed method is used for a WH benchmark problem, and the results show that it improves the identification performance.

Bayesian network-driven clustering analysis with feature selection for high-dimensional multi-modal molecular data

Multi-modal molecular profiling data in bulk tumors or single cells are accumulating at a fast pace. There is a great need for developing statistical and computational methods to reveal molecular structures in complex data types toward biological discoveries. Here, we introduce Nebula, a novel Bayesian integrative clustering analysis for high dimensional multi-modal molecular data to identify directly interpretable clusters and associated biomarkers in a unified and biologically plausible framework. To facilitate computational efficiency, a variational Bayes approach is developed to approximate the joint posterior distribution to achieve model inference in high-dimensional settings. We describe a pan-cancer data analysis of genomic, epigenomic, and transcriptomic alterations in close to 9000 tumor samples across canonical oncogenic signaling pathways, immune and stemness phenotype, with comparisons to state-of-the-art clustering methods. We demonstrate that Nebula has the unique advantage of revealing patterns on the basis of shared pathway alterations, offering biological and clinical insights beyond tumor type and histology in the pan-cancer analysis setting. We also illustrate the utility of Nebula in single cell data for immune cell decomposition in peripheral blood samples.

Robust particle filtering with enhanced outlier resilience and real-time disturbance compensation

This paper is concerned with the state estimation problem for nonlinear/non-Gaussian systems suffered from both time-varying disturbances (TVD) and measurement outliers. Conventional particle filtering (PF) approach can be used to track the non-Gaussian probability density functions, but its sampling efficiency is degraded in the presence TVD. To address this problem, we propose a disturbance observer based PF (DOBPF) method where the knowledge on the dynamics of TVD are fully exploited and real-time disturbance compensation is achieved. Furthermore, to enhance the resilience of our method against outliers, we adopt the skew-t distribution to characterize both skewness and heavy-tailedness of the measurement noise. On this basis, the variational Bayes approach is incorporated into the DOBPF under the marginalization PF framework to infer the noise statistics. Compared with conventional PF approaches, the proposed outlier-resilient DOBPF method exhibits improved resilience against measurement outliers and increased sampling efficiency in the presence of TVD. Simulation and experimental results confirm the effectiveness of the proposed algorithm.

An Adaptive Consensus Filter for Distributed State Estimation With Unknown Noise Statistics

An adaptive consensus filter for sensor networks with unknown process and measurement noise statistics is proposed in this letter. The variational Bayes(VB) approach is exploited to get local estimates of unknown noise covariances with prior inverse Wishart distributions. A distributed averaging approach on exponential-class densities is applied for consensus on the natural parameters of the unknown predicted error covariance. Consensus on measurements is performed in parallel and the two consensus outcomes are fused. Simulation results demonstrate the effectiveness of the proposed adaptive consensus filter compared to conventional, non-adaptive, consensus filters.

Fast Bayesian estimation of spatial count data models

Spatial count data models are used to explain and predict the frequency of phenomena such as traffic accidents in geographically distinct entities such as census tracts or road segments. These models are typically estimated using Bayesian Markov chain Monte Carlo (MCMC) simulation methods, which, however, are computationally expensive and do not scale well to large datasets. Variational Bayes (VB), a method from machine learning, addresses the shortcomings of MCMC by casting Bayesian estimation as an optimisation problem instead of a simulation problem. Considering all these advantages of VB, a VB method is derived for posterior inference in negative binomial models with unobserved parameter heterogeneity and spatial dependence. Pólya-Gamma augmentation is used to deal with the non-conjugacy of the negative binomial likelihood and an integrated non-factorised specification of the variational distribution is adopted to capture posterior dependencies. The benefits of the proposed approach are demonstrated in a Monte Carlo study and an empirical application on estimating youth pedestrian injury counts in census tracts of New York City. The VB approach is around 45 to 50 times faster than MCMC on a regular eight-core processor in a simulation and an empirical study, while offering similar estimation and predictive accuracy. Conditional on the availability of computational resources, the embarrassingly parallel architecture of the proposed VB method can be exploited to further accelerate its estimation by up to 20 times.