Variational Bayes Estimator Research Articles

Auto-Encoding Variational Bayes

This paper employs the Auto-Encoding Variational Bayes (AEVB) estimator based on Stochastic Gradient Variational Bayes (SGVB), designed to optimize recognition models for challenging posterior distributions and large-scale datasets. It has been applied to the mnist dataset and extended to form a Dynamic Bayesian Network (DBN) in the context of time series. The paper delves into Bayesian inference, variational methods, and the fusion of Variational Autoencoders (VAEs) and variational techniques. Emphasis is placed on reparameterization for achieving efficient optimization. AEVB employs VAEs as an approximation for intricate posterior distributions.

Anomaly Detection of Time Series With Smoothness-Inducing Sequential Variational Auto-Encoder.

Deep generative models have demonstrated their effectiveness in learning latent representation and modeling complex dependencies of time series. In this article, we present a smoothness-inducing sequential variational auto-encoder (VAE) (SISVAE) model for the robust estimation and anomaly detection of multidimensional time series. Our model is based on VAE, and its backbone is fulfilled by a recurrent neural network to capture latent temporal structures of time series for both the generative model and the inference model. Specifically, our model parameterizes mean and variance for each time-stamp with flexible neural networks, resulting in a nonstationary model that can work without the assumption of constant noise as commonly made by existing Markov models. However, such flexibility may cause the model fragile to anomalies. To achieve robust density estimation which can also benefit detection tasks, we propose a smoothness-inducing prior over possible estimations. The proposed prior works as a regularizer that places penalty at nonsmooth reconstructions. Our model is learned efficiently with a novel stochastic gradient variational Bayes estimator. In particular, we study two decision criteria for anomaly detection: reconstruction probability and reconstruction error. We show the effectiveness of our model on both synthetic data sets and public real-world benchmarks.

Sparse Modeling for Cross-Industry Returns Predictability

We develop a new variational Bayes estimation method for large-dimensional sparse vector autoregressive models with exogenous predictors. Unlike existing Markov chain Monte Carlo (MCMC) and variational Bayes (VB) algorithms, our approach is not based on a structural form representation of the model, but directly shrinkage the lead-lag cross sectional interdependencies in the transition matrix. We compare the performance of our approach against state-of-the-art MCMC and VB methods, both in simulation and when forecasting a large cross section of industry portfolios spanning almost a hundred years of monthly data. The main results provide robust evidence that by directly shrinking weak cross sectional signals one can substantially improve both the statistical and economic out-of-sample performance of multivariate models for return predictability. This result holds across a variety of alternative shrinkage priors, such as Bayesian adaptive lasso, normal-gamma and horseshoe priors.

A robust generative classifier against transfer attacks based on variational auto-encoders

Deep neural networks (DNNs) are vulnerable to adversarial examples. Even under black-box setting that is without access to the target model, transfer-based attacks can easily fool the DNNs. To alleviate this problem, we propose a robust classification model against transfer attacks based on the framework of variational Auto-Encoders (VAEs) which are probabilistic generative models and have been successfully used to a large mount of tasks. Specifically, our model simulates the data generative process with several multivariate Gaussian distributions and DNNs: (1) We assume that the latent embedding generated by an encoder (a DNN) of each category corresponds to a multivariate Gaussian distribution. (2) A decoder (a DNN) is proposed to decodes the latent embedding into an observable. (3) Theoretical analysis illustrates that our model can predict data’s labels by maximizing the lower bound on the log-likelihood for each category utilizing Bayes’ theorem with excellent robustness against transfer attacks. Inference in our model is done in a variational way so the Stochastic Gradient Variational Bayes (SGVB) estimator and reparamerization trick can be utilized to optimize the evidence lower bound (ELBO). The experiments with quantitative comparisons show that our approach reaches state-of-the-art with significantly better robustness.

Variational Bayes Inference for the DINA Model

In this article, we propose a variational Bayes (VB) inference method for the deterministic input noisy AND gate model of cognitive diagnostic assessment. The proposed method, which applies the iterative algorithm for optimization, is derived based on the optimal variational posteriors of the model parameters. The proposed VB inference enables much faster computation than the existing Markov chain Monte Carlo (MCMC) method, while still offering the benefits of a full Bayesian framework. A simulation study revealed that the proposed VB estimation adequately recovered the parameter values. Moreover, an example using real data revealed that the proposed VB inference method provided similar estimates to MCMC estimation with much faster computation.

Bayesian estimation of mixed multinomial logit models: Advances and simulation-based evaluations

Variational Bayes (VB) methods have emerged as a fast and computationally-efficient alternative to Markov chain Monte Carlo (MCMC) methods for scalable Bayesian estimation of mixed multinomial logit (MMNL) models. It has been established that VB is substantially faster than MCMC at practically no compromises in predictive accuracy. In this paper, we address two critical gaps concerning the usage and understanding of VB for MMNL. First, extant VB methods are limited to utility specifications involving only individual-specific taste parameters. Second, the finite-sample properties of VB estimators and the relative performance of VB, MCMC and maximum simulated likelihood estimation (MSLE) are not known. To address the former, this study extends several VB methods for MMNL to admit utility specifications including both fixed and random utility parameters. To address the latter, we conduct an extensive simulation-based evaluation to benchmark the extended VB methods against MCMC and MSLE in terms of estimation times, parameter recovery and predictive accuracy. The results suggest that all VB variants with the exception of the ones relying on an alternative variational lower bound constructed with the help of the modified Jensen’s inequality perform as well as MCMC and MSLE at prediction and parameter recovery. In particular, VB with nonconjugate variational message passing and the delta-method (VB-NCVMP-Δ) is up to 16 times faster than MCMC and MSLE. Thus, VB-NCVMP-Δ can be an attractive alternative to MCMC and MSLE for fast, scalable and accurate estimation of MMNL models.

Mixture variational autoencoders

Variational autoencoders (VAEs) combine a generative model and a recognition model, and jointly train them to maximize a variational lower bound. VAEs play an important role in unsupervised learning and representation learning. But the isotropic generative model in VAEs cannot sufficiently utilize the latent representative space. In this paper, we present mixture variational autoencoders (MVAEs) which suppose observed data is generated from mixture models. We use a continuous variable with prior of Normal distribution as the latent representation, and introduce a discrete variable with prior of Multinomial distribution as latent indicator for mixture models. Two latent variables are both approximated by recognition models computed from neural networks. Furthermore, we combine the reparameterization trick and stick-breaking parameterization to realize stochastic gradient variational Bayes (SGVB) estimator on our variational objective. The MVAEs improve the generative performance by enlarging the latent representative space. Finally, we demonstrate the performance of MVAEs compared with the state-of-the-art models on MNIST, OMNIGLOT, and Fashion-MNIST.

Neural variational matrix factorization for collaborative filtering in recommendation systems

Matrix factorization as a popular technique for collaborative filtering in recommendation systems computes the latent factors for users and items by decomposing a user-item rating matrix. Most matrix factorization methods including probabilistic matrix factorization that projects (parameterized) users and items probabilistic matrices to maximize their inner product suffer from data sparsity and result in poor latent representations of users and items. To alleviate these problems, we propose a novel deep generative model, namely Neural Variational Matrix Factorization, that incorporates side information (features) of both users and items to capture better latent representations of them for more effective collaborative-filtering recommendation. Our model consists of two end-to-end variational autoencoder neural networks, namely user neural network and item neural network respectively, that are capable of learning complex nonlinear distributed representations of users and items through our proposed variational inference. We present a Stochastic Gradient Variational Bayes estimator to estimate the intractable posterior distributions of latent factors of users and items and parameters of our model, and derive the variational evidence lower bounds of the model. Experiments conducted on three publicly available datasets show that our model significantly outperforms the state-of-the-art methods on recommendation accuracy measured by Hit Ratio and Normalized Discounted Cumulative Gain respectively.

Variational Bayes Estimation of Discrete-Margined Copula Models With Application to Time Series

ABSTRACTWe propose a new variational Bayes (VB) estimator for high-dimensional copulas with discrete, or a combination of discrete and continuous, margins. The method is based on a variational approximation to a tractable augmented posterior and is faster than previous likelihood-based approaches. We use it to estimate drawable vine copulas for univariate and multivariate Markov ordinal and mixed time series. These have dimension rT, where T is the number of observations and r is the number of series, and are difficult to estimate using previous methods. The vine pair-copulas are carefully selected to allow for heteroscedasticity, which is a feature of most ordinal time series data. When combined with flexible margins, the resulting time series models also allow for other common features of ordinal data, such as zero inflation, multiple modes, and under or overdispersion. Using six example series, we illustrate both the flexibility of the time series copula models and the efficacy of the VB estimator for copulas of up to 792 dimensions and 60 parameters. This far exceeds the size and complexity of copula models for discrete data that can be estimated using previous methods. An online appendix and MATLAB code implementing the method are available as supplementary materials.

Uncertainty Estimation in the Neural Model for Aeromagnetic Compensation

Measuring the performance of an aeromagnetic compensation system is usually difficult. The standard deviation of the signal has been used as an index in the industry. While the standard deviation is drawn from frequency statistics, it cannot represent the performance on a single sampling point. On the other hand, as the true geomagnetic intensity is unknown, the signal’s deviation is actually an approximate measurement of the residual error. This letter first analyzes the traditional neural model for aeromagnetic compensation to reveal the fact that the model can only estimate the expectation of interference. Then, we introduce a stochastic hidden variable to predict the standard deviation synchronously. The proposed model is derived from variational inference and trained as a stochastic gradient variational Bayes estimator. Simulations are performed to show the correlation between the true residual error and the estimated standard deviation.

Image Denoising method based on NSCT bivariate model and Variational Bayes threshold estimation

In order to reduce the Gaussian noise introduced during image generation, this paper presents an image denoising algorithm based on variational Bayes (V-Bayes) estimation and nonsubsampled contourlet transform (NSCT) bivariate model. First, the proposed algorithm uses the NSCT’s advantages of translation-invariant and multidirection-selectivity, exploits the intra-scale and inter-scale correlations of NSCT coefficients. Then, the corresponding nonlinear bivariate threshold function of the model is deduced by using V-Bayes estimation theory. Finally, the noise-reduced coefficients are inverse-transformed by NSCT to obtain denoised image. The simulation results show that the denoised image has obvious improvement in subjective visual effects and performance indicators, and effectively preserves the details and texture information in the original image.

Fast and scalable variational Bayes estimation of spatial econometric models for Gaussian data

Spatial econometric models have been widely used for analyzing cross-sectional data in which spatial dependence is of primary interest. Although proven successful, Bayesian estimation via Markov chain Monte Carlo (MCMC) for spatial econometric models can be computationally demanding as the size of data and complexity of models grow. This paper proposes two variational Bayes methods that are more scalable and computationally faster in estimating general spatial autoregressive and matrix exponential spatial specification models: the hybrid mean-field variational Bayes (MFVB) method and the integrated nonfactorized variational Bayes (INFVB) method. The hybrid MFVB method assumes posterior independence and, when applicable, can yield accurate results but tends to underestimate posterior variances. In comparison, the INFVB method provides more robust results by accounting for posterior dependence and is computationally appealing due to parallelization. We demonstrate that variational Bayesian inference can be a faster and more scalable alternative to the MCMC approach for Bayesian spatial econometric modeling. The effectiveness of our proposed methods for spatial econometric models is demonstrated through simulated examples and a real-world data application.

Variational Bayes Estimation of Time Series Copulas for Multivariate Ordinal and Mixed Data

We propose a new variational Bayes method for estimating high-dimensional copulas with discrete, or discrete and continuous, margins. The method is based on a variational approximation to a tractable augmented posterior, and is substantially faster than previous likelihood-based approaches. We use it to estimate drawable vine copulas for univariate and multivariate Markov ordinal and mixed time series. These have dimension rT, where T is the number of observations and r is the number of series, and are difficult to estimate using previous methods. The vine pair-copulas are carefully selected to allow for heteroskedasticity, which is a common feature of ordinal time series data. When combined with flexible margins, the resulting time series models also allow for other common features of ordinal data, such as zero inflation, multiple modes and under- or over-dispersion. Using data on homicides in New South Wales, and also U.S. bankruptcies, we illustrate both the flexibility of the time series copula models, and the efficacy of the variational Bayes estimator for copulas of up to 792 dimensions and 60 parameters. This far exceeds the size and complexity of copula models for discrete data that can be estimated using previous methods.

Fast Bayesian whole-brain fMRI analysis with spatial 3D priors

Spatial whole-brain Bayesian modeling of task-related functional magnetic resonance imaging (fMRI) is a great computational challenge. Most of the currently proposed methods therefore do inference in subregions of the brain separately or do approximate inference without comparison to the true posterior distribution. A popular such method, which is now the standard method for Bayesian single subject analysis in the SPM software, is introduced in Penny et al. (2005b). The method processes the data slice-by-slice and uses an approximate variational Bayes (VB) estimation algorithm that enforces posterior independence between activity coefficients in different voxels. We introduce a fast and practical Markov chain Monte Carlo (MCMC) scheme for exact inference in the same model, both slice-wise and for the whole brain using a 3D prior on activity coefficients. The algorithm exploits sparsity and uses modern techniques for efficient sampling from high-dimensional Gaussian distributions, leading to speed-ups without which MCMC would not be a practical option. Using MCMC, we are for the first time able to evaluate the approximate VB posterior against the exact MCMC posterior, and show that VB can lead to spurious activation. In addition, we develop an improved VB method that drops the assumption of independent voxels a posteriori. This algorithm is shown to be much faster than both MCMC and the original VB for large datasets, with negligible error compared to the MCMC posterior.

Variational Bayes Group Sparse Time-Adaptive Parameter Estimation With Either Known or Unknown Sparsity Pattern

In this paper, we study the problem of time-adaptive group sparse signal estimation from a Bayesian viewpoint. We propose two online variational Bayes schemes that are specifically designed to estimate and track group sparse signals in time. The proposed schemes address both the cases where the grouping information of the signal is either known or not. For the case of known group sparsity pattern, the proposed scheme builds on a novel hierarchical model for the Bayesian adaptive group lasso. Utilizing the variational Bayes framework, update equations for all model parameters are given, for both the batch and time adaptive estimation scenarios. To address the case where the group sparsity pattern is unknown, the hierarchical Bayesian model of the former scheme is extended by organizing the penalty parameters of the Bayesian lasso in a conditional autoregressive model. Intrinsic conditional autoregression is exploited to penalize the signal coefficients in a structured manner and thus obtain group sparse solutions automatically. Again, a robust and computationally efficient online variational Bayes estimator is developed, capitalizing on the conjugacy of the proposed hierarchical Bayesian formulation. Experimental results are reported that corroborate the superior estimation performance of the proposed online schemes, when compared with state-of-the-art methods.

Model-based extraction of input and organ functions in dynamic scintigraphic imaging

Image-based definition of input function (IF) and organ function is a prerequisite for kinetic analysis of dynamic scintigraphy or positron emission tomography. This task is typically done manually by a human operator and suffers from low accuracy and reproducibility. We propose a probabilistic model based on physiological assumption that time–activity curves (TACs) arise as a convolution of an IF and tissue-specific kernels. The model is solved via the Variational Bayes estimation procedure and provides estimates of the IF, tissue-specific TACs and their related spatial distributions (images) as its results. The algorithm was tested with data of dynamic renal scintigraphy. The method was applied to the problem of differential renal function estimation and the IF estimation and the results are compared with competing techniques on data-sets with 99 and 19 patients. The MATLAB implementation of the algorithm is available for download.

Lack of Consistency of Mean Field and Variational break Bayes Approximations for State Space Models

The consistency problem of both mean field and variational Bayes estimators in the context of linear state space models is investigated. We prove that the mean field approximation is asymptotically consistent when the variances of the noise variables in the system are sufficiently small, but neither the mean field estimator nor the variational Bayes estimator is always asymptotically consistent as the 'sample size' becomes large. The 'gap' between the estimators and the true values is roughly estimated.