Stochastic Gradient Variational Bayes Research Articles

Auto-Encoding Variational Bayes

This paper employs the Auto-Encoding Variational Bayes (AEVB) estimator based on Stochastic Gradient Variational Bayes (SGVB), designed to optimize recognition models for challenging posterior distributions and large-scale datasets. It has been applied to the mnist dataset and extended to form a Dynamic Bayesian Network (DBN) in the context of time series. The paper delves into Bayesian inference, variational methods, and the fusion of Variational Autoencoders (VAEs) and variational techniques. Emphasis is placed on reparameterization for achieving efficient optimization. AEVB employs VAEs as an approximation for intricate posterior distributions.

A robust generative classifier against transfer attacks based on variational auto-encoders

Deep neural networks (DNNs) are vulnerable to adversarial examples. Even under black-box setting that is without access to the target model, transfer-based attacks can easily fool the DNNs. To alleviate this problem, we propose a robust classification model against transfer attacks based on the framework of variational Auto-Encoders (VAEs) which are probabilistic generative models and have been successfully used to a large mount of tasks. Specifically, our model simulates the data generative process with several multivariate Gaussian distributions and DNNs: (1) We assume that the latent embedding generated by an encoder (a DNN) of each category corresponds to a multivariate Gaussian distribution. (2) A decoder (a DNN) is proposed to decodes the latent embedding into an observable. (3) Theoretical analysis illustrates that our model can predict data’s labels by maximizing the lower bound on the log-likelihood for each category utilizing Bayes’ theorem with excellent robustness against transfer attacks. Inference in our model is done in a variational way so the Stochastic Gradient Variational Bayes (SGVB) estimator and reparamerization trick can be utilized to optimize the evidence lower bound (ELBO). The experiments with quantitative comparisons show that our approach reaches state-of-the-art with significantly better robustness.

Mixture variational autoencoders

Variational autoencoders (VAEs) combine a generative model and a recognition model, and jointly train them to maximize a variational lower bound. VAEs play an important role in unsupervised learning and representation learning. But the isotropic generative model in VAEs cannot sufficiently utilize the latent representative space. In this paper, we present mixture variational autoencoders (MVAEs) which suppose observed data is generated from mixture models. We use a continuous variable with prior of Normal distribution as the latent representation, and introduce a discrete variable with prior of Multinomial distribution as latent indicator for mixture models. Two latent variables are both approximated by recognition models computed from neural networks. Furthermore, we combine the reparameterization trick and stick-breaking parameterization to realize stochastic gradient variational Bayes (SGVB) estimator on our variational objective. The MVAEs improve the generative performance by enlarging the latent representative space. Finally, we demonstrate the performance of MVAEs compared with the state-of-the-art models on MNIST, OMNIGLOT, and Fashion-MNIST.

Collaborative Variational Deep Learning for Healthcare Recommendation

Healthcare recommender system (HRS) has shown the great potential of targeting medical experts or patients, and plays a key role in improving an individual’s health by providing insightful recommendations. The HRSs generate recommendations based on a successful and widely applied method known as collaborative filtering (CF). Despite its success, the CF suffers from data sparsity and cold-start problem, which results in the poor quality of recommendations. In particular, it is a great challenge to seeking information relevant to patients’ condition, and understanding the medical terms and relationships between them in HRSs. To address these problems, we design a novel collaborative variational deep learning model (CVDL) to exploit multi-sourced information for providing appropriate healthcare recommendations in primary care service. CVDL employs additional variational autoencoder (VAE) to learn deep latent representations for item contents (the description of primary care doctors) in latent space, instead of observation space through an inference network. Meanwhile, the CVDL extracts latent user (patient) features by incorporating user profile in a VAE neural network. Therefore, the CVDL can learn better implicit relationships between items and users from item content, user profile, and rating matrix. In addition, a Stochastic Gradient Variational Bayes (SGVB) approach is proposed to calculate the maximum posterior estimates for learning model parameters. The experiments conducted on three datasets have indicated that our method significantly outperforms the state-of-the-art hybrid CF methods.