Alpha-divergence minimization with mixed variational posterior for Bayesian neural networks and its robustness against adversarial examples

Abstract

Listen

Translate article

In the approximate inference of Bayesian neural networks (BNNs), the variational posterior distribution is often taken an exponential family form (such as Gaussian). We propose to make the mixtures of exponential family distributions instead to get a more flexible approximation posterior. A novel reparameterization trick is introduced in this paper, in order to apply the reparameterization trick to mixed density distributions in Alpha divergence minimization. Our method is extendable to various neural architectures such as fully-connected neural networks and convolutional neural networks. The analysis on time complexity demonstrates that our method has less computation-consuming than normalizing flows. It also outperforms some related state-of-the-art techniques in the experiments of uncertainty estimation and robustness against adversarial examples.