Layerwise Approximate Inference for Bayesian Uncertainty Estimates on Deep Neural Networks

Abstract

A proper representation of predictive uncertainty is vital for deep neural networks (DNNs) to be applied in safety-critical domains such as medical diagnosis and self-driving. State-of-the-art (SOTA) variational inference approximation techniques provide a theoretical framework for modeling uncertainty, however, they have not been proven to work on large and deep networks with practical computation. In this study, we develop a layerwise approximation with a local reparameterization technique to efficiently perform sophisticated variational Bayesian inference on very deep SOTA convolutional neural networks (CNNs) (VGG16, ResNet variants, DenseNet). Theoretical analysis is presented to justify that the layerwise approach remains a Bayesian neural network. We further derive a SOTA <tex xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">$\alpha$</tex> -divergence objective function to work with the layerwise approximate inference, addressing the concern of underestimating uncertainties by the Kullback-Leibler divergence. Empirical evaluation using MNIST, CIFAR-10, and CIFAR-100 datasets consistently shows that with our proposal, deep CNN models can have a better quality of predictive uncertainty than Monte Carlo-dropout in detecting in-domain misclassification and excel in out-of-distribution detection.