Gradient boosting Bayesian neural networks via Langevin MCMC

Abstract

Bayesian neural networks harness the power of Bayesian inference which provides an approach to neural learning that not only focuses on accuracy but also uncertainty quantification. Markov Chain Monte Carlo (MCMC) methods implement Bayesian inference by sampling from the posterior distribution of the model parameters. In the case of Bayesian neural networks, the model parameters refer to weights and biases. MCMC methods suffer from scalability issues in large models, such as deep neural networks with thousands to millions of parameters. In this paper, we present a Bayesian ensemble learning framework that utilizes gradient boosting by combining multiple shallow neural networks (base learners) that are trained by MCMC sampling. We present two Bayesian gradient boosting strategies that employ simple neural networks as base learners with Langevin MCMC sampling. We evaluate the performance of these methods on various classification and time-series prediction problems. We demonstrate that the proposed framework improves the prediction accuracy of canonical gradient boosting while providing uncertainty quantification via Bayesian inference. Furthermore, we demonstrate that the respective methods scale well when the size of the dataset and model increases.