An Effective Bayesian Neural Network Classifier with a Comparison Study to Support Vector Machine

Abstract

We propose a new Bayesian neural network classifier, different from that commonly used in several respects, including the likelihood function, prior specification, and network structure. Under regularity conditions, we show that the decision boundary determined by the new classifier will converge to the true one. We also propose a systematic implementation for the new classifier. In our implementation, the tune of connection weights, the selection of hidden units, and the selection of input variables are unified by sampling from the joint posterior distribution of the network structure and connection weights. The numerical results show that the new classifier consistently outperforms the commonly used Bayesian neural network classifier and the support vector machine in terms of generalization performance. The reason for the inferiority of the commonly used Bayesian neural network classifier and the support vector machine is discussed at length.