Investigation of activation functions in deep belief network

Abstract

Deep Belief Network (DBN) is made up of stacked Restricted Boltzmann Machine layers associated with global weight fine-tuning for pattern recognition. However, DBN suffers from vanishing gradient problem due to the saturation characteristic of activation function. Therefore, the selection of activation function in DBN is critical to reduce the network complexity and improve performance of pattern recognition. Unsaturated activation functions such as rectified linear unit and leaky rectified linear unit were recently proposed to avoid the effect of vanishing gradient for a deep learning neural network. In this paper, we investigated the network performance with both saturated and unsaturated activation functions. Besides that, the randomization of training samples would significantly improve the performance of DBN. The experimental results showed that hyperbolic tangent activation function achieved the lowest error rate which is 1.99% on MNIST handwritten digit dataset.