Bridgeout: Stochastic Bridge Regularization for Deep Neural Networks

Abstract

A major challenge in training deep neural networks is overfitting , i.e. inferior performance on unseen test examples compared to performance on training examples. To reduce overfitting, stochastic regularization methods have shown superior performance compared to deterministic weight penalties on a number of image recognition tasks. Stochastic methods, such as Dropout and Shakeout, in expectation, are equivalent to imposing a ridge and elastic-net penalty on the model parameters, respectively. However, the choice of the norm of the weight penalty is problem dependent and is not restricted to $\{L_{1},L_{2}\}$ . Therefore, in this paper, we propose the Bridgeout stochastic regularization technique and prove that it is equivalent to an $L_{q}$ penalty on the weights, where the norm $q$ can be learned as a hyperparameter from data. Experimental results show that Bridgeout results in sparse model weights, improved gradients, and superior classification performance compared with Dropout and Shakeout on synthetic and real data sets.