An Analysis of the Regularization Between L2 and Dropout in Single Hidden Layer Neural Network

Abstract

In supervised learning, one of the motivations and intuitions behind the design of the learning algorithm is to prevent an overfitting. Overfitting usually occurs when a learning model is excessively complex, especially when having too many parameters to adjust. A typical approach to alleviate this problem is to introduce a form of the penalty term in the objective function, which is called regularization, for preventing the network parameters from growing too large. Another proposed strategy known as dropout is introduced to prevent co-adaptation on each training data by randomly dropping some units out during the training process. In this paper, an analysis of different regularization techniques between L2-norm and dropout in a single hidden layer neural networks are investigated on the MNIST dataset. In our experiment, both regularization methods are applied to the single hidden layer neural network with various scales of network complexity. The results show that dropout is more effective than L2-norm for complex networks i.e., containing large numbers of hidden neurons. The results of this study are helpful to design the neural networks with suitable choice of regularization.