Abstract
In this paper, we propose a group Lasso regularization term as a hidden layer regularization method for feedforward neural networks. Adding a group Lasso regularization term into the standard error function as a hidden layer regularization term is a fruitful approach to eliminate the redundant or unnecessary hidden layer neurons from the feedforward neural network structure. As a comparison, a popular Lasso regularization method is introduced into standard error function of the network. Our novel hidden layer regularization method can force a group of outgoing weights to become smaller during the training process and can eventually be removed after the training process. This means it can simplify the neural network structure and it minimizes the computational cost. Numerical simulations are provided by using K-fold cross-validation method with K = 5 to avoid overtraining and to select the best learning parameters. The numerical results show that our proposed hidden layer regularization method prunes more redundant hidden layer neurons consistently for each benchmark dataset without loss of accuracy. In contrast, the existing Lasso regularization method prunes only the redundant weights of the network, but it cannot prune any redundant hidden layer neurons.
Highlights
Artificial Neural Networks (ANNs) are pretty old ideas to mimic the human brain [1]
We present the numerical results of our proposed batch gradient method with a group Lasso regularization term (BGGLasso) compared to the existing learning methods (i.e., standard batch gradient method without any regularization term (BG) and batch gradient method with a
To check the sparsity ability of the group Lasso and Lasso regularization terms using the zoo dataset, we randomly started with big neural network structure 18 − 26 − 7, including the bias neurons in the input and hidden layers
Summary
ANNs have been intensively studied for many years in the hope of achieving human-like performance in the fields of speech and image recognition [2]. They are designed to solve a variety of problems in the area of pattern recognition, prediction, optimization, associative memory, and control [3]. In FNNs, the neurons are arranged in the form of layers, namely input, hidden and output layers and there exist the connections between the neurons of one layer to those of the layer [7] These connections are repeatedly adjusted to minimize a measure of the difference between the actual network output vector and the desired output vector through the learning process [8]. The FNNs are commonly trained by using the backpropagation (BP) algorithm which uses a gradient descent learning method, called the steepest descent [9]
Sign-in/Register to view highlights & summary Sign-in/Register to access full text options 
Free) 
Talk to us
Join us for a 30 min session where you can share your feedback and ask us any queries you have
Schedule a call
