Abstract
In order to prevent the overfitting and improve the generalization performance of Extreme Learning Machine (ELM), a new regularization method, Biased DropConnect, and a new regularized ELM using the Biased DropConnect and Biased Dropout (BD-ELM) are both proposed in this paper. Like the Biased Dropout to hidden nodes, the Biased DropConnect can utilize the difference of connection weights to keep more information of network after dropping. The regular Dropout and DropConnect set the connection weights and output of the hidden layer to 0 with a single fixed probability. But the Biased DropConnect and Biased Dropout divide the connection weights and hidden nodes into high and low groups by threshold, and set different groups to 0 with different probabilities. Connection weights with high value and hidden nodes with a high-activated value, which make more contribution to network performance, will be kept by a lower drop probability, while the weights and hidden nodes with a low value will be given a higher drop probability to keep the drop probability of the whole network to a fixed constant. Using Biased DropConnect and Biased Dropout regularization, in BD-ELM, the sparsity of parameters is enhanced and the structural complexity is reduced. Experiments on various benchmark datasets show that Biased DropConnect and Biased Dropout can effectively address the overfitting, and BD-ELM can provide higher classification accuracy than ELM, R-ELM, and Drop-ELM.
Highlights
Extreme Learning Machine (ELM) [1], as the latest research achievement of Single-hidden Layer Feedforward Neural Networks (SLFNs), has attracted much attention due to its good generalization performance and fast training speed
TROP-ELM adopted both L1-norm and L2-norm regularization methods, and its generalization performance was significantly improved compared with ELM and OP-ELM
Inspired by Biased Dropout, this paper proposes the Biased DropConnect, which divides the connection weights into high and low groups by setting threshold and gives different drop probabilities. en, applying the Biased DropConnect and Biased Dropout to the ELM, this paper proposes a Biased DropConnect and Biased Dropout based ELM (BD-ELM). e empirical studies show that Biased DropConnect and Biased Dropout can effectively overcome the overfitting of ELM, and compared with ELM, R-ELM, and Drop-ELM, BD-ELM can get higher classification accuracy on various benchmark datasets
Summary
Extreme Learning Machine (ELM) [1], as the latest research achievement of Single-hidden Layer Feedforward Neural Networks (SLFNs), has attracted much attention due to its good generalization performance and fast training speed. Various regularization methods are applied to the ELM algorithm to address the overfitting [6,7,8,9]. A Regularized ELM (RELM) based on L2-norm loss was proposed by Deng et al [7], which avoids overfitting and improves the robustness of the algorithm. TROP-ELM adopted both L1-norm and L2-norm regularization methods, and its generalization performance was significantly improved compared with ELM and OP-ELM. Inspired by Biased Dropout, this paper proposes the Biased DropConnect, which divides the connection weights into high and low groups by setting threshold and gives different drop probabilities. Inspired by Biased Dropout, this paper proposes the Biased DropConnect, which divides the connection weights into high and low groups by setting threshold and gives different drop probabilities. en, applying the Biased DropConnect and Biased Dropout to the ELM, this paper proposes a Biased DropConnect and Biased Dropout based ELM (BD-ELM). e empirical studies show that Biased DropConnect and Biased Dropout can effectively overcome the overfitting of ELM, and compared with ELM, R-ELM, and Drop-ELM, BD-ELM can get higher classification accuracy on various benchmark datasets
Sign-in/Register to view highlights & summary Sign-in/Register to access full text options 
Free) 
Talk to us
Join us for a 30 min session where you can share your feedback and ask us any queries you have
Schedule a call
