GA-SELM: Greedy algorithms for sparse extreme learning machine

Abstract

In the last decade, extreme learning machine (ELM), which is a new learning algorithm for single-hidden layer feed forward networks (SLFNs), has gained much attention in the machine intelligence and pattern recognition communities with numerous successful real-world applications. The ELM structure has several advantageous such as good generalization performance with an extremely fast learning speed and low computational cost especially when dealing with many patterns defined in a high-dimensional space. However, three major problems usually appear using the ELM structure: (i) the dataset may have irrelevant variables, (ii) choosing the number of neurons in the hidden layer would be difficult, and (iii) it may encounter the singularity problem. To overcome these limitations, several methods have been proposed in the regularization framework. In this paper, we propose several sparse ELM schemes in which various greedy algorithms are used for sparse approximation of the output weights vector of the ELM network. In short, we name these new schemes as GA-SELM. We also investigate several greedy algorithms such as Compressive Sampling Matching Pursuit (CoSaMP), Iterative Hard Thresholding (IHT), Orthogonal Matching Pursuit (OMP) and Stagewise Orthogonal Matching Pursuit (StOMP) to obtain a regularized ELM scheme. These new ELM schemes have several benefits in comparing with the traditional ELM schemes such as low computational complexity, being free of parameter adjustment and avoiding the singularity problem. The proposed approach shows its significant advantages when it is compared with the empirical studies on nine commonly used regression benchmarks. Moreover, a comparison with the original ELM and the regularized ELM schemes is performed.