Iterative minimal residual method provides optimal regularization parameter for extreme learning machines

Abstract

Abstract Extreme Learning Machine (ELM) is a single hidden layer feed-forward neural network with the learning speed is much faster than the traditional neural network architecture. The main reason is attributed to the application of slow gradient-based algorithms, and the model parameter of the networks is estimated iteratively and hence, ELM has been widely used in real-life applications due to its fast and easy way of learning. The ill-conditioning of the input-to–hidden layer matrix reduces the stability of ELM and the regularization is employed to overcome this problem. In general, Tikhonov ( l 2 -norm based) and Lasso ( l 1 -norm based) techniques are widely used regularization techniques applied in the current work. The choice of the regularization parameter affects the performance of either of the regularized ELM. The singular value decomposition based generalized cross-validation method is also used in the literature for automatically estimating the value of the regularization parameter. This method increases the computational complexity, as it involves the repeated inversion of a large matrix. In the proposed methods, the regularized Minimum Residual Method (MRM) is adopted along with the Golden-section line search to find the optimal value of the regularization parameter. One of the important contributions of the proposed work is to suitably modify MRM to estimate the regularization parameter for the Lasso technique. MT-ELM is a l 2 -norm regularization based ELM method where the regularization parameter is estimated using MRM. Similarly, ML-ELM is a l 1 -norm regularization based ELM method where the regularization parameter is estimated using a suitable MRM. In order to evaluate the performance of the proposed algorithms, experiments are carried out on several benchmark classification data sets and are compared on several performance metrics such as accuracy, precision, recall, and F 1 -score. Through the extensive numerical results and floating point operation counts, it is shown that the proposed MRM-based methods are capable of estimating the regularization parameter optimally compare to the existing ones. It is also observed that the proposed methods improve the accuracy measure by seven per cent compared to the conventional ELM, thus enhancing the capability of ELM. Further, the proposed methods are verified statistically.