A new pruning algorithm for extreme learning machine

Abstract

This paper presents a new pruning extreme learning machine (N-PELM) algorithm which can generate a compact single-hidden-layer neural network (SLNN) by automatically pruning the number of hidden nodes while keep high accuracy. The proposed N-PELM algorithm initializes a SLNN by using extreme learning machine (ELM) algorithm given superfluous number of hidden nodes. The following part consists of two iterative processes. First, the significance of the hidden node is estimated and the insignificant node is removed. Secondly, once the node be removed, the existing hidden nodes are updated using ELM algorithm. These two processes continue until all of each hidden nodes is estimated or the number of the hidden nodes is small enough. Compared against other neural network algorithms, N-PELM algorithm has mainly three improvements. Firstly, the significance of hidden nodes is estimated in output layer such that the relevance of hidden nodes and classes can be estimated more precisely. Secondly, the pruning threshold is selected automatically from a base set of potential relevance threshold values using Akaike information criterion (AIC) such that the threshold can accommodate any data type. Thirdly, P-ELM uses Kullback-Leibler (KL) divergence and Jensen-Shannon (JS) divergence to measure the significance of hidden nodes. Experimental results have shown that the P-ELM algorithm can automatically achieve a reasonable compact network structure while keep comparable or much higher accuracy in classification and regression.