Single Hidden Layer Feedforward Network Research Articles

ODOC-ELM: Optimal decision outputs compensation-based extreme learning machine for classifying imbalanced data

Extreme learning machine (ELM) has been one widely used learning paradigm to train single hidden layer feedforward network (SLFN). However, like many other classification algorithms, ELM may learn undesirable class boundaries from data with unbalanced classes. This paper first tries to analyze the reason of the damage caused by class imbalance for ELM, and then discusses the influence of several data distribution factors for the damage. Next, we present an optimal decision outputs compensation strategy to deal with the class imbalance problem in the context of ELM. Specifically, the outputs of the minority classes in ELM are properly compensated. For a binary-class problem, the compensation can be regarded as a single variable optimization problem, thus the golden section search algorithm is adopted to find the optimal compensation value. For a multi-class problem, the particle swarm optimization (PSO) algorithm is used to solve the multivariate optimization problem and to provide the optimal combination of compensations. Experimental results on lots of imbalanced data sets demonstrate the superiority of the proposed algorithm. Statistical results indicate that the proposed approach not only outperforms the original ELM, but also yields better or at least competitive results compared with several widely used and state-of-the-art class imbalance learning methods.

Neural adaptive control of hypersonic aircraft with actuator fault using randomly assigned nodes

This paper investigates the fault-tolerant controller for hypersonic aircraft in case of actuator fault. The robust adaptive controller is designed using command filtered back-stepping scheme. The uncertainty caused by the fault is approximated by randomly assigning nodes of the RBF single-hidden layer feedforward network (SLFN). The output weight is updated based on the Lyapunov synthesis approach to guarantee the stability of the overall control system. The method is applied on the control-oriented model whose subsystems are written into the linearly parameterized form. Simulation results show that the proposed approach achieves good tracking performance.

Adaptive backstepping control for magnetic bearing system via feedforward networks with random hidden nodes

This paper presents an adaptive backstepping neural controller (ABNC) to achieve precise rotor position tracking for a nonlinear active magnetic bearing (AMB) system with modeling uncertainties and external disturbances. In the proposed ABNC, the single-hidden layer feedforward networks (SLFNs) are used to approximate the unknown nonlinearities of dynamic systems and then based on the approximated models, the neural controller is constructed. Different from the existing methods, the hidden node parameters of the SLFNs are determined using the recently proposed neural algorithm named extreme learning machine (ELM), where these parameters are assigned randomly without adjusting. This simplifies the controller design process. Using the Lyapunov theory, stable tuning rules are derived for the update of the output weights of the SLFNs and a proof of stability in the uniformly bounded sense is given for the resulting controller. Moreover, to relax the online computation burden existing in the ABNC, a simplified ABNC (Simpl_ABNC) with less parameters to be adjusted online is proposed to improve the control performance. Finally the simulation results demonstrate that the proposed ABNC and Simpl_ABNC achieve better tracking performance comparing with other controllers including PID controller, conventional backstepping controller and adaptive backstepping sliding mode controller. Also the results show that the Simpl_ABNC has much less computation complexity and also better tracking performance than the ABNC.

Discriminative manifold extreme learning machine and applications to image and EEG signal classification

Extreme learning machine (ELM) uses a non-iterative method to train single-hidden-layer feed-forward networks (SLFNs), which has been proven to be an efficient and effective learning model for both classification and regression. The main advantage of ELM lies in that the input weights as well as the hidden layer biases can be randomly generated, which contributes to the analytical solution of output weights. In this paper, we propose a discriminative manifold ELM (DMELM) by simultaneously considering the discriminative information and geometric structure of data; specifically, we exploit the discriminative information in the local neighborhood around each data point. To this end, a graph regularizer based on a newly designed graph Laplacian to characterize both properties is formulated and incorporated into the ELM objective. In DMELM, the output weights can also be obtained in analytical form. Extensive experiments are conducted on image and EEG signal classification to evaluate the effectiveness of DMELM. The results show that DMELM consistently achieves better performance than original ELM and yields promising results in comparison with several state-of-the-art algorithms, which suggests that both the discriminative as well as manifold information are beneficial to classification.

Manifold learning in local tangent space via extreme learning machine

In this paper, we propose a fast manifold learning strategy to estimate the underlying geometrical distribution and develop the relevant mathematical criterion on the basis of the extreme learning machine (ELM) in the high-dimensional space. The local tangent space alignment (LTSA) method has been used to perform the manifold production and the single hidden layer feedforward network (SLFN) is established via ELM to simulate the low-dimensional representation process. The scheme of the ELM ensemble then combines the individual SLFN for the model selection, where the manifold regularization mechanism has been brought into ELM to preserve the local geometrical structure of LTSA. Some developments have been done to evaluate the inherent representation embedding in the ELM learning. The simulation results have shown the excellent performance in the accuracy and efficiency of the developed approach.

Extreme learning machine based transfer learning for data classification

The extreme learning machine (ELM) is a new method for using Single-hidden Layer Feed-forward Networks (SLFNs) with a much simpler training method. While conventional extreme learning machine are based on the training and test data which should be under the same distribution, in reality it is often desirable to learn an accurate model using only a tiny amount of new data and a large amount of old data. Transfer learning (TL) aims to solve related but different target domain problems by using plenty of labeled source domain data. When the task from one new domain comes, new domain samples are relabeled costly, and it would be a waste to discard all the old domain data. Therefore, an algorithm called TL-ELM based on the ELM algorithm is proposed, which uses a small amount of target domain tag data and a large number of source domain old data to build a high-quality classification model. The method inherits the advantages of ELM and makes up for the defects that traditional ELM cannot transfer knowledge. Experimental results indicate that the performance of the proposed methods is superior to or at least comparable with existing benchmarking methods. In addition, a novel domain adaptation kernel extreme learning machine (TL-DAKELM) based on the kernel extreme learning machine was proposed with respect to the TL-ELM. Experimental results show the effectiveness of the proposed algorithm.

QR factorization based Incremental Extreme Learning Machine with growth of hidden nodes

In this paper, a computationally competitive incremental algorithm based on QR factorization is proposed, to automatically determine the number of hidden nodes in generalized single-hidden-layer feedforward networks (SLFNs). This approach, QR factorization based Incremental Extreme Learning Machine (QRI-ELM), is able to add random hidden nodes to SLFNs one by one. The computational complexity of this approach is analyzed in this paper as well. Simulation results show and verify that our new approach is fast and effective with good generalization and accuracy performance.

A Novel Extreme Learning Control Framework of Unmanned Surface Vehicles.

In this paper, an extreme learning control (ELC) framework using the single-hidden-layer feedforward network (SLFN) with random hidden nodes for tracking an unmanned surface vehicle suffering from unknown dynamics and external disturbances is proposed. By combining tracking errors with derivatives, an error surface and transformed states are defined to encapsulate unknown dynamics and disturbances into a lumped vector field of transformed states. The lumped nonlinearity is further identified accurately by an extreme-learning-machine-based SLFN approximator which does not require a priori system knowledge nor tuning input weights. Only output weights of the SLFN need to be updated by adaptive projection-based laws derived from the Lyapunov approach. Moreover, an error compensator is incorporated to suppress approximation residuals, and thereby contributing to the robustness and global asymptotic stability of the closed-loop ELC system. Simulation studies and comprehensive comparisons demonstrate that the ELC framework achieves high accuracy in both tracking and approximation.

Extreme learning control of surface vehicles with unknown dynamics and disturbances

In this paper, an extreme learning control (ELC) scheme using the single-hidden layer feedforward network (SLFN) for tracking surface vehicles with unknown dynamics and external disturbances is proposed. A sliding surface is defined by incorporating tracking errors and first derivatives, and unknown dynamics including system uncertainties and external disturbances are capsulated into a lumped nonlinearity which is further identified online by the SLFN approximator with random hidden nodes generated by the ELM technique. As a consequence, the SLFN approximator does not require a priori any information on unknown dynamics, and avoids the curse of dimensionality in predefining hidden nodes of high dimension. Not only tracking accuracy but also approximation ability are enhanced by an adaptive compensator for approximation errors in addition to adaptive output weights of the SLFN, which are derived from the Lyapunov synthesis and contribute to global asymptotic stability in terms of tracking errors and first derivatives of the entire closed-loop system. Simulation results and comparative studies demonstrate that the ELC scheme achieves high accuracy of both tracking and approximation.

An integrated chaotic time series prediction model based on efficient extreme learning machine and differential evolution

In this paper, an integrated model based on efficient extreme learning machine (EELM) and differential evolution (DE) is proposed to predict chaotic time series. In the proposed model, a novel learning algorithm called EELM is presented and used to model the chaotic time series. The EELM inherits the basic idea of extreme learning machine (ELM) in training single hidden layer feedforward networks, but replaces the commonly used singular value decomposition with a reduced complete orthogonal decomposition to calculate the output weights, which can achieve a much faster learning speed than ELM. Moreover, in order to obtain a more accurate and more stable prediction performance for chaotic time series prediction, this model abandons the traditional two-stage modeling approach and adopts an integrated parameter selection strategy which employs a modified DE algorithm to optimize the phase space reconstruction parameters of chaotic time series and the model parameter of EELM simultaneously based on a hybrid validation criterion. Experimental results show that the proposed integrated prediction model can not only provide stable prediction performances with high efficiency but also achieve much more accurate prediction results than its counterparts for chaotic time series prediction.

A novel data fusion scheme using grey model and extreme learning machine in wireless sensor networks

With the increasing presence and adoption of wireless sensor networks (WSNs), the demand of data acquisition and data fusion are becoming stronger and stronger. In WSN, sensor nodes periodically sense data and send them to the sink node. Since the network consists of plenty of low-cost sensor nodes with limited battery power and the sensed data usually are of high temporal redundancy, prediction- based data fusion has been put forward as an important issue to reduce the number of transmissions and save the energy of the sensor nodes. Considering the fact that the sensor node usually has limited capabilities of data processing and storage, a novel prediction-based data fusion scheme using grey model (GM) and optimally pruned extreme learning machine (OP-ELM) is proposed. The proposed data fusion scheme called GM-OP-ELM uses a dual prediction mechanism to keep the prediction data series at the sink node and sensor node synchronous. During the data fusion process, GM is introduced to initially predict the data of next period with a small number of data items, and an OPELM- based single-hidden layer feedforward network (SLFN) is used to make the initial predicted value approximate its true value with extremely fast speed. As a robust and fast neural network learning algorithm, OP-ELM can adaptively adjust the structure of the SLFN. Then, GM-OP-ELM can provide high prediction accuracy, low communication overhead, and good scalability. We evaluate the performance of GM-OP-ELM on three actual data sets that collected from 54 sensors deployed in the Intel Berkeley Research lab. Simulation results have shown that the proposed data fusion scheme can significantly reduce redundant transmissions and extend the lifetime of the whole network with low computational cost.

Manifold regularized extreme learning machine

Extreme learning machine (ELM) works for generalized single-hidden-layer feedforward networks (SLFNs), and its essence is that the hidden layer of SLFNs need not be tuned. But ELM only utilizes labeled data to carry out the supervised learning task. In order to exploit unlabeled data in the ELM model, we first extend the manifold regularization (MR) framework and then demonstrate the relation between the extended MR framework and ELM. Finally, a manifold regularized extreme learning machine is derived from the proposed framework, which maintains the properties of ELM and can be applicable to large-scale learning problems. Experimental results show that the proposed semi-supervised extreme learning machine is the most cost-efficient method. It tends to have better scalability and achieve satisfactory generalization performance at a relatively faster learning speed than traditional semi-supervised learning algorithms.

A novel multiple instance learning method based on extreme learning machine.

Since real-world data sets usually contain large instances, it is meaningful to develop efficient and effective multiple instance learning (MIL) algorithm. As a learning paradigm, MIL is different from traditional supervised learning that handles the classification of bags comprising unlabeled instances. In this paper, a novel efficient method based on extreme learning machine (ELM) is proposed to address MIL problem. First, the most qualified instance is selected in each bag through a single hidden layer feedforward network (SLFN) whose input and output weights are both initialed randomly, and the single selected instance is used to represent every bag. Second, the modified ELM model is trained by using the selected instances to update the output weights. Experiments on several benchmark data sets and multiple instance regression data sets show that the ELM-MIL achieves good performance; moreover, it runs several times or even hundreds of times faster than other similar MIL algorithms.

Extreme logistic regression

Kernel logistic regression (KLR) is a very powerful algorithm that has been shown to be very competitive with many state-of the art machine learning algorithms such as support vector machines (SVM). Unlike SVM, KLR can be easily extended to multi-class problems and produces class posterior probability estimates making it very useful for many real world applications. However, the training of KLR using gradient based methods or iterative re-weighted least squares can be unbearably slow for large datasets. Coupled with poor conditioning and parameter tuning, training KLR can quickly design matrix become infeasible for some real datasets. The goal of this paper is to present simple, fast, scalable, and efficient algorithms for learning KLR. First, based on a simple approximation of the logistic function, a least square algorithm for KLR is derived that avoids the iterative tuning of gradient based methods. Second, inspired by the extreme learning machine (ELM) theory, an explicit feature space is constructed through a generalized single hidden layer feedforward network and used for training iterative re-weighted least squares KLR (IRLS-KLR) and the newly proposed least squares KLR (LS-KLR). Finally, for large-scale and/or poorly conditioned problems, a robust and efficient preconditioned learning technique is proposed for learning the algorithms presented in the paper. Numerical results on a series of artificial and 12 real bench-mark datasets show first that LS-KLR compares favorable with SVM and traditional IRLS-KLR in terms of accuracy and learning speed. Second, the extension of ELM to KLR results in simple, scalable and very fast algorithms with comparable generalization performance to their original versions. Finally, the introduced preconditioned learning method can significantly increase the learning speed of IRLS-KLR.

Multiple-kernel-learning-based extreme learning machine for classification design

The extreme learning machine (ELM) is a new method for using single hidden layer feed-forward networks with a much simpler training method. While conventional kernel-based classifiers are based on a single kernel, in reality, it is often desirable to base classifiers on combinations of multiple kernels. In this paper, we propose the issue of multiple-kernel learning (MKL) for ELM by formulating it as a semi-infinite linear programming. We further extend this idea by integrating with techniques of MKL. The kernel function in this ELM formulation no longer needs to be fixed, but can be automatically learned as a combination of multiple kernels. Two formulations of multiple-kernel classifiers are proposed. The first one is based on a convex combination of the given base kernels, while the second one uses a convex combination of the so-called equivalent kernels. Empirically, the second formulation is particularly competitive. Experiments on a large number of both toy and real-world data sets (including high-magnification sampling rate image data set) show that the resultant classifier is fast and accurate and can also be easily trained by simply changing linear program.

Parsimonious extreme learning machine using recursive orthogonal least squares.

Novel constructive and destructive parsimonious extreme learning machines (CP- and DP-ELM) are proposed in this paper. By virtue of the proposed ELMs, parsimonious structure and excellent generalization of multiinput-multioutput single hidden-layer feedforward networks (SLFNs) are obtained. The proposed ELMs are developed by innovative decomposition of the recursive orthogonal least squares procedure into sequential partial orthogonalization (SPO). The salient features of the proposed approaches are as follows: 1) Initial hidden nodes are randomly generated by the ELM methodology and recursively orthogonalized into an upper triangular matrix with dramatic reduction in matrix size; 2) the constructive SPO in the CP-ELM focuses on the partial matrix with the subcolumn of the selected regressor including nonzeros as the first column while the destructive SPO in the DP-ELM operates on the partial matrix including elements determined by the removed regressor; 3) termination criteria for CP- and DP-ELM are simplified by the additional residual error reduction method; and 4) the output weights of the SLFN need not be solved in the model selection procedure and is derived from the final upper triangular equation by backward substitution. Both single- and multi-output real-world regression data sets are used to verify the effectiveness and superiority of the CP- and DP-ELM in terms of parsimonious architecture and generalization accuracy. Innovative applications to nonlinear time-series modeling demonstrate superior identification results.

Adaptive neural control for a class of MIMO nonlinear systems with extreme learning machine

This paper presents two adaptive neural control schemes for a class of uncertain continuous-time multi-input multi-output (MIMO) nonlinear dynamic systems. Within these schemes, the single-hidden layer feedforward networks (SLFNs) are applied to approximate the unknown nonlinear functions of the systems and then the neural controller is built based on the approximated neural models. The parameters of the SLFNs are modified using the recently proposed neural algorithm named extreme learning machine (ELM), where the parameters of the hidden nodes are assigned randomly. Different from the original ELM algorithm, the output weights are updated using the adaptive laws derived based on the Lyapunov stability theorem and Barbalat's lemma so that the asymptotical stability of the system can be guaranteed. The robustifying control term is also constructed to compensate for approximation errors of the SLFNs. In order to avoid the requirement of the approximation error bounds, the estimation laws derived based on the Lyapunov stability theorem and Barbalat's lemma are employed to estimate the error bounds in the second adaptive control scheme. Finally the proposed control schemes are applied to control a two-link robot manipulator. The simulation results demonstrate the effectiveness of the proposed control schemes for the MIMO nonlinear system.

Learning deep representations via extreme learning machines

Extreme learning machine (ELM) as an emerging technology has achieved exceptional performance in large-scale settings, and is well suited to binary and multi-class classification, as well as regression tasks. However, existing ELM and its variants predominantly employ single hidden layer feedforward networks, leaving the popular and potentially powerful stacked generalization principle unexploited for seeking predictive deep representations of input data. Deep architectures can find higher-level representations, thus can potentially capture relevant higher-level abstractions. But most of current deep learning methods require solving a difficult and non-convex optimization problem. In this paper, we propose a stacked model, DrELM, to learn deep representations via extreme learning machine according to stacked generalization philosophy. The proposed model utilizes ELM as a base building block and incorporates random shift and kernelization as stacking elements. Specifically, in each layer, DrELM integrates a random projection of the predictions obtained by ELM into the original feature, and then applies kernel functions to generate the resultant feature. To verify the classification and regression performance of DrELM, we conduct the experiments on both synthetic and real-world data sets. The experimental results show that DrELM outperforms ELM and kernel ELMs, which appear to demonstrate that DrELM could yield predictive features that are suitable for prediction tasks. The performances of the deep models (i.e. Stacked Auto-encoder) are comparable. However, due to the utilization of ELM, DrELM is easier to learn and faster in testing.

OMP-ELM: Orthogonal Matching Pursuit-Based Extreme Learning Machine for Regression

AbstractExtreme learning machine (ELM) is a recent scheme for single hidden layer feed forward networks (SLFNs). It has attracted much interest in the machine intelligence and pattern recognition fields with numerous real-world applications. The ELM structure has several advantages, such as its adaptability to various problems with a rapid learning rate and low computational cost. However, it has shortcomings in the following aspects. First, it suffers from the irrelevant variables in the input data set. Second, choosing the optimal number of neurons in the hidden layer is not well defined. In case the hidden nodes are greater than the training data, the ELM may encounter the singularity problem, and its solution may become unstable. To overcome these limitations, several methods have been proposed within the regularization framework. In this article, we considered a greedy method for sparse approximation of the output weight vector of the ELM network. More specifically, the orthogonal matching pursuit (OMP) algorithm is embedded to the ELM. This new technique is named OMP-ELM. OMP-ELM has several advantages over regularized ELM methods, such as lower complexity and immunity to the singularity problem. Experimental works on nine commonly used regression problems indicate that the investigated OMP-ELM method confirms these advantages. Moreover, OMP-ELM is compared with the ELM method, the regularized ELM scheme, and artificial neural networks.

Generalized single-hidden layer feedforward networks for regression problems.

In this paper, traditional single-hidden layer feedforward network (SLFN) is extended to novel generalized SLFN (GSLFN) by employing polynomial functions of inputs as output weights connecting randomly generated hidden units with corresponding output nodes. The significant contributions of this paper are as follows: 1) a primal GSLFN (P-GSLFN) is implemented using randomly generated hidden nodes and polynomial output weights whereby the regression matrix is augmented by full or partial input variables and only polynomial coefficients are to be estimated; 2) a simplified GSLFN (S-GSLFN) is realized by decomposing the polynomial output weights of the P-GSLFN into randomly generated polynomial nodes and tunable output weights; 3) both P- and S-GSLFN are able to achieve universal approximation if the output weights are tuned by ridge regression estimators; and 4) by virtue of the developed batch and online sequential ridge ELM (BR-ELM and OSR-ELM) learning algorithms, high performance of the proposed GSLFNs in terms of generalization and learning speed is guaranteed. Comprehensive simulation studies and comparisons with standard SLFNs are carried out on real-world regression benchmark data sets. Simulation results demonstrate that the innovative GSLFNs using BR-ELM and OSR-ELM are superior to standard SLFNs in terms of accuracy, training speed, and structure compactness.