Hidden Layer Feedforward Neural Network Research Articles

Dolphin Swarm Extreme Learning Machine

As a novel learning algorithm for a single hidden-layer feedforward neural network, the extreme learning machine has attracted much research attention for its fast training speed and good generalization performances. Instead of iteratively tuning the parameters, the extreme machine can be seen as a linear optimization problem by randomly generating the input weights and hidden biases. However, the random determination of the input weights and hidden biases may bring non-optimal parameters, which have a negative impact on the final results or need more hidden nodes for the neural network. To overcome the above drawbacks caused by the non-optimal input weights and hidden biases, we propose a new hybrid learning algorithm named dolphin swarm algorithm extreme learning machine adopting the dolphin swarm algorithm to optimize the input weights and hidden biases efficiently. Each set of input weights and hidden biases is encoded into one vector, namely the dolphin. The dolphins are evaluated by root mean squared error and updated by the four pivotal phases of the dolphin swarm algorithm. Eventually, we will obtain an optimal set of input weights and hidden biases. To evaluate the effectiveness of our method, we compare the proposed algorithm with the standard extreme learning machine and three state-of-the-art methods, which are the particle swarm optimization extreme learning machine, evolutionary extreme learning machine, and self-adaptive evolutionary extreme learning machine, under 13 benchmark datasets obtained from the University of California Irvine Machine Learning Repository. The experimental results demonstrate that the proposed method can achieve superior generalization performances than all the compared algorithms.

Prediction of Air Pollutants Concentration Based on an Extreme Learning Machine: The Case of Hong Kong.

With the development of the economy and society all over the world, most metropolitan cities are experiencing elevated concentrations of ground-level air pollutants. It is urgent to predict and evaluate the concentration of air pollutants for some local environmental or health agencies. Feed-forward artificial neural networks have been widely used in the prediction of air pollutants concentration. However, there are some drawbacks, such as the low convergence rate and the local minimum. The extreme learning machine for single hidden layer feed-forward neural networks tends to provide good generalization performance at an extremely fast learning speed. The major sources of air pollutants in Hong Kong are mobile, stationary, and from trans-boundary sources. We propose predicting the concentration of air pollutants by the use of trained extreme learning machines based on the data obtained from eight air quality parameters in two monitoring stations, including Sham Shui Po and Tap Mun in Hong Kong for six years. The experimental results show that our proposed algorithm performs better on the Hong Kong data both quantitatively and qualitatively. Particularly, our algorithm shows better predictive ability, with increased and root mean square error values decreased respectively.

RFSEN-ELM: SELECTIVE ENSEMBLE OF EXTREME LEARNING MACHINES USING ROTATION FOREST FOR IMAGE CLASSIFICATION

Extreme learning machine (ELM), as a new learning mechanism for single hidden layer feedforward neural networks (SLFNs), has shown its good performance, such as fast computation speed and good generalization performance. However, the weak robustness of ELM is an unavoidable defect for image classification tasks. Thus, we propose a novel ensemble method combined rotation forest and selective ensemble model to overcome this problem in this paper. Firstly, ELM and rotation forest are integrated to construct an ensemble classifier (RF-ELM), which combines the advantages of both rotation forest and ELM. The purpose of rotation forest here is to enhance the diversity of each base classifier, thus improving the robustness of classification. Then several ELMs are removed from the ensemble pool by using genetic algorithm (GA) based selective ensemble model to further enhance the generalization performance. Finally, the remaining ELMs are grouped as a selected ensemble classifier (RFSEN-ELM) for image classification. The performance is analysed and compared with several existing methods on benchmark datasets and the experimental results demonstrate that the proposed algorithm substantially improves the accuracy and robustness of classification at an acceptable level of training cost.

An initial study on the rank of input matrix for extreme learning machine

It is noted that the rank of input data matrix has a critical impact on the performance of a trained classifier model. This paper presents a study on the rank of input data matrix based on a classification model of extreme learning machine which is a single hidden layer feed-forward neural network with non-iterative training. The changing tendency of model accuracy with the increase of input data matrix rank is experimentally investigated and the relationship between the input matrix rank and classification problem complexity is addressed. The analysis and experiments show some meaningful results.

State Preserving Extreme Learning Machine: A Monotonically Increasing Learning Approach

Extreme Learning Machines (ELM) has been introduced as a new algorithm for training single hidden layer feedforward neural networks instead of the classical gradient-based approaches. Based on the consistency property of data, which enforces similar samples to share similar properties, ELM is a biologically inspired learning algorithm that learns much faster with good generalization and performs well in classification tasks. However, the stochastic characteristics of hidden layer outputs from the random generation of the weight matrix in current ELMs leads to the possibility of unstable outputs in the learning and testing phases. This is detrimental to the overall performance when many repeated trials are conducted. To cope with this issue, we present a new ELM approach, named State Preserving Extreme Leaning Machine (SPELM). SPELM ensures the overall training and testing performance of the classical ELM while monotonically increases its accuracy by preserving state variables. For evaluation, experiments are performed on different benchmark datasets including applications in face recognition, pedestrian detection, and network intrusion detection for cyber security. Several popular feature extraction techniques, namely Gabor, pyramid histogram of oriented gradients, and local binary pattern are also incorporated with SPELM. Experimental results show that our SPELM algorithm yields the best performance on tested data over ELM and RELM.

A fast incremental extreme learning machine algorithm for data streams classification

Data streams classification is an important approach to get useful knowledge from massive and dynamic data. Because of concept drift, traditional data mining techniques cannot be directly applied in data streams environment. Extreme learning machine (ELM) is a single hidden layer feedforward neural network (SLFN), comparing with the traditional neural network (e.g. BP network), ELM has a faster speed, so it is very suitable for real-time data processing. In order to deal with the challenge in data streams classification, a new approach based on extreme learning machine is proposed in this paper. The approach utilizes ELMs as base classifiers and adaptively decides the number of the neurons in hidden layer, in addition, activation functions are also randomly selected from a series of functions to improve the performance of the approach. Finally, the algorithm trains a series of classifiers and the decision results for unlabeled data are made by weighted voting strategy. When the concept in data streams keeps stable, every classifier is incrementally updated by using new data; if concept drift is detected, the classifiers with weak performance will be cleared away. In the experiment, we used 7 artificial data sets and 9 real data sets from UCI repository to evaluate the performance of the proposed approach. The testing results showed, comparing with the conventional classification methods for data streams such as ELM, BP, AUE2 and Learn++.MF, on most data sets, the new approach could not only be simplest in the structure, but also get a higher and more stable accuracy with lower time consuming.

Fuzziness based semi-supervised learning approach for intrusion detection system

Countering cyber threats, especially attack detection, is a challenging area of research in the field of information assurance. Intruders use polymorphic mechanisms to masquerade the attack payload and evade the detection techniques. Many supervised and unsupervised learning approaches from the field of machine learning and pattern recognition have been used to increase the efficacy of intrusion detection systems (IDSs). Supervised learning approaches use only labeled samples to train a classifier, but obtaining sufficient labeled samples is cumbersome, and requires the efforts of domain experts. However, unlabeled samples can easily be obtained in many real world problems. Compared to supervised learning approaches, semi-supervised learning (SSL) addresses this issue by considering large amount of unlabeled samples together with the labeled samples to build a better classifier. This paper proposes a novel fuzziness based semi-supervised learning approach by utilizing unlabeled samples assisted with supervised learning algorithm to improve the classifier’s performance for the IDSs. A single hidden layer feed-forward neural network (SLFN) is trained to output a fuzzy membership vector, and the sample categorization (low, mid, and high fuzziness categories) on unlabeled samples is performed using the fuzzy quantity. The classifier is retrained after incorporating each category separately into the original training set. The experimental results using this technique of intrusion detection on the NSL-KDD dataset show that unlabeled samples belonging to low and high fuzziness groups make major contributions to improve the classifier’s performance compared to existing classifiers e.g., naive bayes, support vector machine, random forests, etc.

Forecasting daily streamflow using online sequential extreme learning machines

Summary While nonlinear machine methods have been widely used in environmental forecasting, in situations where new data arrive continually, the need to make frequent model updates can become cumbersome and computationally costly. To alleviate this problem, an online sequential learning algorithm for single hidden layer feedforward neural networks – the online sequential extreme learning machine (OSELM) – is automatically updated inexpensively as new data arrive (and the new data can then be discarded). OSELM was applied to forecast daily streamflow at two small watersheds in British Columbia, Canada, at lead times of 1–3 days. Predictors used were weather forecast data generated by the NOAA Global Ensemble Forecasting System (GEFS), and local hydro-meteorological observations. OSELM forecasts were tested with daily, monthly or yearly model updates. More frequent updating gave smaller forecast errors, including errors for data above the 90th percentile. Larger datasets used in the initial training of OSELM helped to find better parameters (number of hidden nodes) for the model, yielding better predictions. With the online sequential multiple linear regression (OSMLR) as benchmark, we concluded that OSELM is an attractive approach as it easily outperformed OSMLR in forecast accuracy.

A systematic design of interval type-2 fuzzy logic system using extreme learning machine for electricity load demand forecasting

This paper presents a novel design of interval type-2 fuzzy logic systems (IT2FLS) by utilizing the theory of extreme learning machine (ELM) for electricity load demand forecasting. ELM has become a popular learning algorithm for single hidden layer feed-forward neural networks (SLFN). From the functional equivalence between the SLFN and fuzzy inference system, a hybrid of fuzzy-ELM has gained attention of the researchers. This paper extends the concept of fuzzy-ELM to an IT2FLS based on ELM (IT2FELM). In the proposed design the antecedent membership function parameters of the IT2FLS are generated randomly, whereas the consequent part parameters are determined analytically by the Moore–Penrose pseudo inverse. The ELM strategy ensures fast learning of the IT2FLS as well as optimality of the parameters. Effectiveness of the proposed design of IT2FLS is demonstrated with the application of forecasting nonlinear and chaotic data sets. Nonlinear data of electricity load from the Australian National Electricity Market for the Victoria region and from the Ontario Electricity Market are considered here. The proposed model is also applied to forecast Mackey-glass chaotic time series data. Comparative analysis of the proposed model is conducted with some traditional models such as neural networks (NN) and adaptive neuro fuzzy inference system (ANFIS). In order to verify the structure of the proposed design of IT2FLS an alternate design of IT2FLS based on Kalman filter (KF) is also utilized for the comparison purposes.

Sequential Nonlinear Learning for Distributed Multiagent Systems via Extreme Learning Machines.

We study online nonlinear learning over distributed multiagent systems, where each agent employs a single hidden layer feedforward neural network (SLFN) structure to sequentially minimize arbitrary loss functions. In particular, each agent trains its own SLFN using only the data that is revealed to itself. On the other hand, the aim of the multiagent system is to train the SLFN at each agent as well as the optimal centralized batch SLFN that has access to all the data, by exchanging information between neighboring agents. We address this problem by introducing a distributed subgradient-based extreme learning machine algorithm. The proposed algorithm provides guaranteed upper bounds on the performance of the SLFN at each agent and shows that each of these individual SLFNs asymptotically achieves the performance of the optimal centralized batch SLFN. Our performance guarantees explicitly distinguish the effects of data- and network-dependent parameters on the convergence rate of the proposed algorithm. The experimental results illustrate that the proposed algorithm achieves the oracle performance significantly faster than the state-of-the-art methods in the machine learning and signal processing literature. Hence, the proposed method is highly appealing for the applications involving big data.

Liver Tumor Segmentation from MR Images Using 3D Fast Marching Algorithm and Single Hidden Layer Feedforward Neural Network.

Objective. Our objective is to develop a computerized scheme for liver tumor segmentation in MR images. Materials and Methods. Our proposed scheme consists of four main stages. Firstly, the region of interest (ROI) image which contains the liver tumor region in the T1-weighted MR image series was extracted by using seed points. The noise in this ROI image was reduced and the boundaries were enhanced. A 3D fast marching algorithm was applied to generate the initial labeled regions which are considered as teacher regions. A single hidden layer feedforward neural network (SLFN), which was trained by a noniterative algorithm, was employed to classify the unlabeled voxels. Finally, the postprocessing stage was applied to extract and refine the liver tumor boundaries. The liver tumors determined by our scheme were compared with those manually traced by a radiologist, used as the “ground truth.” Results. The study was evaluated on two datasets of 25 tumors from 16 patients. The proposed scheme obtained the mean volumetric overlap error of 27.43% and the mean percentage volume error of 15.73%. The mean of the average surface distance, the root mean square surface distance, and the maximal surface distance were 0.58 mm, 1.20 mm, and 6.29 mm, respectively.

An enhance excavation equipments classification algorithm based on acoustic spectrum dynamic feature

Underground pipeline network surveillance system attracts increasingly attentions recently due to severe breakages caused by external excavation equipments in the mainland of China. In this paper, we study excavation equipments classification algorithm based on acoustic signal processing and machine learning algorithms. A cross-layer microphone array with four elements is designed to collect the acoustic database of representative excavation equipments on real construction sites. The generalized sidelobe canceller algorithm is employed for background noise reduction. The improved spectrum dynamic feature extraction algorithm is then implemented for the benchmark acoustic feature database construction of excavation equipments. To perform classification and background noise identification, the single hidden layer feedforward neural network is employed as the classifier. An improved algorithm based on the popular extreme learning machine (ELM) is proposed for classifier learning. The leave-one-out cross validation strategy is adopted for the regularization parameter optimization in ELM. Comprehensive experiments are conducted to test the effectiveness of the proposed algorithm. Comparisons with state-of-art classifiers and the Mel-frequency cepstrual coefficients acoustic features are also provided to demonstrate the superiority of our approach.

Cholesky factorization based online regularized and kernelized extreme learning machines with forgetting mechanism

In this paper, we propose two alternative schemes of fast online sequential extreme learning machine (ELM) for training the single hidden-layer feedforward neural networks (SLFN), termed as Cholesky factorization based online regularized ELM with forgetting mechanism (CF-FORELM) and Cholesky factorization based online kernelized ELM with forgetting mechanism (CF-FOKELM). First, the solutions of regularized ELM (RELM) and kernelized ELM (KELM) using the matrix Cholesky factorization are introduced; then the recursive method for calculating Cholesky factor of involved matrix in RELM and KELM is designed when RELM and KELM are applied to train SLFN online; consequently, the CF-FORELM and CF-FOKELM are obtained. The numerical simulation results show CF-FORELM demands less computational burden than Dynamic Regression ELM (DR-ELM), and CF-FOKELM also owns higher computational efficiency than both FOKELM and online sequential ELM with kernels (OS-ELMK), and CF-FORELM is less sensitive to model parameters than CF-FOKELM.

Wavelet Network: Online Sequential Extreme Learning Machine for Nonlinear Dynamic Systems Identification

A single hidden layer feedforward neural network (SLFN) with online sequential extreme learning machine (OSELM) algorithm has been introduced and applied in many regression problems successfully. However, using SLFN with OSELM as black-box for nonlinear system identification may lead to building models for the identified plant with inconsistency responses from control perspective. The reason can refer to the random initialization procedure of the SLFN hidden node parameters with OSELM algorithm. In this paper, a single hidden layer feedforward wavelet network (WN) is introduced with OSELM for nonlinear system identification aimed at getting better generalization performances by reducing the effect of a random initialization procedure.

An online sequential learning algorithm for regularized Extreme Learning Machine

The Extreme Learning Machine (ELM) is a novel learning scheme for single hidden layer feedforward neural networks, and it has attracted a great deal of research attention since the last decade because of its extremely fast learning speed. One popular variant of ELM is the Online Sequential ELM (OS-ELM), which can deal with sequential learning tasks. However, limitations exist in the OS-ELM such as requiring the initialization phase, pre-defined important parameters, running into singularity problem, inconsistent and potentially unreliable performance. In this paper, an Online Sequential Regularized ELM (OS-RELM) is proposed to address the aforementioned issues. The main idea is to incorporate the regularization method to further improve its generalization performance, and a new update formula is used to eliminate the initialization phase. To enable the OS-RELM to adapt to new data in an effective and reliable manner, an efficient Leave-One-Out Cross-Validation method is implemented. Finally, a matrix reconstruction method is employed to address the unstable update issue. Unlike some ELM variants that greatly jeopardize the speed advantage of the ELM, we strive to limit the computational load from the proposed scheme. Simulation results on benchmark problems show that the OS-RELM is a reliable and efficient algorithm with superior generalization performance than the OS-ELM.

Incremental regularized extreme learning machine and it׳s enhancement

Extreme Learning Machine (ELM) proposed by Huang et al. [2] is a novel algorithm for single hidden layer feedforward neural networks (SLFNs) with extremely fast learning speed and good generalization performance. When new hidden nodes are added to the existing network, retraining the network would be time consuming, and EM-ELM [13] was proposed to calculate the output weights incrementally. However there are still two issues in EM-ELM: first, the initial hidden layer output matrix may be rank deficient thus the computation will loss accuracy; second, EM-ELM cannot always get good generalization performance due to overfitting. So we propose the improved version of EM-ELM based on regularization method called Incremental Regularized Extreme Learning Machine (IR-ELM). When new hidden node is added one by one, IR-ELM can update output weights recursively in a fast way. Enhancement of IR-ELM (EIR-ELM) that has a selection of hidden nodes to be added to the network is also introduced in this paper. Empirical studies on benchmark data sets for regression and classification problems have shown that IR-ELM (EIR-ELM) always gets better generalization performance than EM-ELM with the similar training time.

Online prediction of glucose concentration in type 1 diabetes using extreme learning machines.

We propose an online machine-learning solution to the problem of nonlinear glucose time series prediction in type 1 diabetes. Recently, extreme learning machine (ELM) has been proposed for training single hidden layer feed-forward neural networks. The high accuracy and fast learning speed of ELM drive us to investigate its applicability to the glucose prediction problem. Given that diabetes self-monitoring data are received sequentially, we focus on online sequential ELM (OS-ELM) and online sequential ELM kernels (KOS-ELM). A multivariate feature set is utilized concerning subcutaneous glucose, insulin therapy, carbohydrates intake and physical activity. The dataset comes from the continuous multi-day recordings of 15 type 1 patients in free-living conditions. Assuming stationarity and evaluating the performance of the proposed method by 10-fold cross- validation, KOS-ELM were found to perform better than OS-ELM in terms of prediction error, temporal gain and regularity of predictions for a 30-min prediction horizon.

On a problem of Hornik

In 1991, Hornik proved that the collection of single hidden layer feedforward neural networks (SLFNs) with continuous, bounded, and non-constant activation function σ is dense in C(K) where K is a compact set in ℝs (see Neural Networks, 4(2), 251–257 (1991)). Meanwhile, he pointed out “Whether or not the continuity assumption can entirely be dropped is still an open quite challenging problem”. This paper replies in the affirmative to the problem and proves that for bounded and continuous almost everywhere (a.e.) activation function σ on ℝ, the collection of SLFNs is dense in C(K) if and only if σ is un-constant a.e..

Extreme Learning Machine for Multilayer Perceptron.

Extreme learning machine (ELM) is an emerging learning algorithm for the generalized single hidden layer feedforward neural networks, of which the hidden node parameters are randomly generated and the output weights are analytically computed. However, due to its shallow architecture, feature learning using ELM may not be effective for natural signals (e.g., images/videos), even with a large number of hidden nodes. To address this issue, in this paper, a new ELM-based hierarchical learning framework is proposed for multilayer perceptron. The proposed architecture is divided into two main components: 1) self-taught feature extraction followed by supervised feature classification and 2) they are bridged by random initialized hidden weights. The novelties of this paper are as follows: 1) unsupervised multilayer encoding is conducted for feature extraction, and an ELM-based sparse autoencoder is developed via l1 constraint. By doing so, it achieves more compact and meaningful feature representations than the original ELM; 2) by exploiting the advantages of ELM random feature mapping, the hierarchically encoded outputs are randomly projected before final decision making, which leads to a better generalization with faster learning speed; and 3) unlike the greedy layerwise training of deep learning (DL), the hidden layers of the proposed framework are trained in a forward manner. Once the previous layer is established, the weights of the current layer are fixed without fine-tuning. Therefore, it has much better learning efficiency than the DL. Extensive experiments on various widely used classification data sets show that the proposed algorithm achieves better and faster convergence than the existing state-of-the-art hierarchical learning methods. Furthermore, multiple applications in computer vision further confirm the generality and capability of the proposed learning scheme.

Distributed Extreme Learning Machine for Nonlinear Learning over Network

Distributed data collection and analysis over a network are ubiquitous, especially over a wireless sensor network (WSN). To our knowledge, the data model used in most of the distributed algorithms is linear. However, in real applications, the linearity of systems is not always guaranteed. In nonlinear cases, the single hidden layer feedforward neural network (SLFN) with radial basis function (RBF) hidden neurons has the ability to approximate any continuous functions and, thus, may be used as the nonlinear learning system. However, confined by the communication cost, using the distributed version of the conventional algorithms to train the neural network directly is usually prohibited. Fortunately, based on the theorems provided in the extreme learning machine (ELM) literature, we only need to compute the output weights of the SLFN. Computing the output weights itself is a linear learning problem, although the input-output mapping of the overall SLFN is still nonlinear. Using the distributed algorithmto cooperatively compute the output weights of the SLFN, we obtain a distributed extreme learning machine (dELM) for nonlinear learning in this paper. This dELM is applied to the regression problem and classification problem to demonstrate its effectiveness and advantages.