Hidden Layer Biases Research Articles

Predicting rockburst with database using particle swarm optimization and extreme learning machine

Rockburst is a major type of geological hazard that has a very adverse impact on underground engineering in deeply buried areas under high geo-stress. In this study, extreme learning machine (ELM) was used to predict and classify rockburst intensity, and particle swarm optimization (PSO) was used to optimize the input weight matrix and the hidden layer bias in ELM. Six quantitative rockburst parameters were used as input for the PSO-ELM network, including the maximum tangential stress of the surrounding rock σθ, the uniaxial compressive strength of rock σc, the tensile strength of rock σt, the stress ratio σθ/σc, the rock brittleness ratio σc/σt and the elastic energy index Wet. The network was used to learn from a database of 344 collected worldwide rockburst cases, on which the PSO-ELM rockburst prediction model was established and verified using 8-fold cross-validation and independent test set validation. The model was then tested on a new set of fifteen typical rockburst cases from Jiangbian hydropower station in China. The results showed that the PSO-ELM model performed well in rockburst level prediction. In addition, the model showed superior performance compared with previously proposed machine learning models for rockburst prediction and the rockburst empirical criteria, which underscores its utility in future rockburst prediction.

An Extreme Learning Machine Model Approach on Airbnb Base Price Prediction

The base price of Airbnb properties prediction is still a new area of prediction research, especially with the Extreme Learning Machine (ELM). The previous studies had several suggestions for the advantages of ELM, such as good generalization performance, fast learning speed, and high prediction accuracy. This paper proposes how the ELM approach is used as a prediction model for Air BnB base price. Generally, the steps are setting hidden neuron numbers, randomly assigning input weight and hidden layer biases, calculating the output layer; and the entire learning measure finished through one numerical change without iteration. The performance of the model is estimated utilizing mean squared error, mean absolute percentage error, and root mean squared error. Experiment with Airbnb dataset in London with twenty-one features as input generates a faster learning speed and better accuracy than the existing model.

Sparse Bayesian approach to fast learning network for multiclassification

This paper proposes a novel artificial neural network called sparse-Bayesian--based fast learning network (SBFLN). In SBFLN, sparse Bayesian regression is used to train the fast learning network (FLN), which is an improved extreme learning machine (ELM). The training process of SBFLN is to randomly generate the input weights and the hidden layer biases, and then find the probability distribution of other weights by the sparse Bayesian approach. SBFLN calculates the predicted output through Bayes estimator, so it can provide a natural marginal possibility for classification problems and can solve the overfitting problem caused by the least-squares estimation in FLN. In addition, the sparse Bayesian approach can automatically trim most redundant neurons in hidden layer, which makes the network more compact and accurate. To verify the effectiveness of the improvements in this paper, the results of SBFLN are evaluated in 15 benchmark classification problems. The experimental results show that SBFLN is not sensitive to the number of neurons in the hidden layer, and the performance of SBFLN is competitive or superior to some other state-of-the-art algorithms.

Learning from correlation with extreme learning machine

A seemingly unrelated regression (SUR) refers to several individual equations among which there is not an explicit connection such as one equation’s observation is another equation’s response, but there exists an implicit relation represented by correlated disturbances of response variables. In this paper, SUR is applied to extreme learning machine (ELM) which is a single hidden layer feed-forward neural network where input weights and hidden layer biases are randomly assigned but the weight parameters between hidden and output layers are least-square solutions of a regression equation. A correlation-based extreme learning machine is built using the auxiliary sample which is related to the main sample which we focus on. Considering the weights between hidden and output layers in ELM as a random vector, we derive an explicit representation for the vector’s covariance matrix. The proof of theorems and simulation process indicate that the stronger correlation between main sample and auxiliary sample is, the higher generalization ability is.

Flood Forecasting Based on an Improved Extreme Learning Machine Model Combined with the Backtracking Search Optimization Algorithm

Flood forecasting plays an important role in flood control and water resources management. Recently, the data-driven models with a simpler model structure and lower data requirement attract much more attentions. An extreme learning machine (ELM) method, as a typical data-driven method, with the advantages of a faster learning process and stronger generalization ability, has been taken as an effective tool for flood forecasting. However, an ELM model may suffer from local minima in some cases because of its random generation of input weights and hidden layer biases, which results in uncertainties in the flood forecasting model. Therefore, we proposed an improved ELM model for short-term flood forecasting, in which an emerging dual population-based algorithm, named backtracking search algorithm (BSA), was applied to optimize the parameters of ELM. Thus, the proposed method is called ELM-BSA. The upper Yangtze River was selected as a case study. Several performance indexes were used to evaluate the efficiency of the proposed ELM-BSA model. Then the proposed model was compared with the currently used general regression neural network (GRNN) and ELM models. Results show that the ELM-BSA can always provide better results than the GRNN and ELM models in both the training and testing periods. All these results suggest that the proposed ELM-BSA model is a promising alternative technique for flood forecasting.

The universal consistency of extreme learning machine

Abstract Extreme learning machine (ELM) can be considered as a single-hidden layer feedforward neural network (FNN)-type learning system, whose input weights and hidden layer biases are randomly assigned, while output weights need tuning. In the framework of regression, a fundamental problem of ELM learning is whether the ELM estimator is universally consistent, that is, whether it can approximate arbitrary regression function to any accuracy, provided the number of training samples is sufficiently large. The aim of this paper is two-fold. One is to verify the strongly universal consistency of the ELM estimator, and the other is to present a sufficient and the necessary condition for the activation function, where the corresponding ELM estimator is strongly universally consistent. The obtained results underlie the feasibility of ELM and provide a theoretical guidance of the selection of activation functions in ELM learning.

Fast Learning Network with Parallel Layer Perceptrons

This paper proposes a novel artificial neural network called Parallel Layer Perceptron Fast Learning Network (PLP-FLN). In PLP-FLN, a parallel single hidden layer feed-forward neural network is added on the basis of Fast Learning Network (FLN) which is an improved Extreme Learning Machine (ELM). Input weights and hidden layer biases are randomly generated. The weights connect the output nodes and the input nodes, and the weights connect the output nodes and the hidden nodes are analytically determined based on least squares methods. In order to test the PLP-FLN validity, this paper compared it with ELM, FLN, Kernel ELM and Incremental ELM through 12 regression applications and 7 classification problems. By comparing the experimental results, it shows that the PLP-FLN with much more compact networks have demonstrated better approximations, classification performances and generalization ability.

Subagging for the improvement of predictive stability of extreme learning machine for spectral quantitative analysis of complex samples

Extreme learning machine (ELM) has been attracted increasing attentions for its fast learning speed and excellent generalization performance. However, the prediction result of a single ELM regression model is usually unstable due to the randomly generating of the input weights and hidden layer bias. To overcome this drawback, an ensemble form of ELM, termed as subagging ELM, was proposed and used for spectral quantitative analysis of complex samples. In the approach, a series of ELM sub-models was built by randomly selecting a certain number of samples from the original training set without replacement, and then the predictions of these sub-models were combined by a simple averaging way to give the final ensemble prediction. The performance of the method was tested with fuel oil and blood samples. Compared to a single ELM model, the results confirm that subagging ELM can achieve much better stability and higher accuracy than ELM.

Stream-flow forecasting using extreme learning machines: A case study in a semi-arid region in Iraq

Monthly stream-flow forecasting can yield important information for hydrological applications including sustainable design of rural and urban water management systems, optimization of water resource allocations, water use, pricing and water quality assessment, and agriculture and irrigation operations. The motivation for exploring and developing expert predictive models is an ongoing endeavor for hydrological applications. In this study, the potential of a relatively new data-driven method, namely the extreme learning machine (ELM) method, was explored for forecasting monthly stream-flow discharge rates in the Tigris River, Iraq. The ELM algorithm is a single-layer feedforward neural network (SLFNs) which randomly selects the input weights, hidden layer biases and analytically determines the output weights of the SLFNs. Based on the partial autocorrelation functions of historical stream-flow data, a set of five input combinations with lagged stream-flow values are employed to establish the best forecasting model. A comparative investigation is conducted to evaluate the performance of the ELM compared to other data-driven models: support vector regression (SVR) and generalized regression neural network (GRNN). The forecasting metrics defined as the correlation coefficient (r), Nash-Sutcliffe efficiency (ENS), Willmott’s Index (WI), root-mean-square error (RMSE) and mean absolute error (MAE) computed between the observed and forecasted stream-flow data are employed to assess the ELM model’s effectiveness. The results revealed that the ELM model outperformed the SVR and the GRNN models across a number of statistical measures. In quantitative terms, superiority of ELM over SVR and GRNN models was exhibited by ENS=0.578, 0.378 and 0.144, r=0.799, 0.761 and 0.468 and WI=0.853, 0.802 and 0.689, respectively and the ELM model attained lower RMSE value by approximately 21.3% (relative to SVR) and by approximately 44.7% (relative to GRNN). Based on the findings of this study, several recommendations were suggested for further exploration of the ELM model in hydrological forecasting problems.

Random vector functional link network for short-term electricity load demand forecasting

Short-term electricity load forecasting plays an important role in the energy market as accurate forecasting is beneficial for power dispatching, unit commitment, fuel allocation and so on. This paper reviews a few single hidden layer network configurations with random weights (RWSLFN). The RWSLFN was extended to eight variants based on the presence or absence of input layer bias, hidden layer bias and direct input–output connections. In order to avoid mapping the weighted inputs into the saturation region of the enhancement nodes’ activation function and to suppress the outliers in the input data, a quantile scaling algorithm to re-distribute the randomly weighted inputs is proposed. The eight variations of RWSLFN are assessed using six generic time series datasets and 12 load demand time series datasets. The result shows that the RWSLFNs with direct input–output connections (known as the random vector functional link network or RVFL network) have statistically significantly better performance than the RWSLFN configurations without direct input–output connections, possibly due to the fact that the direct input–output connections in the RVFL network emulate the time delayed finite impulse response (FIR) filter. However the RVFL network has simpler training and higher accuracy than the FIR based two stage neural network. The RVFL network is also compared with some reported forecasting methods. The RVFL network overall outperforms the non-ensemble methods, namely the persistence method, seasonal autoregressive integrated moving average (sARIMA), artificial neural network (ANN). In addition, the testing time of the RVFL network is the shortest while the training time is comparable to the other reported methods. Finally, possible future research directions are pointed out.

Discriminative manifold extreme learning machine and applications to image and EEG signal classification

Extreme learning machine (ELM) uses a non-iterative method to train single-hidden-layer feed-forward networks (SLFNs), which has been proven to be an efficient and effective learning model for both classification and regression. The main advantage of ELM lies in that the input weights as well as the hidden layer biases can be randomly generated, which contributes to the analytical solution of output weights. In this paper, we propose a discriminative manifold ELM (DMELM) by simultaneously considering the discriminative information and geometric structure of data; specifically, we exploit the discriminative information in the local neighborhood around each data point. To this end, a graph regularizer based on a newly designed graph Laplacian to characterize both properties is formulated and incorporated into the ELM objective. In DMELM, the output weights can also be obtained in analytical form. Extensive experiments are conducted on image and EEG signal classification to evaluate the effectiveness of DMELM. The results show that DMELM consistently achieves better performance than original ELM and yields promising results in comparison with several state-of-the-art algorithms, which suggests that both the discriminative as well as manifold information are beneficial to classification.

Discriminative graph regularized extreme learning machine and its application to face recognition

Extreme Learning Machine (ELM) has been proposed as a new algorithm for training single hidden layer feed forward neural networks. The main merit of ELM lies in the fact that the input weights as well as hidden layer bias are randomly generated and thus the output weights can be obtained analytically, which can overcome the drawbacks incurred by gradient-based training algorithms such as local optima, improper learning rate and low learning speed. Based on the consistency property of data, which enforces similar samples to share similar properties, we propose a discriminative graph regularized Extreme Learning Machine (GELM) for further enhancing its classification performance in this paper. In the proposed GELM model, the label information of training samples are used to construct an adjacent graph and correspondingly the graph regularization term is formulated to constrain the output weights to learn similar outputs for samples from the same class. The proposed GELM model also has a closed form solution as the standard ELM and thus the output weights can be obtained efficiently. Experiments on several widely used face databases show that our proposed GELM can achieve much performance gain over standard ELM and regularized ELM. Moreover, GELM also performs well when compared with the state-of-the-art classification methods for face recognition.

Fast learning network: a novel artificial neural network with a fast learning speed

This paper proposes a novel artificial neural network called fast learning network (FLN). In FLN, input weights and hidden layer biases are randomly generated, and the weight values of the connection between the output layer and the input layer and the weight values connecting the output node and the input nodes are analytically determined based on least squares methods. In order to test the FLN validity, it is applied to nine regression applications, and experimental results show that, compared with support vector machine, back propagation, extreme learning machine, the FLN with much more compact networks can achieve very good generalization performance and stability at a very fast training speed and a quick reaction of the trained network to new observations. In addition, in order to further test the FLN validity, it is applied to model the thermal efficiency and NO x emissions of a 330 WM coal-fired boiler and achieves very good prediction precision and generalization ability at a high learning speed.

Dynamic ensemble extreme learning machine based on sample entropy

Extreme learning machine (ELM) as a new learning algorithm has been proposed for single-hidden layer feed-forward neural networks, ELM can overcome many drawbacks in the traditional gradient-based learning algorithm such as local minimal, improper learning rate, and low learning speed by randomly selecting input weights and hidden layer bias. However, ELM suffers from instability and over-fitting, especially on large datasets. In this paper, a dynamic ensemble extreme learning machine based on sample entropy is proposed, which can alleviate to some extent the problems of instability and over-fitting, and increase the prediction accuracy. The experimental results show that the proposed approach is robust and efficient.

Regularized online sequential learning algorithm for single-hidden layer feedforward neural networks

Online learning algorithms have been preferred in many applications due to their ability to learn by the sequentially arriving data. One of the effective algorithms recently proposed for training single hidden-layer feedforward neural networks (SLFNs) is online sequential extreme learning machine (OS-ELM), which can learn data one-by-one or chunk-by-chunk at fixed or varying sizes. It is based on the ideas of extreme learning machine (ELM), in which the input weights and hidden layer biases are randomly chosen and then the output weights are determined by the pseudo-inverse operation. The learning speed of this algorithm is extremely high. However, it is not good to yield generalization models for noisy data and is difficult to initialize parameters in order to avoid singular and ill-posed problems. In this paper, we propose an improvement of OS-ELM based on the bi-objective optimization approach. It tries to minimize the empirical error and obtain small norm of network weight vector. Singular and ill-posed problems can be overcome by using the Tikhonov regularization. This approach is also able to learn data one-by-one or chunk-by-chunk. Experimental results show the better generalization performance of the proposed approach on benchmark datasets.

AN IMPROVEMENT OF EXTREME LEARNING MACHINE FOR COMPACT SINGLE-HIDDEN-LAYER FEEDFORWARD NEURAL NETWORKS

Recently, a novel learning algorithm called extreme learning machine (ELM) was proposed for efficiently training single-hidden-layer feedforward neural networks (SLFNs). It was much faster than the traditional gradient-descent-based learning algorithms due to the analytical determination of output weights with the random choice of input weights and hidden layer biases. However, this algorithm often requires a large number of hidden units and thus slowly responds to new observations. Evolutionary extreme learning machine (E-ELM) was proposed to overcome this problem; it used the differential evolution algorithm to select the input weights and hidden layer biases. However, this algorithm required much time for searching optimal parameters with iterative processes and was not suitable for data sets with a large number of input features. In this paper, a new approach for training SLFNs is proposed, in which the input weights and biases of hidden units are determined based on a fast regularized least-squares scheme. Experimental results for many real applications with both small and large number of input features show that our proposed approach can achieve good generalization performance with much more compact networks and extremely high speed for both learning and testing.

Small Number of Hidden Units for ELM with Two-Stage Linear Model

The single-hidden-layer feedforward neural networks (SLFNs) are frequently used in machine learning due to their ability which can form boundaries with arbitrary shapes if the activation function of hidden units is chosen properly. Most learning algorithms for the neural networks based on gradient descent are still slow because of the many learning steps. Recently, a learning algorithm called extreme learning machine (ELM) has been proposed for training SLFNs to overcome this problem. It randomly chooses the input weights and hidden-layer biases, and analytically determines the output weights by the matrix inverse operation. This algorithm can achieve good generalization performance with high learning speed in many applications. However, this algorithm often requires a large number of hidden units and takes long time for classification of new observations. In this paper, a new approach for training SLFNs called least-squares extreme learning machine (LS-ELM) is proposed. Unlike the gradient descent-based algorithms and the ELM, our approach analytically determines the input weights, hidden-layer biases and output weights based on linear models. For training with a large number of input patterns, an online training scheme with sub-blocks of the training set is also introduced. Experimental results for real applications show that our proposed algorithm offers high classification accuracy with a smaller number of hidden units and extremely high speed in both learning and testing.

Modelling the Effect of Carbon Content on Hot Strength of Steels Using a Modified Artificial Neural Network.

The hot strength of austenitic steels with the carbon content varying from 0.0037 to 0.79 wt% was modelled using artificial neural networks (ANN). The carbon content has a complex effect on flow strength of austenite. An increase in carbon content reduces the flow stress of the steels at high temperatures and low strain rates, while it increases the flow stress at low temperatures and high strain rates, especially at low strains. In addition, increasing carbon to above 0.4 wt% dramatically reduces the peak strain for the initiation of dynamic recrystallisation at high Zener–Hollomon parameter, Z. Given the complexity of the deformation and recrystallisation behaviours of these steels, no phenomenological or simple empirical models are able to predict the flow stress over the full carbon range. In this work, the back error propagation algorithm of the ANN model with one hidden layer bias was used, with the number if hidden nodes optimised. The data up to a strain of 4 were used to predict the strength in both work hardening and dynamic recrystallisation regimes. The training speed was an important parameter and was optimised by trimming the data set and learning procedures. The effects of the carbon content on flow stress, peak strains and peak stresses observed from the experiment were accurately represented. However, it was found that the training data set also needed to be optimised to accurately predict the hot strength of the steels.

Some new results on neural network approximation

We show that standard feedforward networks with as few as a single hidden layer can uniformly approximate continuous functions on compacta provided that the activation function ψ is locally Riemann integrable and nonpolynomial, and have universal Lp (μ) approximation capabilities for finite and compactly supported input environment measures μ provided that ψ is locally bounded and nonpolynomial. In both cases, the input-to-hidden weights and hidden layer biases can be constrained to arbitrarily small sets; if in addition ψ is locally analytic a single universal bias will do.