Layer Feedforward Neural Network Research Articles

Pruning Multilayered ELM Using Cholesky Factorization and Givens Rotation Transformation

Extreme learning machine is originally proposed for the learning of the single hidden layer feedforward neural network to overcome the challenges faced by the backpropagation (BP) learning algorithm and its variants. Recent studies show that ELM can be extended to the multilayered feedforward neural network in which the hidden node could be a subnetwork of nodes or a combination of other hidden nodes. Although the ELM algorithm with multiple hidden layers shows stronger nonlinear expression ability and stability in both theoretical and experimental results than the ELM algorithm with the single hidden layer, with the deepening of the network structure, the problem of parameter optimization is also highlighted, which usually requires more time for model selection and increases the computational complexity. This paper uses Cholesky factorization strategy and Givens rotation transformation to choose the hidden nodes of MELM and obtains the number of nodes more suitable for the network. First, the initial network has a large number of hidden nodes and then uses the idea of ridge regression to prune the nodes. Finally, a complete neural network can be obtained. Therefore, the ELM algorithm eliminates the need to manually set nodes and achieves complete automation. By using information from the previous generation’s connection weight matrix, it can be evitable to re-calculate the weight matrix in the network simplification process. As in the matrix factorization methods, the Cholesky factorization factor is calculated by Givens rotation transform to achieve the fast decreasing update of the current connection weight matrix, thus ensuring the numerical stability and high efficiency of the pruning process. Empirical studies on several commonly used classification benchmark problems and the real datasets collected from coal industry show that compared with the traditional ELM algorithm, the pruning multilayered ELM algorithm proposed in this paper can find the optimal number of hidden nodes automatically and has better generalization performance.

A Multi-Layer Classification Approach for Intrusion Detection in IoT Networks Based on Deep Learning.

The security of IoT networks is an important concern to researchers and business owners, which is taken into careful consideration due to its direct impact on the availability of the services offered by IoT devices and the privacy of the users connected with the network. An intrusion detection system ensures the security of the network and detects malicious activities attacking the network. In this study, a deep multi-layer classification approach for intrusion detection is proposed combining two stages of detection of the existence of an intrusion and the type of intrusion, along with an oversampling technique to ensure better quality of the classification results. Extensive experiments are made for different settings of the first stage and the second stage in addition to two different strategies for the oversampling technique. The experiments show that the best settings of the proposed approach include oversampling by the intrusion type identification label (ITI), 150 neurons for the Single-hidden Layer Feed-forward Neural Network (SLFN), and 2 layers and 150 neurons for LSTM. The results are compared to well-known classification techniques, which shows that the proposed technique outperforms the others in terms of the G-mean having the value of 78% compared to 75% for KNN and less than 50% for the other techniques.

A Multi-Stage Classification Approach for IoT Intrusion Detection Based on Clustering with Oversampling

Intrusion detection of IoT-based data is a hot topic and has received a lot of interests from researchers and practitioners since the security of IoT networks is crucial. Both supervised and unsupervised learning methods are used for intrusion detection of IoT networks. This paper proposes an approach of three stages considering a clustering with reduction stage, an oversampling stage, and a classification by a Single Hidden Layer Feed-Forward Neural Network (SLFN) stage. The novelty of the paper resides in the technique of data reduction and data oversampling for generating useful and balanced training data and the hybrid consideration of the unsupervised and supervised methods for detecting the intrusion activities. The experiments were evaluated in terms of accuracy, precision, recall, and G-mean and divided into four steps: measuring the effect of the data reduction with clustering, the evaluation of the framework with basic classifiers, the effect of the oversampling technique, and a comparison with basic classifiers. The results show that SLFN classification technique and the choice of Support Vector Machine and Synthetic Minority Oversampling Technique (SVM-SMOTE) with a ratio of 0.9 and the k value of 3 for k-means++ clustering technique give better results than other values and other classification techniques.

Neural Networks with Improved Extreme Learning Machine for Demand Prediction of Bike-sharing

Accurate demand prediction of bike-sharing is an important prerequisite to reducing the cost of scheduling and improving the user satisfaction. However, it is a challenging issue due to stochasticity and non-linearity in bike-sharing systems. In this paper, a model called pseudo-double hidden layer feedforward neural networks is proposed to approximately predict actual demands of bike-sharing. Specifically, to overcome limitations in traditional back-propagation learning process, an algorithm, an extreme learning machine with improved particle swarm optimization, is designed to construct learning rules in neural networks. The performance is verified by comparing with other learning algorithms on the dataset of Streeter Dr bike-sharing station in Chicago.

Design of Fuzzy Ensemble Architecture Realized With the Aid of FCM-Based Fuzzy Partition and NN With Weighted LSE Estimation

Neural networks (NNs) with least square error (LSE) estimation form a certain type of single hidden layer feed-forward NNs. In this class of networks, the input connections (weights) and the biases of hidden neurons are generated randomly and fixed after being generated. The output connections are estimated by the LSE method rather than the back-propagation method. The random generation of the input connection weights and the hidden biases results in the larger number of hidden neurons to assure the quality of classification performance. To reduce the number of neurons in the hidden layer while maintaining the classification performance, we apply a “divide and conquer” strategy in this article. In other words, we divide an overall input space into several subspaces by using information granulation technique (Fuzzy C-Means clustering algorithm) and determine the local decision boundaries among related subspaces. A decision boundary defined in the input space can be considered as being composed of several decision boundaries defined in subspaces that form the entire input space. For the decision boundaries defined in the subspaces, their nonlinearity becomes lower in comparison with the one being encountered when considering the entire input space. Through the weighted LSE estimation instead of using the LSE estimation method, the connections of several NNs can be estimated without interfering with each other. After estimating the weights, the decision boundaries defined in the related subspaces are merged to a single decision boundary by using fuzzy ensemble technique. Several machine learning datasets and one real world application dataset are used to evaluate and validate the proposed fuzzy ensemble classifier. Based on the experimental results, the proposed classifier shows better classification performance when compared with the performance of some selected classifiers.

Online learning of neural networks using random projections and sliding window: A case study of a real industrial process

Online Learning of non-stationary data streams is a challenging task. This work presents an online training method for a Single hidden Layer Feedforward neural Network (SLFN) that learns sample-by-sample, using an adjustable sliding window to adapt the network when data has changed. The method presents a fast training procedure, estimating hidden and output layer parameters independently. Tests with four synthetic datasets showed a good accuracy and quick recovery after drift occurrences. The proposed method is also applied to a real dataset from an industrial process in order to address the anomaly detection task, with the network acting as a classifier. Results show that the method is able to detect drifts prior to anomalies in the pre-fault periods, in the real situation that appeared in the industrial dataset.

Adaptive Framework for Deep Learning Based Dynamic and Temporal Topic Modeling from Big Data

Background: The large amount of data emanated from social media platforms need scalable topic modeling in order to get current trends and themes of events discussed on such platforms. Topic modeling play crucial role in many natural language processing applications like sentiment analysis, recommendation systems, event tracking, summarization, etc. Objective: The aim of the proposed work is to adaptively extract the dynamically evolving topics over streaming data, and infer the current trends and get the notion of trend of topics over time. Because of various world level events, many uncorrelated streaming channels tend to start discussion on similar topics. We aim to find the effect of uncorrelated streaming channels on topic modeling when they tend to start discussion on similar topics. Methods: An adaptive framework for dynamic and temporal topic modeling using deep learning has been put forth in this paper. The framework approximates online latent semantic indexing constrained by regularization on streaming data using adaptive learning method. The framework is designed using deep layers of feedforward neural network. Results: This framework supports dynamic and temporal topic modeling. The proposed approach is scalable to large collection of data. We have performed exploratory data analysis and correspondence analysis on real world Twitter dataset. Results state that our approach works well to extract topic topics associated with a given hashtag. Given the query, the approach is able to extract both implicit and explicit topics associated with the terms mentioned in the query. Conclusion: The proposed approach is a suitable solution for performing topic modeling over Big Data. We are approximating the Latent Semantic Indexing model with regularization using deep learning with differentiable ℓ1 regularization, which makes the model work on streaming data adaptively at real-time. The model also supports the extraction of aspects from sentences based on interrelation of topics and thus, supports aspect modeling in aspect-based sentiment analysis.

Extreme Learning Machine for Heartbeat Classification with Hybrid Time-Domain and Wavelet Time-Frequency Features.

Automatic heartbeat classification via electrocardiogram (ECG) can help diagnose and prevent cardiovascular diseases in time. Many classification approaches have been proposed for heartbeat classification, based on feature extraction. However, the existing approaches face the challenges of high feature dimensions and slow recognition speeds. In this paper, we propose an efficient extreme learning machine (ELM) approach for heartbeat classification with multiple classes, based on the hybrid time-domain and wavelet time-frequency features. The proposed approach contains two sequential modules: (1) feature extraction of heartbeat signals, including RR interval features in the time-domain and wavelet time-frequency features, and (2) heartbeat classification using ELM based on the extracted features. RR interval features are calculated to reflect the dynamic characteristics of heartbeat signals. Discrete wavelet transform (DWT) is used to decompose the heartbeat signals and extract the time-frequency features of the heartbeat signals along the timeline. ELM is a single-hidden layer feedforward neural network with the hidden layer parameters randomly generated in advance and the output layer parameters calculated optimally using the least-square algorithm directly using the training samples. ELM is used as the heartbeat classification algorithm due to its high accuracy training accuracy, fast training speed, and good generalization ability. Experimental testing is carried out using the public MIT-BIH arrhythmia dataset to perform a 16-class classification. Experimental results show that the proposed approach achieves a superior classification accuracy with fast training and recognition speeds, compared with existing classification algorithms.

Wind Turbine Prognosis Models Based on SCADA Data and Extreme Learning Machines

In this paper, a method to build models to monitor and evaluate the health status of wind turbines using Single-hidden Layer Feedforward Neural networks (SLFN) is presented. The models are trained using the Extreme Learning Machines (ELM) strategy. The data used is obtained from the SCADA systems, easily available in modern wind turbines. The ELM technique requires very low computational costs for the training of the models, and thus allows for the integration of a grid-search approach with parallelized instances to find out the optimal model parameters. These models can be built both individually, considering the turbines separately, or as an aggregate for the whole wind plant. The followed strategy consists in predicting a target variable using the rest of the variables of the system/subsystem, computing the error deviation from the real target variable and finally comparing high error values with a selection of alarm events for that system, therefore validating the performance of the model. The experimental results indicate that this methodology leads to the detection of mismatches in the stages of the system’s failure, thus making it possible to schedule the maintenance operation before a critical failure occurs. The simplicity of the ELM systems and the ease with which the parameters can be adjusted make it a realistic option to be implemented in wind turbine models to work in real time.

Processing chain for 3D histogram of gradients based real-time object recognition

3D object recognition has been a cutting-edge research topic since the popularization of depth cameras. These cameras enhance the perception of the environment and so are particularly suitable for autonomous robot navigation applications. Advanced deep learning approaches for 3D object recognition are based on complex algorithms and demand powerful hardware resources. However, autonomous robots and powered wheelchairs have limited resources, which affects the implementation of these algorithms for real-time performance. We propose to use instead a 3D voxel-based extension of the 2D histogram of oriented gradients (3DVHOG) as a handcrafted object descriptor for 3D object recognition in combination with a pose normalization method for rotational invariance and a supervised object classifier. The experimental goal is to reduce the overall complexity and the system hardware requirements, and thus enable a feasible real-time hardware implementation. This article compares the 3DVHOG object recognition rates with those of other 3D recognition approaches, using the ModelNet10 object data set as a reference. We analyze the recognition accuracy for 3DVHOG using a variety of voxel grid selections, different numbers of neurons ( Nh) in the single hidden layer feedforward neural network, and feature dimensionality reduction using principal component analysis. The experimental results show that the 3DVHOG descriptor achieves a recognition accuracy of 84.91% with a total processing time of 21.4 ms. Despite the lower recognition accuracy, this is close to the current state-of-the-art approaches for deep learning while enabling real-time performance.

Online Sequential Complex-Valued ELM for Noncircular Signals: Augmented Structures and Learning Algorithms

Online sequential extreme learning machine (OS-ELM) has become a popular online learning strategy for single-hidden layer feedforward neural networks, and complex-valued signals are ubiquitous in real applications. In order to cater for complex-valued signals, especially for noncircular signals, in this paper we extend OS-ELM to complex domain, and propose two augmented online sequential complex ELM models by incorporating the conjugates of the network input and the hidden layer respectively. In this way, the proposed models are equipped with the capability to capture the complete second-order statistics of noncircular signals, which results in the enhanced generalization ability. The corresponding regularized models are derived to avoid the possible overfitting problem. By exploiting the algebra structure resulted from the augmented architecture, several approaches to reducing the computational complexity are also proposed. Simulation results validate the efficiency of the proposed algorithms.

Potential Fault Diagnosis Method and Classification Accuracy Detection of IGBT Device Based on Improved Single Hidden Layer Feedforward Neural Network.

Insulated Gate Bipolar Transistor (IGBT) is a high-power switch in the field of power electronics. Its reliability is closely related to system stability. Once failure occurs, it may cause irreparable loss. Therefore, potential fault diagnosis methods of IGBT devices are studied in this paper, and their classification accuracy is tested. Due to the shortcomings of incomplete data application in the traditional method of characterizing the device state based on point frequency parameters, a fault diagnosis method based on full frequency threshold screening was proposed in this paper, and its classification accuracy was detected by the Extreme Learning Machine (ELM) algorithm. The randomly generated input layer weight ω and hidden layer deviation lead to the change of output layer weight β and then affect the overall output result. In view of the errors and instability caused by this randomness, an improved Finite Impulse Response Filter ELM (FIR-ELM) training algorithm is proposed. The filtering technique is used to initialize the input weights of the Single Hidden Layer Feedforward Neural Network (SLFN). The hidden layer of SLFN is used as the preprocessor to achieve the minimum output error. To reduce the structural risk and empirical risk of SLFN, the simulation results of 300 groups of spectral data show that the improved FIR-ELM algorithm significantly improves the training accuracy and has good robustness compared with the traditional extreme learning machine algorithm.

An Online Network Intrusion Detection Model Based on Improved Regularized Extreme Learning Machine

Extreme learning machine (ELM) is a novel single-hidden layer feedforward neural network to obtain fast learning speed by randomly initializing weights and deviations. Due to its extremely fast learning speed, it has been widely used in training of massive data in recent years. In order to adapt to the real network environment, based on the ELM, we propose an improved particle swarm optimized online regularized extreme learning machine (IPSO-IRELM) intrusion detection algorithm model. First, the model replaces the traditional batch learning with sequential learning by dynamically adapting the new data obtained in the training network instead of training all collected samples in an offline manner; second, we improve the particle swarm optimization algorithm and compare it with typical improved algorithms to prove its effectiveness; finally, to solve the random initialization problem of IRELM, we use IPSO to optimize the initial weights and deviations of IRELM to improve the classification ability of IRELM. The experimental results show that IPSO-IRELM algorithm has better generalization ability, which not only improves the accuracy of intrusion detection, but also has certain recognition ability for minority class samples.

Optimal Randomness for Stochastic Configuration Network (SCN) with Heavy-Tailed Distributions.

Stochastic Configuration Network (SCN) has a powerful capability for regression and classification analysis. Traditionally, it is quite challenging to correctly determine an appropriate architecture for a neural network so that the trained model can achieve excellent performance for both learning and generalization. Compared with the known randomized learning algorithms for single hidden layer feed-forward neural networks, such as Randomized Radial Basis Function (RBF) Networks and Random Vector Functional-link (RVFL), the SCN randomly assigns the input weights and biases of the hidden nodes in a supervisory mechanism. Since the parameters in the hidden layers are randomly generated in uniform distribution, hypothetically, there is optimal randomness. Heavy-tailed distribution has shown optimal randomness in an unknown environment for finding some targets. Therefore, in this research, the authors used heavy-tailed distributions to randomly initialize weights and biases to see if the new SCN models can achieve better performance than the original SCN. Heavy-tailed distributions, such as Lévy distribution, Cauchy distribution, and Weibull distribution, have been used. Since some mixed distributions show heavy-tailed properties, the mixed Gaussian and Laplace distributions were also studied in this research work. Experimental results showed improved performance for SCN with heavy-tailed distributions. For the regression model, SCN-Lévy, SCN-Mixture, SCN-Cauchy, and SCN-Weibull used less hidden nodes to achieve similar performance with SCN. For the classification model, SCN-Mixture, SCN-Lévy, and SCN-Cauchy have higher test accuracy of 91.5%, 91.7% and 92.4%, respectively. Both are higher than the test accuracy of the original SCN.

Parallel versus cascaded logistic regression trained single-hidden feedforward neural network for medical data

ObjectiveAn important step towards a better healthcare system is fast and accurate diagnosis. In the last decade, the application of intelligent systems in healthcare has led to impressive results. The goal of this paper is to extend the LogSLFN (single-hidden layer feedforward neural network trained using logistic regression) algorithm, which has been deployed successfully in the past for the case of a two-class decision problem, to the case of multiple classes. We have considered and statistically analyzed two approaches: a parallel LogSLFN, and a cascaded LogSLFN. Materials and methodsAccording to the universal approximation theorem, a single-hidden layer feedforward neural network has the ability to approximate arbitrarily closely continuous functions of several real variables under certain reasonable assumptions. Essentially, a single hidden layer containing a finite fixed number of neurons is sufficient to provide an arbitrarily well approximation to a given training set of inputs and a desired target output represented by a continuous function. Parallel LogSLFN and cascaded LogSLFN are two novel approaches that can be applied to multiple-class decision problems. Both methods are extensions of the LogSLFN, which uses logistic regression to compute the weights between the input and hidden layer of a single-hidden layer feedforward network. No error correction is needed, the weights between the hidden and the output layer being computed using the Moore-Penrose pseudoinverse matrix. The proposed models have been tested on two medical datasets regarding cancer diagnosis and liver fibrosis staging. Experimental results and the subsequent statistical analysis have proved the robustness of the proposed models with other machine learning techniques reported in literature. Main findingsThe experimental results showed that the Parallel approach surpasses the Cascaded one. Still, both models are competitive to the other state-of-the-art techniques. ConclusionsThe LogSFLN algorithm can be successfully extended to multiple-class decision problems. By embedding knowledge extracted from the data into the architecture, we obtained a raise by 20% in accuracy when applied on the liver fibrosis dataset.

Intrusion Detection System Based on Gradient Corrected Online Sequential Extreme Learning Machine

Nowadays, Intrusion Detection System (IDS) is an active research topic with machine learning nature. A single-hidden layer feedforward neural network (SLFN) trained on the approach of extreme learning machine (ELM) is used for (IDS). The encouraging factors for its usage are its fast learning and supportability of sequential learning in its online sequential extreme learning machine (OSELM) variant. An issue with OSELM that has been addressed by researchers is its random weights nature of the input-hidden layer. Most approaches use the concept of metaheuristic optimisation for determining the optimal weights of OSELM and resolve the random weight. However, metaheuristic approaches require many trials to determine the optimal one. Hence, there is concern about the convergence aspect and speed. This article proposes a novel approach for finding the optimal weights of the input-hidden layer. This article presents an approach for an integration between OSELM and back-propagation designated as (OSELM-BP). After integration, BP changes the random weights iteratively and uses an iterated evaluation of the generated error for feedback correction of the weights. The approach is evaluated based on various scenarios of activation functions for OSELM on the one hand and the number of iterations for BP on the other. An extensive evaluation of the approach and comparison with the original OSELM reveal a superiority of OSELM-BP in reaching optimal accuracy with a small number of iterations.

On sharpness of error bounds for multivariate neural network approximation

Single hidden layer feedforward neural networks can represent multivariate functions that are sums of ridge functions. These ridge functions are defined via an activation function and customizable weights. The paper deals with best non-linear approximation by such sums of ridge functions. Error bounds are presented in terms of moduli of smoothness. The main focus, however, is to prove that the bounds are best possible. To this end, counterexamples are constructed with a non-linear, quantitative extension of the uniform boundedness principle. They show sharpness with respect to Lipschitz classes for the logistic activation function and for certain piecewise polynomial activation functions. The paper is based on univariate results in Goebbels (Res Math 75(3):1–35, 2020. https://rdcu.be/b5mKH)

Complexity of training ReLU neural network

In this paper, we explore some basic questions on the complexity of training neural networks with ReLU activation function. We show that it is NP-hard to train a two-hidden layer feedforward ReLU neural network. If dimension of the input data and the network topology is fixed, then we show that there exists a polynomial time algorithm for the same training problem. We also show that if sufficient over-parameterization is provided in the first hidden layer of ReLU neural network, then there is a polynomial time algorithm which finds weights such that output of the over-parameterized ReLU neural network matches with the output of the given data.

Road Traffic Prediction Model Using Extreme Learning Machine: The Case Study of Tangier, Morocco

An efficient and credible approach to road traffic management and prediction is a crucial aspect in the Intelligent Transportation Systems (ITS). It can strongly influence the development of road structures and projects. It is also essential for route planning and traffic regulations. In this paper, we propose a hybrid model that combines extreme learning machine (ELM) and ensemble-based techniques to predict the future hourly traffic of a road section in Tangier, a city in the north of Morocco. The model was applied to a real-world historical data set extracted from fixed sensors over a 5-years period. Our approach is based on a type of Single hidden Layer Feed-forward Neural Network (SLFN) known for being a high-speed machine learning algorithm. The model was, then, compared to other well-known algorithms in the prediction literature. Experimental results demonstrated that, according to the most commonly used criteria of error measurements (RMSE, MAE, and MAPE), our model is performing better in terms of prediction accuracy. The use of Akaike’s Information Criterion technique (AIC) has also shown that the proposed model has a higher performance.

Correntropy induced loss based sparse robust graph regularized extreme learning machine for cancer classification

BackgroundAs a machine learning method with high performance and excellent generalization ability, extreme learning machine (ELM) is gaining popularity in various studies. Various ELM-based methods for different fields have been proposed. However, the robustness to noise and outliers is always the main problem affecting the performance of ELM.ResultsIn this paper, an integrated method named correntropy induced loss based sparse robust graph regularized extreme learning machine (CSRGELM) is proposed. The introduction of correntropy induced loss improves the robustness of ELM and weakens the negative effects of noise and outliers. By using the L2,1-norm to constrain the output weight matrix, we tend to obtain a sparse output weight matrix to construct a simpler single hidden layer feedforward neural network model. By introducing the graph regularization to preserve the local structural information of the data, the classification performance of the new method is further improved. Besides, we design an iterative optimization method based on the idea of half quadratic optimization to solve the non-convex problem of CSRGELM.ConclusionsThe classification results on the benchmark dataset show that CSRGELM can obtain better classification results compared with other methods. More importantly, we also apply the new method to the classification problems of cancer samples and get a good classification effect.