Single Hidden Layer Feedforward Network Research Articles

Robust noise-aware algorithm for randomized neural network and its convergence properties

The concept of randomized neural networks (RNNs), such as the random vector functional link network (RVFL) and extreme learning machine (ELM), is a widely accepted and efficient network method for constructing single-hidden layer feedforward networks (SLFNs). Due to its exceptional approximation capabilities, RNN is being extensively used in various fields. While the RNN concept has shown great promise, its performance can be unpredictable in imperfect conditions, such as weight noises and outliers. Thus, there is a need to develop more reliable and robust RNN algorithms. To address this issue, this paper proposes a new objective function that addresses the combined effect of weight noise and training data outliers for RVFL networks. Based on the half-quadratic optimization method, we then propose a novel algorithm, named noise-aware RNN (NARNN), to optimize the proposed objective function. The convergence of the NARNN is also theoretically validated. We also discuss the way to use the NARNN for ensemble deep RVFL (edRVFL) networks. Finally, we present an extension of the NARNN to concurrently address weight noise, stuck-at-fault, and outliers. The experimental results demonstrate that the proposed algorithm outperforms a number of state-of-the-art robust RNN algorithms.

Neural Network-Based Hammerstein Model Identification of a Lab-Scale Batch Reactor.

This paper focuses on two types of neural network-based Hammerstein model identification methods for the acrylamide polymerization reaction of a batch reactor process. The first neural-based identification type formulates the weights of the multilayer network directly as parameters of the nonlinear static and linear dynamic blocks of the Hammerstein model and trains the weights using a gradient-based backpropagation algorithm. In the second identification type, the nonlinear static block of the Hammerstein model is framed as a single hidden-layer feedforward network and both nonlinear and linear block parameters are trained using an extreme learning machine, where the training procedure is exempted from gradient calculation. The primary focus of the paper is neural-based model identification of a complex nonlinear system, which facilitates ease of linear/nonlinear controller design with good learning speed and less computations. A future work toward the machine learning-based nonlinear model predictive controller implementation using the Jetson Orin Nano board is also described.

Classification of Program Texts Represented as Markov Chains with Biology-Inspired Algorithms-Enhanced Extreme Learning Machines

The massive nature of modern university programming courses increases the burden on academic workers. The Digital Teaching Assistant (DTA) system addresses this issue by automating unique programming exercise generation and checking, and provides means for analyzing programs received from students by the end of semester. In this paper, we propose a machine learning-based approach to the classification of student programs represented as Markov chains. The proposed approach enables real-time student submissions analysis in the DTA system. We compare the performance of different multi-class classification algorithms, such as support vector machine (SVM), the k nearest neighbors (KNN) algorithm, random forest (RF), and extreme learning machine (ELM). ELM is a single-hidden layer feedforward network (SLFN) learning scheme that drastically speeds up the SLFN training process. This is achieved by randomly initializing weights of connections among input and hidden neurons, and explicitly computing weights of connections among hidden and output neurons. The experimental results show that ELM is the most computationally efficient algorithm among the considered ones. In addition, we apply biology-inspired algorithms to ELM input weights fine-tuning in order to further improve the generalization capabilities of this algorithm. The obtained results show that ELMs fine-tuned with biology-inspired algorithms achieve the best accuracy on test data in most of the considered problems.

A multi-layered intrusion detection system for software defined networking

The majority of existing DDoS defense mechanisms in SDN impose a significant computational burden on the controller and employ limited flow statistics and packet features. Tackling these issues, this paper presents a multi-layer defense mechanism that detects and mitigates three distinct types of flooding DDoS attacks. In the proposed framework, the detection process consists of flow-based and packet-based attack detection mechanisms employing Extreme Learning Machine-based Single-hidden Layer Feedforward Networks (ELM-SLFNs) and Case-based Information Entropy (C-IE), respectively. Moreover, the affected switches are avoided in the optimal path determined by the Floyd-Warshall algorithm, where the switches are classified based on the Hidden Markov Model (HMM) using the extracted packet features. Our simulation demonstrates the improved performance of our framework compared to similar schemes proposed in the literature in terms of different metrics, including attack detection rate, detection accuracy, false-positive rate, switch failure ratio, packet loss rate, response time, and CPU utilization.

Convergence and gradient algorithm of a class of neural networks based on the polygonal fuzzy numbers representation

As a special case of general fuzzy numbers, the polygonal fuzzy number can describe a fuzzy object by means of an ordered representation of finite real numbers. Different from general fuzzy numbers, the polygonal fuzzy numbers overcome the shortcoming of complex operations based on Zadeh’s traditional expansion principle, and can maintain the closeness of arithmetic operation. Hence, it is feasible to use a polygonal fuzzy number to approximate a general fuzzy number. First, an extension theorem of continuous functions on a real compact set is given according to open set construction theorem. Then using Weierstrass approximation theorem and ordered representation of the polygonal fuzzy numbers, the convergence of a single hidden layer feedforward polygonal fuzzy neural network is proved. Secondly, the gradient vector of the approximation error function and the optimization parameter vector of the network are given by using the ordered representation of polygonal fuzzy numbers, and then the gradient descent algorithm is used to train the optimal parameters of the polygonal fuzzy neural network iteratively. Finally, two simulation examples are given to verify the approximation ability of the network. Simulation result shows that the proposed network and the gradient descent algorithm are effective, and the single hidden layer feedforward network have good abilities in learning and generalization.

Sliding Mode Control for Hypersonic Vehicle Based on Extreme Learning Machine Neural Network Disturbance Observer

The novel extreme learning machine (ELM) neural network disturbance observer (NNDO) - based sliding mode control (SMC) strategy is proposed for the precise tracking control of a hypersonic vehicle (HV) under various disturbance situations. By converting nonlinear dynamics into state-dependent linear model, the control law design process is simplified, and the sliding mode control law based on the power function reaching rate is designed to suppress the chattering effect. Considering the disturbances, the ELM-NNDO is designed based on the single-hidden layer feedforward network (SLFN). Different from conventional ELM using least square optimization approach, the output weight here is updated based on the Lyapunov synthesis approach. In addition, the influences of the disturbances on the velocity and altitude are attenuated by the direct feedback compensation (DFC), and the offset-free tracking control is realized for the output reference signal. Comparison of simulation results verify the superior control performance of the proposed method.

Multi-feature learning-based extreme learning machine for rolling bearing fault diagnosis

Rolling bearing has been becoming an important part of human life and work. The working environment of rolling bearing is very complex and variable, which makes it difficult for fault diagnosis and monitor of rolling bearing from raw vibration data. Then, in this paper, a novel multi-feature learning-based extreme learning machine is proposed for rolling bearing fault diagnosis (FL-ELM). Extreme learning machine (ELM) is a fast and generalized algorithm proposed for training single-hidden-layer feed-forward networks (SLFNs), which has fast computing speed and small testing error. The novel architecture has two hidden layers and an experience pool sandwiched between two hidden layers. The first hidden layer consists of multi-feature learning methods. The experience pool is used to sort and choose new data, with old data being filtered out. Firstly, the first hidden layer is adopted for feature extraction. Secondly, the experience pool is used to rearrange and select data, which is extracted by first hidden layer. Thirdly, ELM is employed to further learn and classify. The proposed method (FL-ELM) is applied to the rolling bearing fault diagnosis. The results confirm that the proposed method is more effective than traditional methods and standard deep learning methods.

Fault‐tolerant control design for a class of nonlinear systems with actuator malfunctions

AbstractThis article presents a single‐hidden‐layer feedforward network (SLFN) aided fault‐tolerant control (FTC) scheme for a class of nonlinear systems subject to actuator faults. First, the upper bounds of the norms of the unknown functions, which cover information of actuator faults and amplitude saturation degrees, are deployed. The upper bounds are approximated by SLFNs with adaptive techniques. Subsequently, a sliding mode control approach is presented to allow for prompt corrective reactions, with explicit consideration of multivariable conditions and control input constraints. Finally, simulation results based on a nonlinear Boeing 747 aircraft model are presented to demonstrate the effectiveness of the proposed FTC scheme.

Radar HRRP Target Recognition Based on Dynamic Learning with Limited Training Data

For high-resolution range profile (HRRP)-based radar automatic target recognition (RATR), adequate training data are required to characterize a target signature effectively and get good recognition performance. However, collecting enough training data involving HRRP samples from each target orientation is hard. To tackle the HRRP-based RATR task with limited training data, a novel dynamic learning strategy is proposed based on the single-hidden layer feedforward network (SLFN) with an assistant classifier. In the offline training phase, the training data are used for pretraining the SLFN using a reduced kernel extreme learning machine (RKELM). In the online classification phase, the collected test data are first labeled by fusing the recognition results of the current SLFN and assistant classifier. Then the test samples with reliable pseudolabels are used as additional training data to update the parameters of SLFN with the online sequential RKELM (OS-RKELM). Moreover, to improve the accuracy of label estimation for test data, a novel semi-supervised learning method named constraint propagation-based label propagation (CPLP) was developed as an assistant classifier. The proposed method dynamically accumulates knowledge from training and test data through online learning, thereby reinforcing performance of the RATR system with limited training data. Experiments conducted on the simulated HRRP data from 10 civilian vehicles and real HRRP data from three military vehicles demonstrated the effectiveness of the proposed method when the training data are limited.

Parallel versus cascaded logistic regression trained single-hidden feedforward neural network for medical data

ObjectiveAn important step towards a better healthcare system is fast and accurate diagnosis. In the last decade, the application of intelligent systems in healthcare has led to impressive results. The goal of this paper is to extend the LogSLFN (single-hidden layer feedforward neural network trained using logistic regression) algorithm, which has been deployed successfully in the past for the case of a two-class decision problem, to the case of multiple classes. We have considered and statistically analyzed two approaches: a parallel LogSLFN, and a cascaded LogSLFN. Materials and methodsAccording to the universal approximation theorem, a single-hidden layer feedforward neural network has the ability to approximate arbitrarily closely continuous functions of several real variables under certain reasonable assumptions. Essentially, a single hidden layer containing a finite fixed number of neurons is sufficient to provide an arbitrarily well approximation to a given training set of inputs and a desired target output represented by a continuous function. Parallel LogSLFN and cascaded LogSLFN are two novel approaches that can be applied to multiple-class decision problems. Both methods are extensions of the LogSLFN, which uses logistic regression to compute the weights between the input and hidden layer of a single-hidden layer feedforward network. No error correction is needed, the weights between the hidden and the output layer being computed using the Moore-Penrose pseudoinverse matrix. The proposed models have been tested on two medical datasets regarding cancer diagnosis and liver fibrosis staging. Experimental results and the subsequent statistical analysis have proved the robustness of the proposed models with other machine learning techniques reported in literature. Main findingsThe experimental results showed that the Parallel approach surpasses the Cascaded one. Still, both models are competitive to the other state-of-the-art techniques. ConclusionsThe LogSFLN algorithm can be successfully extended to multiple-class decision problems. By embedding knowledge extracted from the data into the architecture, we obtained a raise by 20% in accuracy when applied on the liver fibrosis dataset.

Multi-step wind speed forecast based on sample clustering and an optimized hybrid system

At present, accurate forecast of very-short-term wind speed is still a critical issue, mainly due to the complex characteristics of wind variations such as intermittence, fluctuation and randomness. On this topic, our paper contributes to the development of an effective multi-step forecasting method termed ECKIE, which provides multi-step forecast for the very-short-term wind speed in specific stations. This method consists of three stages: a data filtering process driven by the ensemble empirical mode decomposition (EEMD), an improved K-harmonic mean (KHM) clustering optimized by the Cuckoo search (CS) algorithm and a single-hidden-layer feedforward network (SLFN) trained by the incremental extreme learning machine (IELM) algorithm. The developed method is capable of clustering the model inputs into groups according to their characteristics and of constructing the models for each group. It is further capable of reducing forecasting errors by choosing a suitable model. It is a purely data-driven process and is an effective method for very-short-term wind speed forecasts. The simulation demonstrates that the developed method drastically improves upon original model performance and performs the best among comparable models.

Extreme Learning-Based Monocular Visual Servo of an Unmanned Surface Vessel

In this article, suffering from unmatched visual-servo uncertainties and unknown dynamics/disturbances, an extreme learning-based monocular visual-servo (ELMVS) scheme is developed for maneuvering an unmanned surface vessel (USV) to reach the desired pose. By virtue of the backstepping philosophy, complex visual-servo unknowns are elaborately encapsulated into lumped nonlinearities, which are further accurately accommodated by devising a single-hidden layer feedforward network based adaptive compensating identifier (SACI). Within the SACI architecture, hidden nodes are completely model free and are randomly generated without tedious learning, and thereby dramatically expediting fast-dynamics identification. Moreover, by exploiting approximation residuals, direct hyperbolic-tangent links between input and output layers are deployed to enhance identification accuracy. Eventually, the Lyapunov synthesis guarantees that the proposed ELMVS scheme can asymptotically render visual-servo errors arbitrarily small while target features can be kept within the field of view. Remarkable performance and superiority is finally demonstrated on a prototype USV.

Adaptive online sequential extreme learning machine for dynamic modeling

Extreme learning machine (ELM) is an emerging machine learning algorithm for training single-hidden-layer feedforward networks (SLFNs). The salient features of ELM are that its hidden layer parameters can be generated randomly, and only the corresponding output weights are determined analytically through the least-square manner, so it is easier to be implemented with faster learning speed and better generalization performance. As the online version of ELM, online sequential ELM (OS-ELM) can deal with the sequentially coming data one by one or chunk by chunk with fixed or varying chunk size. However, OS-ELM cannot function well in dealing with dynamic modeling problems due to the data saturation problem. In order to tackle this issue, in this paper, we propose a novel OS-ELM, named adaptive OS-ELM (AOS-ELM), for enhancing the generalization performance and dynamic tracking capability of OS-ELM for modeling problems in nonstationary environments. The proposed AOS-ELM can efficiently reduce the negative effects of the data saturation problem, in which approximate linear dependence (ALD) and a modified hybrid forgetting mechanism (HFM) are adopted to filter the useless new data and alleviate the impacts of the outdated data, respectively. The performance of AOS-ELM is verified using selected benchmark datasets and a real-world application, i.e., device-free localization (DFL), by comparing it with classic ELM, OS-ELM, FOS-ELM, and DU-OS-ELM. Experimental results demonstrate that AOS-ELM can achieve better performance.

Robust extreme learning machine for modeling with unknown noise

Extreme learning machine (ELM) is an emerging machine learning technique for training single hidden layer feedforward networks (SLFNs). During the training phase, ELM model can be created by simultaneously minimizing the modeling errors and norm of the output weights. Usually, squared loss is widely utilized in the objective function of ELMs, which is theoretically optimal for the Gaussian error distribution. However, in practice, data collected from uncertain and heterogeneous environments trivially result in unknown noise, which may be very complex and cannot be described well using any single distribution. In order to tackle this issue, in this paper, a robust ELM (R-ELM) is proposed for improving the modeling capability and robustness with Gaussian and non-Gaussian noise. In R-ELM, a modified objective function is constructed to fit the noise using mixture of Gaussian (MoG) to approximate any continuous distribution. In addition, the corresponding solution for the new objective function is developed based on expectation maximization (EM) algorithm. Comprehensive experiments, both on selected benchmark datasets and real world applications, demonstrate that the proposed R-ELM has better robustness and generalization performance than state-of-the-art machine learning approaches.

Smoothing L0 Regularization for Extreme Learning Machine

Extreme learning machine (ELM) has been put forward for single hidden layer feedforward networks. Because of its powerful modeling ability and it needs less human intervention, the ELM algorithm has been used widely in both regression and classification experiments. However, in order to achieve required accuracy, it needs many more hidden nodes than is typically needed by the conventional neural networks. This paper considers a new efficient learning algorithm for ELM with smoothing L0 regularization. A novel algorithm updates weights in the direction along which the overall square error is reduced the most and then this new algorithm can sparse network structure very efficiently. The numerical experiments show that the ELM algorithm with smoothing L0 regularization has less hidden nodes but better generalization performance than original ELM and ELM with L1 regularization algorithms.

Robust adaptive integral terminal sliding mode control for steer-by-wire systems based on extreme learning machine

In this paper, a novel extreme-learning-machine (ELM)-based robust adaptive integral terminal sliding mode (AITSM) control strategy is developed for the precise tracking control of a steer-by-wire (SBW) system with uncertain dynamics. The proposed control not only ensures the finite-time error convergence but also effectively estimates the lumped uncertainty via a single-hidden layer feedforward network (SLFN) with ELM. Different from conventional ELM using least square optimization approach, the ELM in this work is designed to adaptively estimate the lumped uncertainty from the perspective of global stability of the closed-loop system. The stability of the closed-loop control system is proved in Lyapunov sense. Simulations are carried out to demonstrate the superior control performance of the proposed control.

A new 3D segmentation approach using extreme learning machine algorithm and morphological operations

Segmentation is one of the most crucial steps of image processing. Because 3D images contain depth information, they have gradually gained importance for numerical systems in image analysis. In the present study, a new 3D segmentation method based on extreme learning machine and morphological operations (3DS-ELM) is proposed. The present study benefits from extreme learning machine (ELM) algorithm, which is a novel and fast learning algorithm for single-hidden layer feedforward networks (SLFNs), for training objects. Because a 3D model contains many points, direct segmentation on a 3D model is time-consuming and causes problems in the segmentation process, the proposed approach minimizes these problems and offers a quick and high-performance 3D segmentation method that can be used in various industrial fields. The proposed 3DS-ELM was compared with different approaches in order to analyze its 3D segmentation performance. Experimental studies proved that the proposed 3DS-ELM performed better than other approaches.

Relaxation of the Radio-Frequency Linewidth for Coherent-Optical Orthogonal Frequency-Division Multiplexing Schemes by Employing the Improved Extreme Learning Machine

A coherent optical (CO) orthogonal frequency division multiplexing (OFDM) scheme gives a scalable and flexible solution for increasing the transmission rate, being extremely robust to chromatic dispersion as well as polarization mode dispersion. Nevertheless, as any coherent-detection OFDM system, the overall system performance is limited by laser phase noises. On the other hand, extreme learning machines (ELMs) have gained a lot of attention from the machine learning community owing to good generalization performance, negligible learning speed, and minimum human intervention. In this manuscript, a phase-error mitigation method based on the single-hidden layer feedforward network prone to the improved ELM algorithm for CO-OFDM systems is introduced for the first time. In the training step, two steps are distinguished. Firstly, pilots are used, which is very common in OFDM-based systems, to diminish laser phase noises as well as to correct frequency-selective impairments and, therefore, the bandwidth efficiency can be maximized. Secondly, the regularization parameter is included in the ELM to balance the empirical and structural risks, namely to minimize the root mean square error in the test stage and, consequently, the bit error rate (BER) metric. The operational principle of the real-complex (RC) ELM is analytically explained, and then, its sub-parameters (number of hidden neurons, regularization parameter, and activation function) are numerically found in order to enhance the system performance. For binary and quadrature phase-shift keying modulations, the RC-ELM outperforms the benchmark pilot-assisted equalizer as well as the fully-real ELM, and almost matches the common phase error (CPE) compensation and the ELM defined in the complex domain (C-ELM) in terms of the BER over an additive white Gaussian noise channel and different laser oscillators. However, both techniques are characterized by the following disadvantages: the CPE compensator reduces the transmission rate since an additional preamble is mandatory for channel estimation purposes, while the C-ELM requires a bounded and differentiable activation function in the complex domain and can not follow semi-supervised training. In the same context, the novel ELM algorithm can not compete with the CPE compensator and C-ELM for the 16-ary quadrature amplitude modulation. On the other hand, the novel ELM exposes a negligible computational cost with respect to the C-ELM and PAE methods.

A competitive swarm optimizer with hybrid encoding for simultaneously optimizing the weights and structure of Extreme Learning Machines for classification problems

Extreme Learning Machine (ELM) is a learning algorithm proposed recently to train single hidden layer feed forward networks (SLFN). It has many attractive properties that include better generalization performance and very fast learning. ELM starts by assigning random values to the input weights and hidden biases and then in one step it determines the output weights using Moore-Penrose generalized inverse. Despite the aforementioned advantages, ELM performance might be affected by the random initialization of weights and biases or by the large generated network which might contain unnecessary number of neurons. In order to increase the generalization performance and to produce more compact networks, a hybrid model that combines ELM with competitive swarm optimizer (CSO) is proposed in this paper. The proposed model (CSONN-ELM) optimizes the weights and biases and dynamically determines the most appropriate number of neurons. To evaluate the effectiveness of the CSONN-ELM, it is experimented using 23 benchmark datasets, and compared to a set of static rules extracted from literature that are used to determine the number of neurons of SLFN. Moreover, it is compared to two dynamic methods that are used to enhance the performance of ELM, that are Optimally pruned ELM (OP-ELM) and metaheuristic based ELMs (Particle Swarm Optimization-ELM and Differential Evolution-ELM). The obtained results show that the proposed method enhances the generalization performance of ELM and overcomes the static and dynamic methods.

Multiple-Instance Learning Approach via Bayesian Extreme Learning Machine

Multiple-instance learning (MIL) can solve supervised learning tasks, where only a bag of multiple instances is labeled, instead of a single instance. It is considerably important to develop effective and efficient MIL algorithms, because real-world datasets usually contain large instances. Known for its good generalization performance, MIL based on extreme learning machines (ELM-MIL) has proven to be more efficient than several typical MIL classification methods. ELM-MIL selects the most qualified instances from each bag through a single hidden layer feedforward network (SLFN) and trains modified ELM models to update the output weights. This learning approach often performs susceptible to the number of hidden nodes and can easily suffer from over-fitting problem. Using Bayesian inferences, this study introduces a Bayesian ELM (BELM)-based MIL algorithm (BELM-MIL) to address MIL classification problems. First, weight self-learning method based on a Bayesian network is applied to determine the weights of instance features. The most qualified instances are then selected from each bag to represent the bag. Second, BELM can improve the classification model via regularization of automatic estimations to reduce possible over-fitting during the calibration process. Experiments and comparisons are conducted with several competing algorithms on Musk datasets, images datasets, and inductive logic programming datasets. Superior classification accuracy and performance are demonstrated by BELM-MIL.