Feed-forward Neural Network Algorithm Research Articles

An Optimal PID Control Algorithm for Training Feedforward Neural Networks

The training problem of feedforward neural networks (FNNs) is formulated into a proportional integral and derivative (PID) control problem of a linear discrete dynamic system in terms of the estimation error. The robust control approach greatly facilitates the analysis and design of robust learning algorithms for multiple-input-multiple-output (MIMO) FNNs using robust control methods. The drawbacks of some existing learning algorithms can therefore be revealed clearly, and an optimal robust PID-learning algorithm is developed. The optimal learning parameters can be found by utilizing linear matrix inequality optimization techniques. Theoretical analysis and examples including function approximation, system identification, exclusive-or (XOR) and encoder problems are provided to illustrate the results.

Diffusion learning algorithms for feedforward neural networks

The problem of training feedforward neural networks is considered. To solve it, new algorithms are proposed. They are based on the asymptotic analysis of the extended Kalman filter (EKF) and on a separable network structure. Linear weights are interpreted as diffusion random variables with zero expectation and a covariance matrix proportional to an arbitrarily large parameter λ. Asymptotic expressions for the EKF are derived as λ→∞. They are called diffusion learning algorithms (DLAs). It is shown that they are robust with respect to the accumulation of rounding errors in contrast to their prototype EKF with a large but finite λ and that, under certain simplifying assumptions, an extreme learning machine (ELM) algorithm can be obtained from a DLA. A numerical example shows that the accuracy of a DLA may be higher than that of an ELM algorithm.

Extreme learning machine: algorithm, theory and applications

Extreme learning machine (ELM) is a new learning algorithm for the single hidden layer feedforward neural networks. Compared with the conventional neural network learning algorithm it overcomes the slow training speed and over-fitting problems. ELM is based on empirical risk minimization theory and its learning process needs only a single iteration. The algorithm avoids multiple iterations and local minimization. It has been used in various fields and applications because of better generalization ability, robustness, and controllability and fast learning rate. In this paper, we make a review of ELM latest research progress about the algorithms, theory and applications. It first analyzes the theory and the algorithm ideas of ELM, then tracking describes the latest progress of ELM in recent years, including the model and specific applications of ELM, finally points out the research and development prospects of ELM in the future.

Artificial Neural Network Modeling of Deposition Thickness Through the Pipeline

Modeling and prediction of deposition thickness is a key to developing crude oil transportation process. Usually, the heat transfer method has been used to calculate deposition thickness through the pipeline, but this method requires vast information and cumbersome calculations. To eliminate this problem, a feed-forward multilayer perceptron neural network (MLPNN) algorithm has been applied to predict the deposition thickness through the pipeline. We present an optimized one-layer feed-forward neural network using properties of the oil pipeline such as inlet and outlet oil temperatures, environmental (mixture of water and ethylene glycol) temperature, and oil Reynolds’ numbers. Different networks are considered and trained using 15313 data sets; the accuracy of the proposed network is validated by 5104 testing data sets. Using validating data set, the network that is having the lowest mean square error (MSE) and average absolute relative deviation percent (AARD%) and the highest regression coefficient (R2) is selected as an optimal configuration. Statistical analysis show that the artificial neural network (ANN) predictions have an excellent agreement (MSE = 9.4949e-006, AARD% = 1.0109 and R2 = 0.9999) with the heat transfer method. In the other words, accuracy of the proposed ANN model is approximately equal to the heat transfer method without required too much information and cumbersome calculations.

Indian summer monsoon rainfall prediction using artificial neural network

Forecasting the monsoon temporally is a major scientific issue in the field of monsoon meteorology. The ensemble of statistics and mathematics has increased the accuracy of forecasting of Indian summer monsoon rainfall (ISMR) up to some extent. But due to the nonlinear nature of ISMR, its forecasting accuracy is still below the satisfactory level. Mathematical and statistical models require complex computing power. Therefore, many researchers have paid attention to apply artificial neural network in ISMR forecasting. In this study, we have used feed-forward back-propagation neural network algorithm for ISMR forecasting. Based on this algorithm, we have proposed the five neural network architectures designated as BP1, BP2, $$\ldots, $$ BP5 using three layers of neurons (one input layer, one hidden layer and one output layer). Detail architecture of the neural networks is provided in this article. Time series data set of ISMR is obtained from Pathasarathy et al. (Theor Appl Climatol 49:217–224 1994) (1871–1994) and IITM ( http://www.tropmet.res.in/ , 2012) (1995–2010) for the period 1871–2010, for the months of June, July, August and September individually, and for the monsoon season (sum of June, July, August and September). The data set is trained and tested separately for each of the neural network architecture, viz., BP1–BP5. The forecasted results obtained for the training and testing data are then compared with existing model. Results clearly exhibit superiority of our model over the considered existing model. The seasonal rainfall values over India for next 5 years have also been predicted.

Evolutionary Extreme Learning Machine and Its Application to Image Analysis

Extreme learning machine (ELM) and evolutionary ELM (E-ELM) were proposed as a new class of learning algorithm for single-hidden layer feedforward neural network (SLFN). In order to achieve good generalization performance, E-ELM calculates the error on a subset of testing data for parameter optimization. Since E-ELMemploys extra data for validation to avoid the overfitting problem, more samples are needed for model training. In this paper, the cross-validation strategy is proposed to be embedded into the training phase so as to solve the overtraining problem. Based on this new learning structure, two extensions of E-ELM are introduced. Experimental results demonstrate that the proposed algorithms are efficient for image analysis.

A Novel Extreme Learning Machine Based on Hybrid Kernel Function

Extreme learning machine is a new learning algorithm for the single hidden layer feedforward neural networks (SLFNs). ELM has been widely used in various fields and applications to overcome the slow training speed and over-fitting problems of the conventional neural network learning algorithms. ELM algorithm is based on the empirical risk minimization, without considering the structural risk and this may lead to over-fitting problems and at the same time, it is with poor controllability and robustness. For these deficiencies, an optimization method is proposed in this paper, a novel extreme learning machine based on hybrid kernel function (HKELM). The method constructs a hybrid kernel function with better performance by fully combining local kernel function strong learning ability and global kernel function strong generalization ability. Compared with traditional ELM, the results show that this method can effectively improve the ELM classification results, avoid local minimum, with better generalization, robustness, controllability and faster learning rate.

An Algorithm Research for Prediction of Extreme Learning Machines Based on Rough Sets

As is known, the extreme learning machine (ELM) algorithm is a new learning algorithm for the single hidden layer feedforward neural networks (SLFNs). This paper combines rough sets theory with extreme learning machine and proposes a new prediction algorithm of extreme learning machines based on rough sets theory. Firstly, using the rough sets theory to do attribute reduction, and then using ELM to train and predict the new datasets. Verified by the final experimental results and data analysis we know that compared with the traditional ELM the proposed algorithm has higher prediction accuracy and better efficiency.

ELM-based sensorless speed control of permanent magnet synchronous machine

This paper deals with Extreme Learning Machine (ELM) based sensorless speed estimation and speed control of Permanent Magnet Synchronous Machines (PMSMs). ELM, first proposed by G.B. Huang as a new class of learning algorithm for Single-Hidden Layer Feedforward Neural Networks (SLFNs), is extremely fast and accurate, and has better generalisation performance than the traditional gradient-based training methods. To implement Field-Oriented Control (FOC) in PMSMs, the stator magnetic field is always kept 90 degrees ahead of the rotor. This requires rotor position information all the time. This information is accurately obtained with an ELM-based observer without the position sensor for PMSMs, and hence, the cost of the system is reduced, while the problems associated with the sensors are minimised.

Toward the scalability of neural networks through feature selection

In the past few years, the bottleneck for machine learning developers is not longer the limited data available but the algorithms inability to use all the data in the available time. For this reason, researches are now interested not only in the accuracy but also in the scalability of the machine learning algorithms. To deal with large-scale databases, feature selection can be helpful to reduce their dimensionality, turning an impracticable algorithm into a practical one. In this research, the influence of several feature selection methods on the scalability of four of the most well-known training algorithms for feedforward artificial neural networks (ANNs) will be analyzed over both classification and regression tasks. The results demonstrate that feature selection is an effective tool to improve scalability.

A new automatic target recognition system based on wavelet extreme learning machine

In this paper, an automatic system is presented for target recognition using target echo signals of High Resolution Range (HRR) radars. This paper especially deals with combination of the feature extraction and classification from measured real target echo signal waveforms by using X-band pulse radar. The past studies in the field of radar target recognition have shown that the learning speed of feedforward neural networks is in general much slower than required and it has been a major disadvantage. There are two key reasons forth is status of feedforward neural networks: (1) the slow gradient-based learning algorithms are extensively used to train neural networks, and (2) all the parameters of the networks are tuned iteratively by using such learning algorithms (Feng, Huang, Lin, & Gay, 2009; Huang & Siew, 2004, 2005; Huang & Chen, 2007, 2008; Huang, Chen, & Siew, 2006; Huang, Ding, & Zhou, 2010; Huang, Zhu, & Siew, 2004; Huang, Liang, Rong, Saratchandran, & Sundararajan, 2005; Huang, Zhou, Ding, & Zhang, 2012; Huang, Li, Chen, & Siew, 2008; Huang, Wang, & Lan, 2011; Huang et al., 2006; Huang, Zhu, & Siew, 2006a, 2006b; Lan, Soh, & Huang, 2009; Li, Huang, Saratchandran, & Sundararajan, 2005; Liang, Huang, Saratchandran, & Sundararajan, 2006; Liang, Saratchandran, Huang, & Sundararajan, 2006; Rong, Huang, Saratchandran, & Sundararajan, 2009; Wang & Huang, 2005; Wang, Cao, & Yuan, 2011; Yeu, Lim, Huang, Agarwal, & Ong, 2006; Zhang, Huang, Sundararajan, & Saratchandran, 2007; Zhu, Qin, Suganthan, & Huang, 2005). To resolve these disadvantages of feedforward neural networks for automatic target recognition area in this paper suggested a new learning algorithm called extreme learning machine (ELM) for single-hidden layer feedforward neural networks (SLFNs) (Feng, Huang, Lin, & Gay, 2009; Huang & Siew, 2004, 2005; Huang & Chen, 2007, 2008; Huang, Chen, & Siew, 2006; Huang, Ding, & Zhou, 2010; Huang, Zhu, & Siew, 2004; Huang, Liang, Rong, Saratchandran, & Sundararajan, 2005; Huang, Zhou, Ding, & Zhang, 2012; Huang, Li, Chen, & Siew, 2008; Huang, Wang, & Lan, 2011; Huang et al., 2006; Huang, Zhu, & Siew, 2006a, 2006b; Lan, Soh, & Huang, 2009; Li, Huang, Saratchandran, & Sundararajan, 2005; Liang, Huang, Saratchandran, & Sundararajan, 2006; Liang, Saratchandran, Huang, & Sundararajan, 2006; Rong, Huang, Saratchandran, & Sundararajan, 2009; Wang & Huang, 2005; Wang, Cao, & Yuan, 2011; Yeu, Lim, Huang, Agarwal, & Ong, 2006; Zhang, Huang, Sundararajan, & Saratchandran, 2007; Zhu, Qin, Suganthan, & Huang, 2005) which randomly choose hidden nodes and analytically determines the output weights of SLFNs. In theory, this algorithm tends to provide good generalization performance at extremely fast learning speed. Moreover, the Discrete Wavelet Transform (DWT) and wavelet entropy is used for adaptive feature extraction in the time-frequency domain in feature extraction stage to strengthen the premium features of the ELM in this study. The correct recognition performance of this new system is compared with feedforward neural networks. The experimental results show that the new algorithm can produce good generalization performance in most cases and can learn thousands of times faster than conventional popular learning algorithms for feedforward neural networks.

A new optimization algorithm for single hidden layer feedforward neural networks

Feedforward neural networks are the most commonly used function approximation techniques in neural networks. By the universal approximation theorem, it is clear that a single-hidden layer feedforward neural network (FNN) is sufficient to approximate the corresponding desired outputs arbitrarily close. Some researchers use genetic algorithms (GAs) to explore the global optimal solution of the FNN structure. However, it is rather time consuming to use GA for the training of FNN. In this paper, we propose a new optimization algorithm for a single-hidden layer FNN. The method is based on the convex combination algorithm for massaging information in the hidden layer. In fact, this technique explores a continuum idea which combines the classic mutation and crossover strategies in GA together. The proposed method has the advantage over GA which requires a lot of preprocessing works in breaking down the data into a sequence of binary codes before learning or mutation can apply. Also, we set up a new error function to measure the performance of the FNN and obtain the optimal choice of the connection weights and thus the nonlinear optimization problem can be solved directly. Several computational experiments are used to illustrate the proposed algorithm, which has good exploration and exploitation capabilities in search of the optimal weight for single hidden layer FNNs.

Decouple implementation of weight decay for recursive least square

In the conventional recursive least square (RLS) algorithm for multilayer feedforward neural networks, controlling the initial error covariance matrix can limit weight magnitude. However, the weight decay effect decreases linearly as the number of learning epochs increases. Although we can modify the original RLS algorithm to maintain a constant weight decay effect, the computational and space complexities of the modified RLS algorithm are very high. This paper first presents a set of more compact RLS equations for this modified RLS algorithm. Afterwards, to reduce the computational and space complexities, we propose a decoupled version for this algorithm. The effectiveness of this decoupled algorithm is demonstrated by computer simulations.

A feed-forward neural network algorithm to detect thermal lesions induced by high intensity focused ultrasound in tissue

Non-invasive ultrasound surgeries such as high intensity focused ultrasound have been developed to treat tumors or to stop bleeding. In this technique, incorporation of a suitable imaging modality to monitor and control the treatments is essential so several imaging methods such as X-ray, Magnetic resonance imaging and ultrasound imaging have been proposed to monitor the induced thermal lesions. Currently, the only ultrasound imaging technique that is clinically used for monitoring this treatment is standard pulse-echo B-mode ultrasound imaging. This paper describes a novel method for detecting high intensity focused ultrasound-induced thermal lesions using a feed forward neural-network. This study was carried on in vitro animal tissue samples. Backscattered radio frequency signals were acquired in real-time during treatment in order to detect induced thermal lesions. Changes in various tissue properties including tissue's attenuation coefficient, integrated backscatter, scaling parameter of Nakagami distribution, frequency dependent scatterer amplitudes and tissue vibration derived from the backscattered radio frequency data acquired 10 minutes after treatment regarding to before treatment were used in this study. These estimated parameters were used as features of the neural network. Estimated parameters of two sample tissues including two thermal lesions and their segmented B-mode images were used along with the pathological results as training data for the neural network. The results of the study shows that the trained feed forward neural network could effectively detect thermal lesions in vitro. Comparing the estimated size of the thermal lesion (9.6 mm × 8.5 mm) using neural network with the actual size of that from physical examination (10.1 mm × 9 mm) shows that we could detect high intensity focused ultrasound thermal lesions with the difference of 0.5 mm × 0.5 mm.

Conjugate Gradient Back-propagation with Modified Polack –Rebier updates for training feed forward neural network

Several learning algorithms for feed-forward (FFN) neural networks have been developed, many of these algorithms are based on the gradient descent algorithm well-known in optimization theory which have poor performance in practical applications. In this paper we modify the Polak-Ribier conjugate gradient method to train feed forward neural network. Our modification is based on the secant equation (Quasi-Newton condition). The suggested algorithm is tested on some well known test problems and compared with other algorithms in this field.

A New Learning Algorithm for Single hidden Layer Feedforward Neural Networks

high dimensional pattern recognition problems, the learning speed of gradient based training algorithms (back-propagation) is generally very slow. Local minimum, improper learning rate and over-fitting are some of the other issues. Extreme learning machine was proposed as a non-iterative learning algorithm for single-hidden layer feed forward neural network (SLFN) to overcome these issues. The input weight and biases are chosen randomly in ELM which makes the classification system of non- deterministic behavior. In this paper, a new learning algorithm is presented in which the input weights and the hidden layer biases of SLFN are assigned from basis vectors generated by training space. The output weights and biases are decided through simple generalized inverse operation on output matrix of hidden layer. This makes very fast learning speed and better generalization performance in comparison to conventional learning algorithm as well as ELM.

Regularized online sequential learning algorithm for single-hidden layer feedforward neural networks

Online learning algorithms have been preferred in many applications due to their ability to learn by the sequentially arriving data. One of the effective algorithms recently proposed for training single hidden-layer feedforward neural networks (SLFNs) is online sequential extreme learning machine (OS-ELM), which can learn data one-by-one or chunk-by-chunk at fixed or varying sizes. It is based on the ideas of extreme learning machine (ELM), in which the input weights and hidden layer biases are randomly chosen and then the output weights are determined by the pseudo-inverse operation. The learning speed of this algorithm is extremely high. However, it is not good to yield generalization models for noisy data and is difficult to initialize parameters in order to avoid singular and ill-posed problems. In this paper, we propose an improvement of OS-ELM based on the bi-objective optimization approach. It tries to minimize the empirical error and obtain small norm of network weight vector. Singular and ill-posed problems can be overcome by using the Tikhonov regularization. This approach is also able to learn data one-by-one or chunk-by-chunk. Experimental results show the better generalization performance of the proposed approach on benchmark datasets.

An Imperialist Competitive Algorithm Artificial Neural Network Method to Predict Oil Flow Rate of the Wells

rates of oil, gas and water are most important parameters of oil production that is detected by Multiphase Flow Meters (MFM). Conventional MFM collects data on long-term, because of the radioactive source is used for detection and in unmanned location used due to being away from wells. In this work, a new method based on feed-forward artificial neural network (ANN) and Imperialist Competitive Algorithm (ICA) have been proposed to predict of oil flow rate of the wells. The proposed algorithm combines the local searching ability of the gradient- based back-propagation (BP) strategy with the global searching ability of imperialist competitive algorithm. Imperialist Competitive Algorithm is used to decide the initial weights of the neural network. The ICA-ANN is applied to predict oil flow rate of the wells utilizing data set of 31 wells in one of the northern Persian Gulf oil fields of Iran. The performance of the ICA-ANN is compared with ANN and the results demonstrate the effectiveness of the ICA-ANN.

Content Based Image Retrieval using Novel Gaussian Fuzzy Feed Forward-Neural Network

Problem statement: With extensive digitization of images, diagrams and paintings, traditional keyword based search has been found to be inefficient for retrieval of the required data. Content-Based Image Retrieval (CBIR) system responds to image queries as input and relies on image content, using techniques from computer vision and image processing to interpret and understand it, while using techniques from information retrieval and databases to rapidly locate and retrieve images suiting an input query. CBIR finds extensive applications in the field of medicine as it assists a doctor to make better decisions by referring the CBIR system and gain confidence. Approach: Various methods have been proposed for CBIR using image low level image features like histogram, color layout, texture and analysis of the image in the frequency domain. Similarly various classification algorithms like Naïve Bayes classifier, Support Vector Machine, Decision tree induction algorithms and Neural Network based classifiers have been studied extensively. We proposed to extract features from an image using Discrete Cosine Transform, extract relevant features using information gain and Gaussian Fuzzy Feed Forward Neural Network algorithm for classification. Results and Conclusion: We apply our proposed procedure to 180 brain MRI images of which 72 images were used for testing and the remaining for training. The classification accuracy obtained was 95.83% for a three class problem. This research focused on a narrow search, where further investigation is needed to evaluate larger classes.

A study on effectiveness of extreme learning machine

Extreme learning machine (ELM), proposed by Huang et al., has been shown a promising learning algorithm for single-hidden layer feedforward neural networks (SLFNs). Nevertheless, because of the random choice of input weights and biases, the ELM algorithm sometimes makes the hidden layer output matrix H of SLFN not full column rank, which lowers the effectiveness of ELM. This paper discusses the effectiveness of ELM and proposes an improved algorithm called EELM that makes a proper selection of the input weights and bias before calculating the output weights, which ensures the full column rank of H in theory. This improves to some extend the learning rate (testing accuracy, prediction accuracy, learning time) and the robustness property of the networks. The experimental results based on both the benchmark function approximation and real-world problems including classification and regression applications show the good performances of EELM.