Slow Learning Speed Research Articles

One-Class Classification with Extreme Learning Machine

One-class classification problem has been investigated thoroughly for past decades. Among one of the most effective neural network approaches for one-class classification, autoencoder has been successfully applied for many applications. However, this classifier relies on traditional learning algorithms such as backpropagation to train the network, which is quite time-consuming. To tackle the slow learning speed in autoencoder neural network, we propose a simple and efficient one-class classifier based on extreme learning machine (ELM). The essence of ELM is that the hidden layer need not be tuned and the output weights can be analytically determined, which leads to much faster learning speed. The experimental evaluation conducted on several real-world benchmarks shows that the ELM based one-class classifier can learn hundreds of times faster than autoencoder and it is competitive over a variety of one-class classification methods.

Natural gradient learning algorithms for RBF networks.

Radial basis function (RBF) networks are one of the most widely used models for function approximation and classification. There are many strange behaviors in the learning process of RBF networks, such as slow learning speed and the existence of the plateaus. The natural gradient learning method can overcome these disadvantages effectively. It can accelerate the dynamics of learning and avoid plateaus. In this letter, we assume that the probability density function (pdf) of the input and the activation function are gaussian. First, we introduce natural gradient learning to the RBF networks and give the explicit forms of the Fisher information matrix and its inverse. Second, since it is difficult to calculate the Fisher information matrix and its inverse when the numbers of the hidden units and the dimensions of the input are large, we introduce the adaptive method to the natural gradient learning algorithms. Finally, we give an explicit form of the adaptive natural gradient learning algorithm and compare it to the conventional gradient descent method. Simulations show that the proposed adaptive natural gradient method, which can avoid the plateaus effectively, has a good performance when RBF networks are used for nonlinear functions approximation.

Structural damage assessment using linear approximation with maximum entropy and transmissibility data

Supervised learning algorithms have been proposed as a suitable alternative to model updating methods in structural damage assessment, being Artificial Neural Networks the most frequently used. Notwithstanding, the slow learning speed and the large number of parameters that need to be tuned within the training stage have been a major bottleneck in their application. This article presents a new algorithm for real-time damage assessment that uses a linear approximation method in conjunction with antiresonant frequencies that are identified from transmissibility functions. The linear approximation is handled by a statistical inference model based on the maximum-entropy principle. The merits of this new approach are twofold: training is avoided and data is processed in a period of time that is comparable to the one of Neural Networks. The performance of the proposed methodology is validated by considering three experimental structures: an eight-degree-of-freedom (DOF) mass-spring system, a beam, and an exhaust system of a car. To demonstrate the potential of the proposed algorithm over existing ones, the obtained results are compared with those of a model updating method based on parallel genetic algorithms and a multilayer feedforward neural network approach.

A Novel Improved ELM Algorithm for a Real Industrial Application

It is well known that the feedforward neural networks meet numbers of difficulties in the applications because of its slow learning speed. The extreme learning machine (ELM) is a new single hidden layer feedforward neural network method aiming at improving the training speed. Nowadays ELM algorithm has received wide application with its good generalization performance under fast learning speed. However, there are still several problems needed to be solved in ELM. In this paper, a new improved ELM algorithm named R-ELM is proposed to handle the multicollinear problem appearing in calculation of the ELM algorithm. The proposed algorithm is employed in bearing fault detection using stator current monitoring. Simulative results show that R-ELM algorithm has better stability and generalization performance compared with the original ELM and the other neural network methods.

A New SVM Reduction Strategy of Large-Scale Training Sample Sets

There has become a bottleneck to use support vector machine (SVM) due to the problems such as slow learning speed, large buffer memory requirement, low generalization performance and so on. These problems are caused by large-scale training sample set and outlier data immixed in the other class. Aiming at these problems, this paper proposed a new reduction strategy for large-scale training sample set according to analyzing on the structure of the training sample set based on the point set theory. By using fuzzy clustering method in this new strategy, the potential support vectors are obtained and the non-boundary outlier data immixed in the other class is removed. In view of reducing greatly the scale of the training sample set, it improves the generalization performance of SVM and effectively avoids over-learning. Finally, the experimental results shown the given reduction strategy can not only reduce the train samples of SVM and speed up the train process, but also ensure accuracy of classification.

The SVM Based on Reduction Strategy Applied in Bus Passenger Flow Counting

There has become a bottleneck to use support vector machine (SVM) due to the problems such as slow learning speed, large buffer memory requirement, low generalization performance and so on. These problems are caused by large-scale training sample set and outlier data immixed in the other class. Aiming at these problems, this paper proposed a new reduction strategy for large-scale training sample set according to analyzing on the structure of the training sample set based on the point set theory. By using fuzzy clustering method in this new strategy, the potential support vectors are obtained and the non-boundary outlier data immixed in the other class is removed. In view of reducing greatly the scale of the training sample set, it improves the generalization performance of SVM and effectively avoids over-learning. Then the proposed method was applied to bus passenger flow counting. The experimental results show that the method reposed in this paper obtains higher classification accuracy.

Research on Improved BP Learning Algorithm of BP Neural Network

Aiming at the existence of the BP neural network learning algorithm in the slow learning speed, the possibility of failure is large, poor generalization ability, there are multiple issues, extreme value point and network structure are difficult to determine, in this paper, we study algorithm improvement methods. Explain the algorithm principle, on the basis of three improved methods are studied, respectively is dynamic learning rate, conjugate gradient, improved error function. Among them, the dynamic learning rate, it reaches the learning rate of the hidden layer and output layer; Conjugate gradient, this paper gives three calculating formula; Improved error function, to solve different problems are also given in three types of error function. BP learning algorithm in this paper, the research contents, make the convergence stability, convergence speed, initial value sensitivity, it has good effect, which has large significant in terms of academic and applied significance.

Study Speech Recognition System Based on Manifold Learning

This paper conducts a comprehensive research and discussion on the relevant technologies and manifold learning.Traditional MFCC phonetic feature will lead a slower learning speed on account of it has high dimension and is large in data quantities. In order to solve this problem, we introduce a manifold learning, putting forward two new extraction methods of MFCC-Manifold phonetic feature. We can reduce dimensions by making use of ISOMAP algorithm which bases on the classical MDS (Multidimensional scaling). Introducing geodesic distance to replace the original European distance data will make twenty-four dimensional data, which using the traditional MFCC feature extraction down to ten dimensional data.

An Extreme Learning Machine Algorithm to Predict the In-flight Particle Characteristics of an Atmospheric Plasma Spray Process

A robust single hidden layer feed forward neural network (SLFN) is used in this study to model the in-flight particle characteristics of the atmospheric plasma spray (APS) process with regard to the input processing parameters. The in-flight particle characteristics influence the structure and properties of the APS coating and, thus, are considered important parameters to comprehend the manufacturing process. The training times of traditional back propagation algorithms, mostly used to model such processes, are far slower than desired for implementation of an on-line control system. Use of slow gradient based learning methods and iterative tuning of all network parameters during the learning process are the two major causes for such slower learning speed. An extreme learning machine (ELM) algorithm, which randomly selects the input weights and biases and analytically determines the output weights, is used in this work to train the SLFNs. Performance comparisons of the networks trained with ELM algorithm and standard error back propagation algorithms are presented. It is found that networks trained with ELM have good generalization performance, much shorter training times and stable performance with regard to the changes in number of hidden layer neurons. The trends represent robustness of the trained networks and enhance reliability of the application of the artificial neural network in modelling APS processes.

Support Vector Regression Based Nonlinear Model Reference Adaptive Control

Model reference adaptive control (MRAC) is widely used in linear system control areas, and Neural Networks (NN) is often used to extend MRAC to nonlinear areas. However, this kind of solution inherits some drawbacks of NN, including slow learning speed, weak generalization ability, local minima tendency, etc. Given these drawbacks, this paper attempts to use support vector regression (SVR) as a substitute of NN. In this approach, SVR is employed to compensate the nonlinear part of the plant. A stable controller-parameter adjustment mechanism is constructed by using the practical stability theory. Simulation results show that the proposed approach could reach desired performance.

A New SVM Reduction Strategy of Large-Scale Training Sample Sets

There has become a bottleneck to use support vector machine (SVM) due to the problems such as slow learning speed, large buffer memory requirement, low generalization performance and so on. These problems are caused by large-scale training sample set and outlier data immixed in the other class. Aiming at these problems, this paper proposed a new reduction strategy for large-scale training sample set according to analyzing on the structure of the training sample set based on the point set theory. By using fuzzy clustering method in this new strategy, the potential support vectors are obtained and the non-boundary outlier data immixed in the other class is removed. In view of reducing greatly the scale of the training sample set, it improves the generalization performance of SVM and effectively avoids over-learning. Finally, the experimental results shown the given reduction strategy can not only reduce the train samples of SVM and speed up the train process, but also ensure accuracy of classification.

The Method and Application of Time Series Prediction Based Wavelet Neural Network

This paper presents a research on modeling and prediction with wavelet neural network in the nonlinear time series of gas emitted. Because accurately predicting the amount of gas emitted from the mine is a very important matter for safety, this paper proposes a new algorithm of wavelet neural network model for time series gas emission prediction. The nervous cells function is the basis of nonlinear wavelets. A wavelet network composed by the wavelet basis function is computed by an expansion and contraction factor and a translation factor to reach the global best approximation effect. Which wavelet basis function has the features of extraction capabilities, self-learning neural network and wavelet transform of the localized nature. The intrinsic defects of artificial BP neural network, e.g., its slow learning speed, difficulty to determine rationally the network structure and existence of partial minimum points, are solved. The simulation results obtained show that the new prediction system has faster convergence and more accurate prediction.

Extreme learning machines: a survey

Computational intelligence techniques have been used in wide applications. Out of numerous computational intelligence techniques, neural networks and support vector machines (SVMs) have been playing the dominant roles. However, it is known that both neural networks and SVMs face some challenging issues such as: (1) slow learning speed, (2) trivial human intervene, and/or (3) poor computational scalability. Extreme learning machine (ELM) as emergent technology which overcomes some challenges faced by other techniques has recently attracted the attention from more and more researchers. ELM works for generalized single-hidden layer feedforward networks (SLFNs). The essence of ELM is that the hidden layer of SLFNs need not be tuned. Compared with those traditional computational intelligence techniques, ELM provides better generalization performance at a much faster learning speed and with least human intervene. This paper gives a survey on ELM and its variants, especially on (1) batch learning mode of ELM, (2) fully complex ELM, (3) online sequential ELM, (4) incremental ELM, and (5) ensemble of ELM.

Improving rival penalized competitive learning using density‐evaluated mechanism

Rival penalized competitive learning (RPCL) and its variants have provided attractive ways to perform clustering without knowing the exact cluster number. However, they are always accompanied by problems of falling in local optima and slow learning speed. Thus we investigate the RPCL and propose a mechanism to directly prune the RPCL's structure by evaluating the data density of each unit. We call the new strategy density‐evaluated RPCL (DERPCL). The communication channel state is estimated by the DERPCL in the simulations, and comprehensive comparisons are made with other RPCLs. Results show that the DERPCL is superior in terms of convergence accuracy and speed.

Robust algorithm for arrhythmia classification in ECG using extreme learning machine

BackgroundRecently, extensive studies have been carried out on arrhythmia classification algorithms using artificial intelligence pattern recognition methods such as neural network. To improve practicality, many studies have focused on learning speed and the accuracy of neural networks. However, algorithms based on neural networks still have some problems concerning practical application, such as slow learning speeds and unstable performance caused by local minima.MethodsIn this paper we propose a novel arrhythmia classification algorithm which has a fast learning speed and high accuracy, and uses Morphology Filtering, Principal Component Analysis and Extreme Learning Machine (ELM). The proposed algorithm can classify six beat types: normal beat, left bundle branch block, right bundle branch block, premature ventricular contraction, atrial premature beat, and paced beat.ResultsThe experimental results of the entire MIT-BIH arrhythmia database demonstrate that the performances of the proposed algorithm are 98.00% in terms of average sensitivity, 97.95% in terms of average specificity, and 98.72% in terms of average accuracy. These accuracy levels are higher than or comparable with those of existing methods. We make a comparative study of algorithm using an ELM, back propagation neural network (BPNN), radial basis function network (RBFN), or support vector machine (SVM). Concerning the aspect of learning time, the proposed algorithm using ELM is about 290, 70, and 3 times faster than an algorithm using a BPNN, RBFN, and SVM, respectively.ConclusionThe proposed algorithm shows effective accuracy performance with a short learning time. In addition we ascertained the robustness of the proposed algorithm by evaluating the entire MIT-BIH arrhythmia database.

Automatic thesaurus construction for spam filtering using revised back propagation neural network

Email has become one of the fastest and most economical forms of communication. Email is also one of the most ubiquitous and pervasive applications used on a daily basis by millions of people worldwide. However, the increase in email users has resulted in a dramatic increase in spam emails during the past few years. This paper proposes a new spam filtering system using revised back propagation (RBP) neural network and automatic thesaurus construction. The conventional back propagation (BP) neural network has slow learning speed and is prone to trap into a local minimum, so it will lead to poor performance and efficiency. The authors present in this paper the RBP neural network to overcome the limitations of the conventional BP neural network. A well constructed thesaurus has been recognized as a valuable tool in the effective operation of text classification, it can also overcome the problems in keyword-based spam filters which ignore the relationship between words. The authors conduct the experiments on Ling-Spam corpus. Experimental results show that the proposed spam filtering system is able to achieve higher performance, especially for the combination of RBP neural network and automatic thesaurus construction.

Combining neural networks and semantic feature space for email classification

Email is one of the most ubiquitous and pervasive applications used on a daily basis by millions of people worldwide, individuals and organizations more and more rely on the emails to communicate and share information and knowledge. However, the increase in email users has resulted in a dramatic increase in spam emails during the past few years. It is becoming a big challenge to process and manage the emails efficiently for and individuals and organizations. This paper proposes new email classification models using a linear neural network trained by perceptron learning algorithm and a nonlinear neural network trained by back-propagation learning algorithm. An efficient semantic feature space (SFS) method is introduced in these classification models. The traditional back-propagation neural network (BPNN) has slow learning speed and is prone to trap into a local minimum, so the modified back-propagation neural network (MBPNN) is presented to overcome these limitations. The vector space model based email classification system suffers from a large number of features and ambiguity in the meaning of terms, which will lead to sparse and noisy feature space. So we use the SFS to convert the original sparse and noisy feature space to a semantically richer feature space, which will helps to accelerate the learning speed. The experiments are conducted based on different training set size and extracted feature size. Experimental results show that the models using MBPNN outperform the traditional BPNN, and the use of SFS can greatly reduce the feature dimensionality and improve email classification performance.

Part 2: Multilayer perceptron and natural gradient learning

Since the perceptron was developed for learning to classify input patterns, there have been plenty of studies on simple perceptrons and multilayer perceptrons. Despite wide and active studies in theory and applications, multilayer perceptrons still have many unsettled problems such as slow learning speed and overfitting. To find a thorough solution to these problems, it is necessary to consolidate previous studies, and find new directions for uplifting the practical power of multilayer perceptrons. As a first step toward the new stage of studies on multilayer perceptrons, we give short reviews on two interesting and important approaches; one is stochastic approach and the other is geometric approach. We also explain an efficient learning algorithm developed from the statistical and geometrical studies, which is now well known as the natural gradient learning method.

Implementing Online Natural Gradient Learning: Problems and Solutions

The online natural gradient learning is an efficient algorithm to resolve the slow learning speed and poor performance of the standard gradient descent method. However, there are several problems to implement this algorithm. In this paper, we proposed a new algorithm to solve these problems and then compared the new algorithm with other known algorithms for online learning, including Almeida-Langlois-Amaral-Plakhov algorithm (ALAP), Vario-eta, local adaptive learning rate and learning with momentum etc., using sample data sets from Proben1 and normalized handwritten digits, automatically scanned from envelopes by the U.S. Postal Services. The strong and weak points of these algorithms were analyzed and tested empirically. We found out that using the online training error as the criterion to determine whether the learning rate should be changed or not is not appropriate and our new algorithm has better performance than other existing online algorithms.

Improving generalization performance of natural gradient learning using optimized regularization by NIC.

Natural gradient learning is known to be efficient in escaping plateau, which is a main cause of the slow learning speed of neural networks. The adaptive natural gradient learning method for practical implementation also has been developed, and its advantage in real-world problems has been confirmed. In this letter, we deal with the generalization performances of the natural gradient method. Since natural gradient learning makes parameters fit to training data quickly,the overfitting phenomenon may easily occur, which results in poor generalization performance. To solve the problem, we introduce the regularization term in natural gradient learning and propose an efficient optimizing method for the scale of regularization by using a generalized Akaike information criterion (network information criterion). We discuss the properties of the optimized regularization strength by NIC through theoretical analysis as well as computer simulations. We confirm the computational efficiency and generalization performance of the proposed method in real-world applications through computational experiments on benchmark problems.