Gradient Descent Learning Research Articles

An efficient FLANN model with CRO-based gradient descent learning for classification

Due to the nonlinear nature of real world data, it is difficult to determine the optimal ANN classification model with accurate and fast convergence. Although, many higher order ANN have been designed and integrated with competitive optimisation method in order to construct an accurate classification model, but the parameter adjustment and variability in performance in different runs of the classification model leads to statistically insignificant result. In this paper, a FLANN model (CRO-GDL-FLANN) has been proposed for classification with gradient descent learning (GDL) based on chemical reaction optimisation (CRO). The proposed CRO-GDL-FLANN method has been tested with various benchmark datasets from the UCI machine learning repository under five fold cross-validations. The classification accuracy of CRO-GDL-FLANN is compared with FLANN, GA-FLANN and PSO-FLANN. To prove the proposed method is statistically better and significantly different from other alternatives, the CRO-GDL-FLANN is verified under multiple comparisons of classifiers by using Friedman, Tukey and Dunnett statistical test. Finally, one-way-ANOVA test has been carried out for generalised comparison of CRO-GDL-FLANN with other classifiers.

A self adaptive harmony search based functional link higher order ANN for non-linear data classification

In the data classification process involving higher order ANNs, it’s a herculean task to determine the optimal ANN classification model due to non-linear nature of real world datasets. To add to the woe, it is tedious to adjust the set of weights of ANNs by using appropriate learning algorithm to obtain better classification accuracy. In this paper, an improved variant of harmony search (HS), called self-adaptive harmony search (SAHS) along with gradient descent learning is used with functional link artificial neural network (FLANN) for the task of classification in data mining. Using its past experiences, SAHS adjusts the harmonies according to the maximum and minimum values in the current harmony memory. The powerful combination of this unique strategy of SAHS and searching capabilities of gradient descent search is used to obtain optimal set of weights for FLANN. The proposed method (SAHS–FLANN) is implemented in MATLAB and the results are contrasted with other alternatives (FLANN, GA based FLANN, PSO based FLANN, HS based FLANN, improved HS based FLANN and TLBO based FLANN). To illustrate its effectiveness, SAHS–FLANN is tested on various benchmark datasets from UCI machine learning repository by using 5-fold cross validation technique. Under the null-hypothesis, the proposed method is analyzed by using various statistical tests for statistical correctness of results. The performance of the SAHS–FLANN is found to be better and statistically significant in comparison with other alternatives. The SAHS–FLANN differs from HS–FLANN (HS based FLANN) by the elimination of constant parameters (bandwidth and pitch adjustment rate). Furthermore, it leads to the simplification of steps for the improvisation of weight-sets in IHS–FLANN (improved HS based FLANN) by incorporating adjustments of new weight-sets according to the weight-sets with maximum and minimum fitness.

Brain Dynamics in Predicting Driving Fatigue Using a Recurrent Self-Evolving Fuzzy Neural Network.

This paper proposes a generalized prediction system called a recurrent self-evolving fuzzy neural network (RSEFNN) that employs an on-line gradient descent learning rule to address the electroencephalography (EEG) regression problem in brain dynamics for driving fatigue. The cognitive states of drivers significantly affect driving safety; in particular, fatigue driving, or drowsy driving, endangers both the individual and the public. For this reason, the development of brain-computer interfaces (BCIs) that can identify drowsy driving states is a crucial and urgent topic of study. Many EEG-based BCIs have been developed as artificial auxiliary systems for use in various practical applications because of the benefits of measuring EEG signals. In the literature, the efficacy of EEG-based BCIs in recognition tasks has been limited by low resolutions. The system proposed in this paper represents the first attempt to use the recurrent fuzzy neural network (RFNN) architecture to increase adaptability in realistic EEG applications to overcome this bottleneck. This paper further analyzes brain dynamics in a simulated car driving task in a virtual-reality environment. The proposed RSEFNN model is evaluated using the generalized cross-subject approach, and the results indicate that the RSEFNN is superior to competing models regardless of the use of recurrent or nonrecurrent structures.

PID Neural Network Decoupling Control Based on Hybrid Particle Swarm Optimization and Differential Evolution

For complex systems with high nonlinearity and strong coupling, the decoupling control technology based on proportion integration differentiation (PID) neural network (PIDNN) is used to eliminate the coupling between loops. The connection weights of the PIDNN are easy to fall into local optimum due to the use of the gradient descent learning method. In order to solve this problem, a hybrid particle swarm optimization (PSO) and differential evolution (DE) algorithm (PSO-DE) is proposed for optimizing the connection weights of the PIDNN. The DE algorithm is employed as an acceleration operation to help the swarm to get out of local optima traps in case that the optimal result has not been improved after several iterations. Two multivariable controlled plants with strong coupling between input and output pairs are employed to demonstrate the effectiveness of the proposed method. Simulation results show that the proposed method has better decoupling capabilities and control quality than the previous approaches.

A Global-best Harmony Search based Gradient Descent Learning FLANN (GbHS-GDL-FLANN) for data classification

While dealing with real world data for classification using ANNs, it is often difficult to determine the optimal ANN classification model with fast convergence. Also, it is laborious to adjust the set of weights of ANNs by using appropriate learning algorithm to obtain better classification accuracy. In this paper, a variant of Harmony Search (HS), called Global-best Harmony Search along with Gradient Descent Learning is used with Functional Link Artificial Neural Network (FLANN) for classification task in data mining. The Global-best Harmony Search (GbHS) uses the concepts of Particle Swarm Optimization from Swarm Intelligence to improve the qualities of harmonies. The problem solving strategies of Global-best Harmony Search along with searching capabilities of Gradient Descent Search are used to obtain optimal set of weight for FLANN. The proposed method (GbHS-GDL-FLANN) is implemented in MATLAB and compared with other alternatives (FLANN, GA based FLANN, PSO based FLANN, HS based FLANN, Improved HS based FLANN, Self Adaptive HS based FLANN, MLP, SVM and FSN). The GbHS-GDL-FLANN is tested on benchmark datasets from UCI Machine Learning repository by using 5-fold cross validation technique. The proposed method is analyzed under null-hypothesis by using Friedman Test, Holm and Hochberg Procedure and Post-Hoc ANOVA Statistical Analysis (Tukey Test & Dunnett Test) for statistical analysis and validity of results. Simulation results reveal that the performance of the proposed GbHS-GDL-FLANN is better and statistically significant from other alternatives.

Multisensor Data Fusion Techniques With ELM for Pulverized-Fuel Flow Concentration Measurement in Cofired Power Plant

Due to the phase concentration, measurement of coal/biomass/air three-phase flow is closely associated with the quantity of pollutant discharged and combustion efficiency in the pneumatic conveying system. This paper proposed a multisensor data fusion technique for the online volume concentration measurement of coal/biomass pulverized-fuel flow in cofired power plant. The techniques combine electrostatic sensors with capacitance sensors and incorporates a data fusion technique based on an adaptive wavelet neural network (AWNN), and gradient descent learning algorithm and genetic learning algorithm are used for training of the network parameters. The flow regime is first identified by performing an extreme learning machine on the electrostatic fluctuation signals, thus making the concentration measurement less affected by variations in the flow regimes. Then under the certain identified flow regime, the AWNN data fusion method was applied to determine the phase concentration. An experimental platform was built for phase concentration measurement of pulverized coal (PC)/biomass/air three-phase flow, and the test results confirmed that the technology of the concentration measurement with flow regime identification is better than that of the one without, and the maximum fiducial errors of sawdust and PC are 2.1% and 1.2%.

An adaptive local linear optimized radial basis functional neural network model for financial time series prediction

For financial time series, the generation of error bars on the point of prediction is important in order to estimate the corresponding risk. In recent years, optimization techniques-driven artificial intelligence has been used to make time series approaches more systematic and improve forecasting performance. This paper presents a local linear radial basis functional neural network (LLRBFNN) model for classifying finance data from Yahoo Inc. The LLRBFNN model is learned by using the hybrid technique of backpropagation and recursive least square algorithm. The LLRBFNN model uses a local linear model in between the hidden layer and the output layer in contrast to the weights connected from hidden layer to output layer in typical neural network models. The obtained prediction result is compared with multilayer perceptron and radial basis functional neural network with the parameters being trained by gradient descent learning method. The proposed technique provides a lower mean squared error and thus can be considered as superior to other models. The technique is also tested on linear data, i.e., diabetic data, to confirm the validity of the result obtained from the experiment.

Inverse Kinematic Solution of Robot Manipulator Using Hybrid Neural Network

Inverse kinematics of robot manipulator is to determine the joint variables for a given Cartesian position and orientation of an end effector. There is no unique solution for the inverse kinematics thus necessitating application of appropriate predictive models from the soft computing domain. Although artificial neural network (ANN) can be gainfully used to yield the desired results but the gradient descent learning algorithm does not have ability to search for global optimum and it gives slow convergence rate. This paper proposes structured ANN with hybridization of Gravitational Search Algorithm to solve inverse kinematics of 6R PUMA robot manipulator. The ANN model used is multi-layered perceptron neural network (MLPNN) with back-propagation (BP) algorithm which is compared with hybrid multi layered perceptron gravitational search algorithm (MLPGSA). An attempt has been made to find the best ANN configuration for the problem. It has been observed that MLPGSA gives faster convergence rate and improves the problem of trapping in local minima. It is found that MLPGSA gives better result and minimum error as compared to MLPBP.

A Novel Fuzzy Based Method for Heart Rate Variability Prediction

Abstract In this paper, a novel technique based on fuzzy method is presented for chaotic nonlinear time series prediction. Fuzzy approach with the gradient learning algorithm and methods constitutes the main components of this method. This learning process in this method is similar to conventional gradient descent learning process, except that the input patterns and parameters are stored in memory as a look-up table after upgrade. In the testing phase according to input patterns, the nearest neighbors and the weights corresponding to the test pattern, similar patterns are extracted from memory. Eventually by extracted weights and input pattern, prediction is performed. In order to validate the proposed method for predicting, the Mackey-Glass, Lorenz and biological Heart Rate Variability (HRV) time series is used. Finally, results of proposed method with the conventional methods of time-series prediction are also compared. The results demonstrate the capability of proposed method in chaotic time series prediction.

Projection Rule for Rotor Hopfield Neural Networks

A rotor Hopfield neural network (RHNN) is an extension of a complex-valued Hopfield neural network (CHNN). RHNNs have some excellent properties. For example, the storage capacity of an RHNN is twice that of a CHNN. The most important property of an RHNN is that it does not store rotated patterns of training patterns, unlike CHNNs, which have less noise robustness because they store rotated patterns. However, conventional learning methods for RHNNs, such as Hebbian learning rule and gradient descent learning rules, present difficulties with regard to, for example, storage capacity, noise robustness, and learning time. In this paper, we propose a projection rule for RHNN and demonstrate that the noise robustness of RHNN is better than that of CHNN. The proposed algorithm improves the noise robustness of RHNN. As the number of training patterns increases, the noise robustness of CHNN rapidly deteriorates. On the other hand, the noise robustness of RHNN reduces less rapidly for the same case. Moreover, RHNN can easily recover from rotated patterns, unlike CHNN. We show this ability by computer simulation.

Kernelized vector quantization in gradient-descent learning

Abstract Prototype based vector quantization is usually proceeded in the Euclidean data space. In the last years, also non-standard metrics became popular. For classification by support vector machines, Hilbert space representations, which are based on so-called kernel metrics, seem to be very successful. In this paper we show that gradient based learning in prototype-based vector quantization is possible by means of kernel metrics instead of the standard Euclidean distance. We will show that an appropriate handling requires differentiable universal kernels defining the feature space metric. This allows a prototype adaptation in the original data space but equipped with a metric determined by the kernel and, therefore, it is isomorphic to respective kernel Hilbert space. However, this approach avoids the Hilbert space representation as known for support vector machines. We give the mathematical justification for the isomorphism and demonstrate the abilities and the usefulness of this approach for several examples including both artificial and real world datasets.

Concentration measurement of three-phase flow based on multi-sensor data fusion using adaptive fuzzy inference system

This paper proposes a new method for the volumetric-concentration measurement of coal/biomass/air three-phase flow using multi-sensor data fusion techniques. The method integrates capacitive and electrostatic sensors and incorporates the data fusion model of an adaptive network based fuzzy inference system (ANFIS), which simulates the human׳s understanding of things. The features of the two sensor signals are extracted as the input of the ANFIS under various experimental conditions. The fusion model of the ANFIS establishes the relationship between the volumetric-concentration of the solid phase and the signal features by training with two different learning rules: the gradient descent method only and the hybrid method combining the Kalman filter algorithm with the gradient descent algorithm. Experimental results show that the ANFIS based on the hybrid learning rule outperforms the system based on the gradient descent learning rule and that the fiducial error for biomass and pulverized coal flows are 1.2% and 0.7%, respectively.

OPTIMIZATION OF MAIN BOILER PARAMETERS USING SOFT COMPUTING TECHNIQUES

In most of the large scale industrial applications, the controlling and optimization of the parameters has to be done efficiently so as to attain the smooth operation of the industry. In this paper, the main control parameters of any process industries such as level of feed water, steam temperature and flow of steam were optimized by using ANFIS with PID controller. An ANFIS system is an adaptive network in which each node perform a particular function with respect to the incoming signals, with parameters are updated according to the given training data and a gradient-descent learning procedure. The simulation is done in MATLAB software. By using Neuro-fuzzy controller with PID, the transient response was improved and the settling time was reduced.

A Fuzzy Model With Online Incremental SVM and Margin-Selective Gradient Descent Learning for Classification Problems

This paper proposes a new incremental learning approach to endow a Takagi-Sugeno-type fuzzy classification model with high generalization ability. The proposed fuzzy model is learned through incremental support vector machine (SVM) and margin-selected gradient descent learning and is called FM3. In this learning approach, training samples are fed incrementally one-by-one instead of all in one batch. The FM3 evolves from an empty rule set. A one-pass clustering algorithm is used to determine the number of rules and initial fuzzy sets in the rule antecedent part. An online incremental linear SVM is proposed to tune the rule consequent parameters to endow the FM3 with high generalization ability. The use of incremental instead of batch SVM enables the FM3 to handle online training problems with only one new sample available at a time. For antecedent parameter learning, a margin-selected gradient descent algorithm is proposed to prevent overtraining. Simulation results and comparisons with SVMs and fuzzy classifiers with different learning algorithms demonstrate the advantage of the FM3.

Second order optimization for the inference of gene regulatory pathways

With the increasing availability of experimental data on gene interactions, modeling of gene regulatory pathways has gained special attention. Gradient descent algorithms have been widely used for regression and classification applications. Unfortunately, results obtained after training a model by gradient descent are often highly variable. In this paper, we present a new second order learning rule based on the Newton's method for inferring optimal gene regulatory pathways. Unlike the gradient descent method, the proposed optimization rule is independent of the learning parameter. The flow vectors are estimated based on biomass conservation. A set of constraints is formulated incorporating weighting coefficients. The method calculates the maximal expression of the target gene starting from a given initial gene through these weighting coefficients. Our algorithm has been benchmarked and validated on certain types of functions and on some gene regulatory networks, gathered from literature. The proposed method has been found to perform better than the gradient descent learning. Extensive performance comparison with the extreme pathway analysis method has underlined the effectiveness of our proposed methodology.

Modeling Volume Loss of Heat Treated Al 6061 Composites Using an Artificial Neural Network

In the present study, artificial neural network (ANN) approach was used to predict the volume loss of heat treated Al 6061 metal matrix composites reinforced with 10% SiC particles and 2% graphite particles. Composite was produced using stir casting process. Volume loss of composite was measured during wear testing in a pin on disc apparatus. Microstructure examination at wear surface was investigated by Scanning Electron Microscope (SEM). In Artificial Neural Network (ANN), Multi Layer Perceptron (MLP) architecture with back-propagation neural network that uses gradient descent learning algorithm is utilized. The results clearly revealed that the developed ANN model is reliable and accurate.

Learning Entropy: Multiscale Measure for Incremental Learning

First, this paper recalls a recently introduced method of adaptive monitoring of dynamical systems and presents the most recent extension with a multiscale-enhanced approach. Then, it is shown that this concept of real-time data monitoring establishes a novel non-Shannon and non-probabilistic concept of novelty quantification, i.e., Entropy of Learning, or in short the Learning Entropy. This novel cognitive measure can be used for evaluation of each newly measured sample of data, or even of whole intervals. The Learning Entropy is quantified in respect to the inconsistency of data to the temporary governing law of system behavior that is incrementally learned by adaptive models such as linear or polynomial adaptive filters or neural networks. The paper presents this novel concept on the example of gradient descent learning technique with normalized learning rate.

Compensation of gap sensor for high-speed maglev train with RBF neural network

The gap sensor plays an important role for a electro-magnetic levitation system, which is a critical component of high-speed maglev trains. An artificial neural network is a promising area in the development of intelligent sensors. In this paper, a radial basis function (RBF) neural network modelling approach is introduced for the compensation of the non-contact inductive gap sensor of the high-speed maglev train. As an inverse model compensator, the designed RBF-based model is connected in series to the output terminal of the gap sensor. The network is trained by using a gradient descent learning algorithm with momentum. This scheme could estimate accurately the correct air-gap distance in a wide range of temperatures. The simulation studies of this model show that it can provide a compensated gap value with an error of less than ±0.4 mm at any temperature from 20° to 80°C. In particulr, the maximum estimation error can be reduced to ±0.1 mm when the working gap varies from 8 to 12 mm. The experimental results indicate that the compensated gap signal could meet the requirements of the levitation control system.

Prediction of fretting wear behavior of surface mechanical attrition treated Ti–6Al–4V using artificial neural network

In the present work, multi layer perceptron feed forward artificial neural network (ANN) technique was employed to predict the fretting wear behavior of surface mechanical attrition treated and untreated Ti–6Al–4V fretted against alumina and steel counterbodies. A three-layer neural network with a gradient descent learning algorithm was used to train the network. Three input parameters normal load (L), surface hardness of the test material (H) and hardness of counterbody material (CB) were employed in construction of ANN. Tangential force coefficient (TFC), fretting wear volume and wear rate obtained from a series of fretting wear tests were used in the training and testing data sets of ANN. Ranking of the importance of input parameters on the output TFC was found to be in the order of L>CB>H. For wear volume and wear rate, it was found in the order of L>H>CB. The degrees of accuracy of predictions were 96.6%, 96.1% and 92.2% for TFC, wear volume and wear rate respectively. Owing to the good correlation between the predicted values and the experimental results, ANN can be used in the prediction of fretting wear behavior.

A New Gradient Descent Approach for Local Learning of Fuzzy Neural Models

The majority of reported learning methods for Takagi-Sugeno-Kang (TSK) fuzzy neural models to date mainly focus on improvement of their accuracy. However, one of the key design requirements in building an interpretable fuzzy model is that each obtained rule consequent must match well with the system local behavior when all the rules are aggregated to produce the overall system output. This is one of the distinctive characteristics from black-box models such as neural networks. Therefore, how to find a desirable set of fuzzy partitions and, hence, identify the corresponding consequent models which can be directly explained in terms of system behavior, presents a critical step in fuzzy neural modeling. In this paper, a new learning approach considering both nonlinear parameters in the rule premises and linear parameters in the rule consequents is proposed. Unlike the conventional two-stage optimization procedure widely practiced in the field where the two sets of parameters are optimized separately, the consequent parameters are transformed into a dependent set on the premise parameters, thereby enabling the introduction of a new integrated gradient descent learning approach. Thus, a new Jacobian matrix is proposed and efficiently computed to achieve a more accurate approximation of the cost function by using the second-order Levenberg-Marquardt optimization method. Several other interpretability issues regarding the fuzzy neural model are also discussed and integrated into this new learning approach. Numerical examples are presented to illustrate the resultant structure of the fuzzy neural models and the effectiveness of the proposed new algorithm, and compared with the results from some well-known methods.