Adaptive Learning Rate Algorithm Research Articles

A Comparative Study of MLP Networks Using Backpropagation Algorithms in Electrical Equipment Thermography

Nowadays, infrared thermographic inspection has increasingly been utilized for fault diagnosis tools of electrical equipment, because it maintains non-interrupting operation of power system and ensures early diagnosis of faults. The article discusses the performance comparison of multilayer perceptron (MLP) networks using various backpropagation algorithms for thermal imaging-based condition monitoring of electrical hotspots. The training algorithms are Levenberg–Marquardt, Broyden–Fletcher–Goldfarb–Shanno quasi-Newton, resilient backpropagation, gradient descent with momentum and adaptive learning rate and standard gradient descent training algorithms. The performance of the training algorithms are evaluated in terms of percentage of accuracy, sensitivity, specificity, false-positive and false-negative results. In the beginning, thermal images of electrical equipment are captured using infrared camera. Six statistical intensity features, namely mean, maximum, minimum, median, standard deviation and variance extracted from each hotspot of equipment thermal image are used as the inputs of multilayered perceptron networks to classify the thermal conditions of hotspots into two classes, namely normal and defective. The comparison of MLP networks using training algorithms proves that the MLP network trained with resilient backpropagation algorithm gives the best performance in thermal condition recognition.

An Adaptive Neural Network Learning-based Solution for the Inverse Kinematics of Humanoid Fingers

This paper presents an adaptive neural network learning-based solution for the inverse kinematics of humanoid fingers. For the purpose, we specify an effective finger model by considering the interphalangeal joint coordination inherent in human fingers. In order to find a proper joint combination for any fingertip trajectory, we propose an adaptive learning scheme by using a multi-layered neural network. It is interesting to use an adaptive learning rate algorithm that leads the neural network to get the inverse kinematic solution quickly. The usefulness of the proposed approach is verified by exemplary simulations for the general motion of humanoid fingers.

A Study on Approximation Performances of Improved Bp Neural Networks Based on LM Algorithms

When approximating nonlinear functions, standard BP algorithms and traditional improved BP algorithms have low convergence rate and tend to be stuck in local minimums. In this paper, standard BP algorithm is improved by numerical optimization algorithm. Firstly, the principle of Levenberg-Marquardt algorithm is introduced. Secondly, to test its approximation performance, LMBP neural network is programmed via MATLAB7.0 taking specific nonlinear function as an example. Thirdly, its approximation result is compared with those of standard BP algorithm and adaptive learning rate algorithm. Simulation results indicate that compared with standard BP algorithm and adaptive learning rate algorithm, LMBP algorithm overcomes deficiencies ranging from poor convergence ability, prolonged convergence time, increasing iteration steps to nonconvergence. Thus with its good approximation ability, LMBP algorithm is the most suitable for medium-sized networks.

Diagnosis Method for Analog Circuit Hard fault and Soft Fault

Because the traditional BP neural network slow convergence speed, easily falling in local minimum and the learning process will appear oscillation phenomena. This paper introduces a tolerance analog circuit hard fault and soft fault diagnosis method based on adaptive learning rate and the additional momentum algorithm BP neural network. Firstly, tolerance analog circuit is simulated by OrCAD / Pspice circuit simulation software, accurately extracts fault waveform data by matlab program automatically. Secondly, using the adaptive learning rate and momentum BP algorithm to train neural network, and then applies it to analog circuit hard fault and soft fault diagnosis. With shorter training time, high precision and global convergence effectively reduces the misjudgment, missing, it can improve the accuracy of fault diagnosis and fast. DOI: http://dx.doi.org/10.11591/telkomnika.v11i9.3301

Analog Circuit Fault Diagnosis Based on Particle Swarm Optimization Algorithm and Adaptive Learning Rate Algorithm

BP neural network is widely used as a multilayer feed forward neural network model. The paper puts forward a kind of adaptive learning rate algorithm and particle swarm optimization algorithm hybrid algorithm combining in order to solve the traditional BP algorithm is easy to fall into local extremum problem. So that the particle swarm optimization algorithm and adaptive learning rate algorithm are complementary. The hybrid algorithm has extensive mapping ability of neural networks and particle swarm rapid, global convergence characteristics. The simulation shows that the hybrid algorithm realizes the detection and location of analog circuit fault avoidance, has satisfied effect.

Tolerance Analog Circuit Hard Fault and Soft Fault Diagnosis Based on Particle Swarm Neural Network

This paper presents a tolerance analog circuit hard fault and soft fault diagnosis method based on the BP neural network and particle swarm optimization algorithm. First, select the mean square error function of BP neural network as the fitness function of the PSO algorithm. Second, change the guidance of neural network algorithms rely on gradient information to adjust the network weights and threshold methods, through the use of the characteristics of the particle swarm algorithm groups parallel search to find more appropriate network weights and threshold. Then using the adaptive learning rate and momentum BP algorithm to train the BP neural network. Finally, the network is applied to fault diagnosis of analog circuit, can quickly and effectively to the circuit fault diagnosis.

A MATLAB-Based Study on Approximation Performances of Improved Algorithms of Typical BP Neural Networks

BP neural networks are widely used and the algorithms are various. This paper studies the advantages and disadvantages of improved algorithms of five typical BP networks, based on artificial neural network theories. First, the learning processes of improved algorithms of the five typical BP networks are elaborated on mathematically. Then a specific network is designed on the platform of MATLAB 7.0 to conduct approximation test for a given nonlinear function. At last, a comparison is made between the training speeds and memory consumption of the five BP networks. The simulation results indicate that for small scaled and medium scaled networks, LM optimization algorithm has the best approximation ability, followed by Quasi-Newton algorithm, conjugate gradient method, resilient BP algorithm, adaptive learning rate algorithm. Keywords: BP neural network; Improved algorithm; Function approximation; MATLAB

Modeling and assessment of VWNN for signal processing of structural systems

This study aimed to develop a model to accurately predict the acceleration of structural systems during an earthquake. The acceleration and applied force of a structure were measured at current time step and the velocity and displacement were estimated through linear integration. These data were used as input to predict the structural acceleration at next time step. The computation tool used was the Volterra/Wiener neural network (VWNN) which contained the mathematical model to predict the acceleration. For alleviating problems of relatively large-dimensional and nonlinear systems, the VWNN model was utilized as the signal processing tool, including the Taylor series components in the input nodes of the neural network. The number of the intermediate layer nodes in the neural network model, containing the training and simulation stage, was evaluated and optimized. Discussions on the influences of the gradient descent with adaptive learning rate algorithm and the Levenberg-Marquardt algorithm, both for determining the network weights, on prediction errors were provided. During the simulation stage, different earthquake excitations were tested with the optimized settings acquired from the training stage to find out which of the algorithms would result in the smallest error, to determine a proper simulation model.

Application of neural network prediction model to full-scale anaerobic sludge digestion

BACKGROUND: Process modeling is a useful tool for description and prediction of the performance of anaerobic digestion systems under varying operation conditions. The objective of this study was to implement a model to simulate the dynamic behavior of a large-scale anaerobic sewage sludge digestion system. Artificial neural network (ANN) models using algorithms best suited to environmental problems (the Levenberg–Marquardt algorithm and the ‘gradient descent with adaptive learning rate’ back propagation algorithms) were used to model the anaerobic sludge digester of the Ankara Central Wastewater Treatment Plant (ACWTP) using dynamic data. RESULTS: Based on the relatively low mean square error (MSE), mean absolute error (MAE) and mean absolute percentage error (MAPE) and very high r values, ANN models predicted effluent volatile solid (VS) concentration and methane yield satisfactorily. Effluent VS and methane yields were predicted by the ANN using only conventional parameters such as pH, temperature, flow rate, volatile fatty acids, alkalinity, dry matter and organic matter. The best back propagation algorithm was the gradient descent with adaptive learning rate algorithm in both models. In the training of the neural network, four-fold cross-validation was used for validation of the model for better reliability. CONCLUSION: The proposed ANN models were shown to be capable of dynamically predicting the VS and CH4 production rates for real system behavior. Only relatively simple monitoring parameters are needed to build the model for this complex anaerobic digestion process. Copyright © 2011 Society of Chemical Industry

Novel Maximum-Margin Training Algorithms for Supervised Neural Networks

This paper proposes three novel training methods, two of them based on the backpropagation approach and a third one based on information theory for multilayer perceptron (MLP) binary classifiers. Both backpropagation methods are based on the maximal-margin (MM) principle. The first one, based on the gradient descent with adaptive learning rate algorithm (GDX) and named maximum-margin GDX (MMGDX), directly increases the margin of the MLP output-layer hyperplane. The proposed method jointly optimizes both MLP layers in a single process, backpropagating the gradient of an MM-based objective function, through the output and hidden layers, in order to create a hidden-layer space that enables a higher margin for the output-layer hyperplane, avoiding the testing of many arbitrary kernels, as occurs in case of support vector machine (SVM) training. The proposed MM-based objective function aims to stretch out the margin to its limit. An objective function based on Lp-norm is also proposed in order to take into account the idea of support vectors, however, overcoming the complexity involved in solving a constrained optimization problem, usually in SVM training. In fact, all the training methods proposed in this paper have time and space complexities O(N) while usual SVM training methods have time complexity O(N (3)) and space complexity O(N (2)) , where N is the training-data-set size. The second approach, named minimization of interclass interference (MICI), has an objective function inspired on the Fisher discriminant analysis. Such algorithm aims to create an MLP hidden output where the patterns have a desirable statistical distribution. In both training methods, the maximum area under ROC curve (AUC) is applied as stop criterion. The third approach offers a robust training framework able to take the best of each proposed training method. The main idea is to compose a neural model by using neurons extracted from three other neural networks, each one previously trained by MICI, MMGDX, and Levenberg-Marquard (LM), respectively. The resulting neural network was named assembled neural network (ASNN). Benchmark data sets of real-world problems have been used in experiments that enable a comparison with other state-of-the-art classifiers. The results provide evidence of the effectiveness of our methods regarding accuracy, AUC, and balanced error rate.

The prediction of the building precision in the Laser Engineered Net Shaping process using advanced networks

Laser Engineered Net Shaping (LENS) is an advanced manufacturing technology, but it is difficult to control the depositing height (DH) of the prototype because there are many technology parameters influencing the forming process. The effect of main parameters (laser power, scanning speed and powder feeding rate) on the DH of single track is firstly analyzed, and then it shows that there is the complex nonlinear intrinsic relationship between them. In order to predict the DH, the back propagation (BP) based network improved with Adaptive learning rate and Momentum coefficient (AM) algorithm, and the least square support vector machine (LS-SVM) network are both adopted. The mapping relationship between above parameters and the DH is constructed according to training samples collected by LENS experiments, and then their generalization ability, function-approximating ability and real-time are contrastively investigated. The results show that although the predicted result by the BP-AM approximates the experimental result, above performance index of the LS-SVM are better than those of the BP-AM. Finally, high-definition thin-walled parts of AISI316L are successfully fabricated. Hence, the LS-SVM network is more suitable for the prediction of the DH.

Reference evapotranspiration forecasting using different artificial neural networks algorithms

The present study aims to apply artificial neural networks (ANNs) for reference evapotranspiration (ETo) prediction. Three different feed-forward artifical neural network (ANN) models, each using varied input combinations of previous months ETo, have been trained and tested. The output of the network was the one-month-ahead ETo. The networks learned to forecast one-month-ahead ETo for Mahanadi reservoir project area using the three learning methods namely quasi-Newton algorithm, Levenberg–Marquardt algorithm and backpropagation with adaptive learning rate algorithm. The training results were compared with each other, and performance evaluations were done for untrained data. The performance evaluations measured were standard error of estimates (SEE), raw standard error of estimates (RSEE), and model efficiency. The best ANN architecture for prediction of ETo was obtained for Mahanadi reservoir project area. The monthly reference evapotranspiration data were estimated by the Penman–Monteith method and used for training and testing of the ANN models. Further ANNs predicted results were compared with those obtained using the statistical multiple regression technique. Based on results obtained, the ANN model with architecture of 3–9-1 (three, nine, and one neuron(s) in the input, hidden, and output layers, respectively) trained using quasi-Newton algorithm was found to be the best amongst all the models with minimum SEE and RSEE of 0.45 and 0.45 mm/d respectively and maximum model efficiency of 93%. It is concluded that ANN can be used to predict ETo.

Estimation of Medicine Amount Used Anesthesia by an Artificial Neural Network

In this study, Elman's recurrent neural networks using Resilient Back Propagation (RP) algorithm and feed-forward neural networks using adaptive learning rate algorithm (gdx) have been compared in order to determine the depth of anesthesia in the continuation stage of anesthesia and to estimate the amount of medicine to be applied at that moment. EEG data have been recorded by being sampled once in every 2ms. From 30 patients, 57 distinct EEG recordings have been collected prior to during anaesthesia of different levels. The applied artificial neural network is composed of three layers, namely the input layer, the middle layer and the output layer. The nonlinear activation function sigmoid (sigmoid function) has been used in the hidden layer and the output layer. Prediction has been made by means of ANN. Training and testing the ANN have been used previous anaesthesia amount, total power/normal power and total power/previous. When Elman Resilient BP and feed-forward network are compared, it is observed that resilient back propagation algorithm has generated values which are quite close to the applied anesthesia amount compared to gdx which is an adaptive learning algorithm. The system has been able to correctly purposeful responses in average accuracy of 95% of the cases. This method is also computationally fast and acceptable real-time clinical performance has been obtained.

A study on quantitative classification of binary gas mixture using neural networks and adaptive neuro-fuzzy inference systems

In this study, the feed forward neural networks (FFNNs) were applied and an adaptive neuro-fuzzy inference system (ANFIS) was proposed for quantitative identification of individual gas concentrations (trichloroethylene and acetone) in their gas mixtures. The quartz crystal microbalance (QCM) type sensors were used as gas sensors. The components in the binary mixture were quantified by applying the steady state sensor responses from the QCM sensor array as inputs to the FFNNs and ANFISs. The back propagation (BP) with momentum and adaptive learning rate algorithm, resilient BP (RP) algorithm, Fletcher–Reeves conjugate-gradient (CG) algorithm, Broyden, Fletcher, Goldfarb, and Shanno quasi-Newton (QN) algorithm, and Levenberg–Marquardt (LM) algorithm were performed as the training methods of the FFNNs. A hybrid training method, which was the combination of least-squares and back propagation algorithms, was used as the training method of the ANFISs. Quantitative analysis of trichloroethylene and acetone was evaluated in terms of training algorithms and methods.

Autonomous learning algorithm for fully connected recurrent networks

In this paper, fully connected RTRL neural networks are studied. In order to learn dynamical behaviours of continuous time processes or to predict numerical time series, an autonomous learning algorithm has been developed. The originality of this method consists in the gradient-based adaptation of the learning rate and time parameter of neurons using a small perturbations method. Starting from zero initial conditions (neural states, rate of learning, time parameter and matrix of weights) the evolution is completely driven by the dynamic of the learning data. Stability issues are discussed, and several examples are investigated in order to compare the performances of the adaptive learning rate and time parameter algorithm with the constant parameters one.

Restoring images using a feed-forward neural network and the gradient adaptive learning rate algorithm

The problem of restoring high-quality images from degraded imaging systems without any prior knowledge of the degradation model and statistics of noise is considered. A novel training algorithm, the gradient adaptive learning rate, is employed to train a feedforward neural network, the nonlinear restoration model. A linearization model of the neuron's sigmoidal activation function is utilized to speed up the convergence of the algorithm. Restoration is accomplished by making use of the generalization capabilities of the network. Ideal data sets representing flat and edge regions that would be occurring in real images have been proposed to train the network while real images have been used only during operation of the system. Computer simulation examples are given to illustrate the significance of this method. Comparison with one of the conventional restoration methods is also presented. Simulation results indicate better performance of the proposed method to other competing methods.

Restoring images using a feed-forward neural network and the gradient adaptive learning rate algorithm

Restoring images using a feed-forward neural network and the gradient adaptive learning rate algorithm

Adaptive algorithm for training pRAM neural networkson unbalanced data sets

A novel algorithm for training pyramidal pRAM neural networks on an unbalanced training set is proposed. The behaviour of the standard reinforcement learning algorithm is analysed and an adaptive learning rate algorithm that modifies the reinforcement learning algorithm based on readily available a priori class probability is developed.

Multivariable Neural Network Based Controllers for Smart Structures

This paper details identification and robust control of smart structures using artificial neural networks. To demonstrate the use of artificial neural networks in the control of smart structural systems, two smart structure test articles were fabricated. Active materials like piezoelectric (PZT), polyvinylidene (PVDF) and shape memory alloys (SMA) were used as actuators and sensors. The Eigensystem Realization Algorithm (ERA), a structural identification method has been utilized to determine a minimal order discrete time state space model of the test articles. The ERA requires the Markov parameters of the physical system. A neural network based method has been developed to estimate the Markov parameters of a multi input multi output system from experimental test data. The accelerated adaptive learning rate algorithm and the adaptive activation function were utilized to improve the learning characteristics of the network and reduce the learning time. The identified models were used to design a robust controllers for vibration suppression of smart structures using a modified Linear Quadratic Gaussian with Loop Transfer Recovery (LQG/LTR) method. This control design methodology has better loop transfer recovery properties while accommodating the limited control force available from the SMA and the PZT actuators. This controller was copied into a feedforward neural network using the connectionist approach. This neural network controller was implemented using a PC based data acquisition system. The closed loop performance and robustness properties of the conventional and the neural network based controller are compared experimentally.