Variable Learning Rate Research Articles

A faster dynamic convergency approach for self-organizing maps

This paper proposes a novel variable learning rate to address two main challenges of the conventional Self-Organizing Maps (SOM) termed VLRSOM: high accuracy with fast convergence and low topological error. We empirically showed that the proposed method exhibits faster convergence behavior. It is also more robust in topology preservation as it maintains an optimal topology until the end of the maximum iterations. Since the learning rate adaption and the misadjustment parameter depends on the calculated error, the VLRSOM will avoid the undesired results by exploiting the error response during the weight updation. Then the learning rate is updated adaptively after the random initialization at the beginning of the training process. Experimental results show that it eliminates the tradeoff between the rate of convergence and accuracy and maintains the data's topological relationship. Extensive experiments were conducted on different types of datasets to evaluate the performance of the proposed method. First, we experimented with synthetic data and handwritten digits. For each data set, two experiments with a different number of iterations (200 and 500) were performed to test the stability of the network. The proposed method was further evaluated using four benchmark data sets. These datasets include Balance, Wisconsin Breast, Dermatology, and Ionosphere. In addition, a comprehensive comparative analysis was performed between the proposed method and three other SOM techniques: conventional SOM, parameter-less self-organizing map (PLSOM2), and RA-SOM in terms of accuracy, quantization error (QE), and topology error (TE). The results indicated the proposed approach produced superior results to the other three methods.

Effect of Hidden Neuron Size on Different Training Algorithm in Neural Network

We use different types of training algorithms in the neural network. But, we cannot say which kind of training algorithm is fast for a given problem. So, in this survey paper, we are trying to find which types of training algorithm are better for categorization problems. For this purpose, we used ten types of training algorithms in the pattern network in MATLAB. We used Levenberg-Marquardt (LM), Bayesian regularization backpropagation (BR), BFGS Quasi-Newton (BFG), Resilient Backpropagation (RP), Scaled Conjugate gradient backpropagation (SCG), Conjugate Gradient with Powell/Beale Restarts (CGB), Fletcher-Powell Conjugate Gradient (CGF), Polak-Ribiere Conjugate Gradient (GDM), One Step Secant (OSS), and Variable Learning Rate Backpropagation (GD) algorithm. In this survey paper, we also check, affects of these algorithms on neural network when we applied different types of hidden neuron size. During this survey we found some new facts. We found that RP, SCG, CGB, CGF and OSS are fastest algorithms. BFG takes more time with respect to hidden neuron size. GDM and GD take more epochs. BR algorithm is not acceptable for image categorization.

Clinical Characteristics and Mathematical Analysis of Curative Effect of Hemodialysis in Curing Poisoning Caused by Snakebite.

In order to explore the clinical characteristics of hemodialysis in curing poisoning from snakebites, a two-classification model of nuclear logistic neural network based on restricted Boltzmann machine is proposed. The model combines kernel logistic regression with artificial neural networks, enabling the model to both learn autonomously and handle linearly inseparable problems. The network first performs feature learning through unsupervised training of restricted Boltzmann machines and obtains the initial values of the parameters to be identified, which reduces the influence of the randomness of the initial parameters. The variable universe learning rate with scaling factor is used to learn the parameters to be identified, and the model convergence speed is improved by dynamic adjustment of the learning rate. Experimental results show the following: Compared with before treatment, patient's activated partial thromboplastin time (APTT) after treatment and the prothrombin time (PT) level decrease, fibrinogen (FIB) levels are elevated, aspartate transferase (AST) and creatine kinase isoenzyme (CK-MB) level decreased, and the differences were statistically significant (p < 0.05). It is proved that continuous hemodiafiltration combined with plasma exchange treatment can effectively improve the blood coagulation index and myocardial index of severe snakebite poisoning patients.

Performance Analysis of Neural Network Based Classifiers for the Prediction of Diabetes

Diabetes is the most harmful diseases to consider in recent years since it causes severe damage to human beings in the form of elevated sugar levels. In a recent survey, it was projected that over 385 million public were affected in the entire world. Several investigators were conducted various experiments for prediction of diabetes using various classification techniques. This paper deals with a neural classifier based prediction system to recognize diabetes. Two learning algorithms namely, Levenberg Marquardt back propagation (LM), and gradient descent with variable learning rate are is investigated for different architecture and the best architecture with good accuracy was identified. The data are together from the Government Hospital of Pondicherry and it is formed as a database. Totally, datasets of 500 have been together, out of which 350 datasets as training sets for training process and 150 datasets as testing sets for the testing process. The recognition accuracy is obtained. For comparison, k-Nearest Neigourhood and the K- nearest neighbor and Radial Basis Function (RBF) network are also implemented and it is trained and tested with the same datasets. The result shows that Neural Network outperforms well with other classifiers.

Gradient Techniques To Predict Distributed Denial-Of-Service Attack

A distributed denial-of-service (DDoS) attack attempts to prevent people from accessing a server. A website may become inaccessible due to a DDoS attack because the server is inundated with fake requests and cannot handle real ones. A DDoS attack affects a large number of computers. Attackers employ a zombie network, which is a collection of infected machines on which the attacker has hidden the denial-of-service attacking application to carry out a DDoS attack. The MATLAB 2018a simulator was used in this study for training. Additionally, during design, the knowledge discovery dataset (KDD) was cleaned and the values of attacks were incorporated. A neural network model was subsequently developed, and the KDD was trained using a recursive artificial neural network. This network was developed using five distinct training algorithms: 1) Fletcher–Powell conjugate gradient, 2) Polak–Ribiére conjugate gradient of, 3) resilient backpropagation, 4) gradient conjugation with Powell/Beale restarts, and 5) gradient descent algorithm with variable learning rate. The artificial neural network toolset in MATLAB was used to investigate the detection of DDoS attacks. The conjugate gradient with Powell/Beale restart algorithm had a success rate of 99.9% and a training time of 00:53. This inquiry uses the KDD-CUP99 dataset. Has a better level of accuracy, according to the results

Fault Diagnosis of Aeroengine Control System Sensor Based on Optimized and Fused Multidomain Feature

Improving the efficacy and dependability of aeroengines requires timely and effective sensor fault diagnosis. Deep learning-based fault diagnosis method is a current research hotspot. To overcome some of the method’s existing shortcomings and improve the reliability of fault diagnosis, this paper proposes a novel intelligent fault diagnosis framework with higher quality features and more effective fault classifiers. The proposed plan includes three stages. Firstly, multidomain features (time and frequency domain features) are extracted to describe the sensor’s health from several dimensions. Secondly, the advanced Henry gas solubility optimization algorithm (HGSO) is applied to improve classification accuracy through feature selection, and the operating conditions and the features extracted by the network are fused as fault indicators. Finally, an adaptive deep belief network (ADBN) with relu-softsign combination activation layers, variable learning rate, and optimized network structure is proposed as the fault identifier. The advantages of the first two stages lie in the complete utilization of information and reducing the data dimension. In addition, the detection performance and the convergence speed is enhanced by the proposed ADBN. The experimental data are derived from a combination of measured and simulated data generated from the aeroengine model. The experimental results indicate that the improved method can produce better performance and outcomes than the unimproved methods for all fault scenarios, with a higher diagnostic accuracy of 98.1% and a reduced time of 98 s. The efforts of this study provide a efficient and adaptable way to aeroengine sensor fault diagnosis.

Emotion model of associative memory possessing variable learning rates with time delay

Lots of researchers have used memristors to realize the emotion model of associative memory. In previous works, researchers analyzed this associative memory from two perspectives—forgetting and variable learning rate. In the previous emotion model, neutral stimulus(message notification) and unconditioned reflex(good or bad message) were applied simultaneously. But the variable learning rate with time delay is not considered in the emotion model. When the unconditioned reflex lags behind the neutral stimulus, the associative memory can also be formed. This article proposes an emotion model of variable learning rate with time delay. We also consider three kinds of forgetting: only a stimulus of unconditioned reflex applied, only a neutral stimulus applied and neither stimulus of unconditioned reflex nor neutral stimulus applied. In the end, the software PSPICE is used to simulate the whole circuit. This paper provides an option to realize emotional learning based on memristor.

BER Analysis of Neural Equalizer in OFDM systems

Digital communication gives larger data capacity and best communication speed. The signal transmitted through the wireless channels in digital communication affected by nonlinearities like noise. Inter Symbol Interference (ISI) dominantly causes huge data loss. It is caused due to multipath propagation, band-limited channels noise. Equalization, a process of inducing inverse channel response to compensate the effects of ISI is used to combat these effects. Orthogonal Frequency Division Multiplexing (OFDM) system supports high data rates and offers resistance towards the channel effect. Adding equalizer to OFDM systems make it more robust.In real time situations the response of channel is not known in prior. So, adaptive equalizers are implemented to compensate channel effects with the help of pilot symbols which are used to estimate the channel response and equalize the signal accordingly. Minimum Mean Square Error (MMSE), Least Square (LS) are some of the existing algorithms used to implement the equalizers. These conventional algorithms require some statistical properties of channel which is not possible in real time scenarios. Also, Linear Transverse equalizer failed to provide satisfactory results in the case of Non Minimum Phase channel. Neural Networks is capable to perform this task and can be used as an alternative to these algorithms.Back Propagation Neural Networks (BPNN) and Deep Neural Networks (DNN) can achieve better results when compared to conventional algorithms. Here, Deep Neural Networks with Adam optimization is proposed to enhance the performance of equalization. DNN can be used in both linear and non-linear channels. Also, Adam optimization used with Back Propagation uses variable and adaptive learning rate which reduces slow convergence and increases the efficiency of neural network unlike stochastic gradient descent which uses fixed learning rate.In this work the performance of proposed equalizer in OFDM systems is analyzed over flat and frequency selective fading of Rayleigh channels. The performance is measured in terms of Bit Error rate over SNR range -30dB to 25 dB. The results are compared with that of MMSE and it is observed that over the mentioned range of SNR the proposed equalizer achieved acceptable results and has low Bit error rate than that of MMSE in both flat and frequency selective fading channels.

Retraction Note to: Integral Backstepping Control of LPMSM Drive System Using Revised Recurrent Fuzzy NN and Mended Particle Swarm Optimization

A linear permanent magnet synchronous motor (LPMSM) drive system is keeping in many nonlinear effects such as the external load force, the flux saturation, the cogging force, the column friction and Stribeck force, and the parameters variations. Due to the uncertainty effects the existing linear controllers can not achieve better control performances for the LPMSM drive system. To raise robustness under occurrence of uncertainty, the integral backstepping control system with hitting function is proposed for controlling the LPMSM drive system in accordance with the Lyapunov function. To improve larger chattering phenomenon under uncertainties effects, the integral backstepping control system with revised recurrent fuzzy neural network (RRFNN) and mended particle swarm optimization (MPSO) is proposed to operate the LPMSM drive system to raise robustness of system. The RRFNN is used to estimate the value of the external lumped force uncertainty. Moreover, the error compensation control with the error compensation mechanism is proposed to compensate the minimum reconstructed error of the error estimation law. Besides, four variable learning rates in the weights of the RRFNN are regulated by virtue of MPSO with segment regulation to speed-up parameter’s convergence. Finally, comparative performances through some tentative upshots are verified that the integral backstepping control system by virtue of RRFNN with MPSO has better control performances than those of the proposed methods for the LPMSM drive system.

Monitoring of weld defects of visual sensing assisted GMAW process with galvanized steel

ABSTRACT Due to the existence of high-temperature and high-pressure zinc vapor in the gas metal arc welding (GMAW) process of galvanized steel, it is difficult to achieve accurate and real-time detection of weld defects, which brings great challenges to robotic welding manufacturing. In order to realize the automatic classification and prediction of weld defects, a method of galvanized steel weld defects detection based on active visual sensing and machine learning was proposed. First, Gabor filter was used to remove arc light, noise, and other interference signals to obtain the weld centerline image. Then, according to the five different weld defects of the geometric and spatial distribution characteristics in weld centerline image were analyzed with the principle of sub-pixel level. Finally, using the eight feature parameters extracted from the weld feature points, a variable learning rate and momentum factor back propagation (VL-MFBP) neural network model was designed. The model introduced a variable learning rate and momentum factor to quickly find the optimal solution in a short time, its performance was better than traditional machine learning algorithms. The experimental results show that the accuracy of weld defect recognition is 98.15%, and the average processing time of a single image is only 183.74 ms.

Performance of DVR Using Optimized PI Controller Based Gradient Adaptive Variable Step LMS Control Algorithm

A custom power device known as dynamic voltage restorer (DVR) has been investigated to operate a distribution system at its desired performance. The distribution system with voltage-related power quality issues in the supply side has been addressed through this article using DVR. Four different grid disturbances, such as voltage sag, swell, unbalances, and distortions, have been taken into account while testing the DVR capability. DVR has been controlled using optimized proportional and integral (PI) gains integrated with gradient adaptive variable learning rate least mean square control algorithm. Adaptiveness of variable step-size in LMS makes the control robust in case of dynamics in the system, which assures better performance. Also, the other major contribution of this article is the implementation of optimization-based self-tuning PI gains in the proposed control. The evaluation of optimizer in estimating the PI gains is presented in terms of the response of dc-link voltage DVR. The simulation work of proposed control algorithms on DVR has been done and satisfactory results are found. For the experimental validation, d-SPACE made Micro Lab Box is used as control processor, and both dynamic and steady-state results are discussed for its effectiveness.

Viscosity prediction of hydrocarbon binary mixture using an artificial neural network-group contribution method

Abstract In this study, the viscosity of hydrocarbon binary mixtures has been predicted with an artificial neural network and a group contribution method (ANN-GCM) by utilizing various training algorithm including Scaled Conjugate Gradient (SCG), Levenberg-Marquardt (LM), Resilient back Propagation (RP), and Gradient Descent with variable learning rate back propagation (GDX). Moreover, different transfer functions such as Tan-sigmoid (tansig), Log-sigmoid (logsig), and purelin were investigated in hidden and output layer and their effects on network precision were estimated. Accordingly, 796 experimental data points of viscosity of hydrocarbon binary mixture were collected from the literature for a wide range of operating parameters. The temperature, pressure, mole fraction, molecular weight, and structural group of the system were selected as the independent input parameters. The statistical analysis results with R 2 = 0.99 revealed a small value for Average absolute relative deviation (AARD) of 1.288 and Mean square error (MSE) of 0.001018 by comparing the ANN predicted data with experimental data. Neural network configuration was also optimized. Based on the results, the network with one hidden layer and 27 neurons with the Levenberg-Marquardt training algorithm and tansig transfer function for hidden layer along with purelin transfer function for output layer constituted the best network structure. Further, the weights and bias were optimized to minimize the error. Then, the obtained results of the present study were compared with the data from some previous methods. The results suggested that this work can predict the viscosity of hydrocarbon binary mixture with better AARD. In general, the results indicated that combining ANN and GCM model is capable to predict the viscosity of hydrocarbon binary mixtures with a good accuracy.

Application of Neural Network Predictive Control Methods to Solve the Shipping Container Sway Control Problem in Quay Cranes

Smart control systems are mostly applied in industry to control the movements of heavy machinery while optimizing overall operational efficiency. Major shipping companies use large quay cranes to load and unload containers from ships and still rely on the experience of on-site operators to perform transportation control procedures using joysticks and visual contact methods. This paper presents the research results of an EU-funded project for the Klaipeda container terminal to develop a novel container transportation security and cargo safety assurance method and system. It was concluded that many risks arise during the container handling procedures performed by the quay cranes and operators. To minimize these risks, the authors proposed controlling the sway of the spreader using a model predictive control method which applies a multi-layer perceptron (MLP) neural network (NN). The paper analyzes current neural network architectures and case studies and provides the engineering community with a unique case study which applies real operation statistical data. Several key training algorithms were tested, and the initial results suggest that the Levenberg–Marquardt (LM) algorithm and variable learning rate backpropagation perform better than methods which use the multi-layer perceptron neural network structure.

Investigation of neutron-induced reaction cross section calculations for the fusion reactor structural materials using artificial neural networks

Using artificial neural networks for an estimation of the nuclear reaction cross section data is discussed. Approximately rate of the fitting criteria is determined by the calculated experimental data obtained from using Variable Learning Rate Backpropagation (traingdx) algorithm in artificial neural networks. This method has been applied to obtain the cross section for 14–15 MeV neutron-induced (n,α) and (n,p) reactions in the fusion reactor structural materials. In comparison to the reaction cross section calculation by experimental cross sections reported in EXFOR, TALYS 1.9 and EMPIRE 3.2, the proposed method has better prediction ability when the target output has a large variation between the experimental and the calculated data. This study is substantial for the new method validation development of the nuclear model approaches with the increased prediction power of the neutron-induced reactions for fusion reactor systems.

Latency-Based Analytic Approach to Forecast Cloud Workload Trend for Sustainable Datacenters

Cloud datacenters are turning out to be massive energy consumers and environment polluters, which necessitate the need for promoting sustainable computing approaches for achieving environment-friendly datacentre execution. Direct causes of excess energy consumption of the datacentre include running servers at low level of workloads and over-provisioning of server resources to the arriving workloads during execution. To this end, predicting the future workload demands and their respective behaviors at the datacenters are being the focus of recent researches in the context of sustainable datacenters. But prediction analytics of cloud workloads suffer various limitations imposed by the dynamic and unclear characteristics of Cloud workloads. This paper proposes a novel forecasting model named K-means based Rand Variable Learning Rate Backpropagation Neural Network (K-RVLBPNN) for predicting the future workload arrival trend, by exploiting the latency sensitivity characteristics of Cloud workloads, based on a combination of improved K-means clustering algorithm and Backpropagation Neural Network (BPNN) algorithm. Experiments conducted on real-world Cloud datasets shows that the proposed model shows better prediction accuracy, outperforming the traditional Hidden Markov Model, Naive Bayes Classifier, and our earlier RVLBPNN model, respectively.

빅데이터를 이용한 인공신경망 기반 시간별 전력수요 예측

Short-term power load prediction is crucial for scheduling generators, securing transmission capacity, and determining economic market price. This paper proposes a Bigdata-based Artificial Neural Network (B-ANN) model for a short-term load forecasting by utilizing decision-tree method. The bigdata has been composed of 2018 time index, meteorological data, exchange rates, oil price, and past power demand in Seoul. The decision-tree method is applied to classify the data in accordance with the value of decreasing entropy. In addition, the resilient back propagation algorithm (RPROP) is applied for ANN that derives fast results and result in reduced error through variable learning rates. The applicability and effectiveness of the proposed model are verified by conducting simulations for forecasting 1-day hourly power load. The results confirm that the proposed model has decreased the mean absolute percentage error (MAPE) dramatically in comparison with other forecasting methods such as multiple regression analysis, time series analysis, and auto-regressive integrated moving average (ARIMA) model.

Neural network approach for solving nonlocal boundary value problems

This paper proposes a radial basis function (RBF) network-based method for solving a nonlinear second-order elliptic equation with Dirichlet boundary conditions. The nonlocal term involved in the differential equation needs a completely different approach from the up-to-now-known methods for solving boundary value problems by using neural networks. A numerical integration procedure is developed for computing the local $$L^2$$ -inner product. It is known that the non-variational methods are not effective in solving nonlocal problems. In this paper, the weak formulation of the nonlocal problem is reduced to the minimization of a nonlinear functional. Unlike many previous works, we use an integral objective functional for implementing the learning procedure. Well-distributed nodes are used as the centers of the RBF neural network. The weights of the RBF network are determined by a two-point step size gradient method. The neural network method proposed in this paper is an alternative to the finite-element method (FEM) for solving nonlocal boundary value problems in non-Lipschitz domains. A new variable learning rate strategy has been developed and implemented in order to avoid the divergence of the training process. A comparison between the proposed neural network approach and the FEM is illustrated by challenging examples, and the performance of both methods is thoroughly analyzed.

Linear permanent magnet synchronous motor drive system using AAENNB Control system with error compensation controller and CPSO

Due to nonlinear friction and uncertainty effects of linear permanent magnet synchronous motor (LPMSM), the existing linear controllers cannot achieve good control performance. In order to increase the robustness of system under some uncertainties disturbances, the proposed adaptive amended Elman neural network backstepping (AAENNB) control system with error compensation controller is adopted for controlling the LPMSM drive system. Firstly, the field-oriented control (FOC) is applied to formulate the dynamic equation of the LPMSM drive system. Secondly, a backstepping approach is proposed for controlling the motion of LPMSM drive system. The proposed backstepping control system and the mover position of the LPMSM drive system possess good transient control performance under the uncertainties action for the tracking of periodic references. Because of the LPMSM with nonlinear and time-varying dynamic characteristics, an adaptive amended Elman neural network uncertainty observer (AAENNUO) with the adaptive law is proposed to estimate the required lumped uncertainty. Moreover, the error compensation controller with the error estimation law is proposed to compensate the minimum reconstructed error according to Lyapunov stability theorem. Furthermore, to improve convergent speed and to obtain better learning performance, the varied learning rate of the weight in the AENN is regulated by use of the corrected particle swarm optimization (CPSO) algorithm with segment regulation mechanics, that is, the innovativeness for using the CPSO algorithm. At last, the usefulness of the proposed control system is confirmed by experimental results.

Embedding ecological knowledge into artificial neural network training: A marine phytoplankton primary production model case study

Enhancing the understanding of marine phytoplankton primary production is paramount due to the relationships with oceanic food webs, energy fluxes, carbon cycle and Earth's climate.As field measurements of this process are both expensive and time consuming, indirect approaches, which can estimate primary production from remotely sensed imagery, are the only viable large-scale solution. We boosted the quality of phytoplankton primary production estimates, with respect to a previously developed model, by embedding ecological knowledge into the training of an artificial neural network. In order to achieve this goal, we drove the training procedure on the basis of both theoretical and data-derived ecological knowledge about phytoplankton primary production. A “single peak” constraint exploits the theoretical knowledge about the vertical shape of the production profile; a “depth-weighted error” procedure was based on available information about the production magnitude along the water column; a variable learning rate and momentum approach allowed to better exploit the available data to train our artificial neural network.Thanks to this customized procedure, we improved the quality of the primary production estimates from both a theoretical and a numerical point of view. Accordingly, the new artificial neural networks not only provided ecologically sounder estimates, but also explained up to 4% more variance with respect to the traditional error back-propagation solution. This result was achieved exclusively through an ecologically-driven customization of the basic algorithm, since the dataset and predictive variables were the same utilized in the conventional counterpart, trained with a classic error back-propagation algorithm.We suggest that the proposed rationale could lead to improved performances in similar modelling applications.

RETRACTED: Integral backstepping control with RRFNN and MPSO of LPMSM drive system

A linear permanent magnet synchronous motor drive system is existed in many nonlinear effects such as the external load force, the flux saturation, the cogging force, the column friction and Stribeck force, and the parameters variations. Due to the uncertainty effects, the linear permanent magnet synchronous motor drive system is hard to achieve the good control performance by using linear controller. To raise robustness under occurrence of uncertainty, the integral backstepping control system with hitting function is first proposed for controlling the linear permanent magnet synchronous motor drive system. The used integrator can ameliorate the system’s robustness under the parameters uncertainties and external force disturbances. To reduce vibration of control strength, the integral backstepping control system by means of the revised recurrent fuzzy neural network with mended particle swarm optimization is next proposed to operate the linear permanent magnet synchronous motor drive system to raise robustness of system. Furthermore, four variable learning rates in the weights of the revised recurrent fuzzy neural network are adopted by using mended particle swarm optimization to speed up parameter’s convergence. Finally, comparative performances through some experimental upshots are verified that the integral backstepping control system by means of revised recurrent fuzzy neural network with mended particle swarm optimization has better control performances than those of the proposed methods for the linear permanent magnet synchronous motor drive system.