Adaptive Learning Rate Algorithm Research Articles

Using adaptive learning rate to generate adversarial images

Abstract Convolutional neural networks (CNNs) have proved their efficiency in performing image classification tasks, as they can automatically extract the image features and make the corresponding prediction. Meanwhile, the CNNs application is highly challenged by their vulnerability to adversarial samples. These samples are slightly different from the legitimate samples, but the CNN gives wrong classification. There are various ways to find the adversarial samples. The most common method is using backpropagation to generate gradients as the directed perturbation. Contrarily to set a constrained limitation, in this paper, we use iterative fast gradient sign method to generate adversarial images with the minimum perturbation. The CNNs were trained to perform surgical tool recognition as a configuration for the modern operation room. The coefficient or the learning rate which influenced the modification per iteration, was set to be adaptive instead of a fixed number. A few functions were utilized to perform the learning rate decay to compare the performance. Especially, we propose a new adaptive learning rate algorithm that consider the loss as a part of influence factor constitute the learning rate for the rest iterations. According to the experiments, our loss adaptive learning rate method was proved to be efficient to get the minimal perturbations for adversarial attack.

Intelligent fault diagnosis scheme for rolling bearing based on domain adaptation in one dimensional feature matching

The rapid development of intelligent manufacturing technology has made data-driven fault diagnosis technology an important research at present. Most existing intelligent approaches, on the other hand, have a flaw: training and testing data are collected under the same operational conditions. As a result, a feature knowledge transfer learning method for rotating machinery fault detection is presented in this study to encourage the effective implementation of intelligent fault diagnosis. This method combines the domain adaptive ability of transfer learning and the ability of deep learning to automatically extract features Firstly, unsupervised convolutional auto-encoder is used for learning information from multiple source domains to construct feature subspace. Then adaptive learning rate, weight initialization and weight change algorithm are used to construct feature matching algorithm and incorporated into the stack-encoder. Finally, the encoder is used for diagnostic recognition of feature subspace information. Three datasets from various experimental platforms are used to validate the proposed method’s efficacy. The experimental findings demonstrate the method’s capacity to diagnose and identify multi-domain faults in bearings while having strong generalization and flexibility. Comparatively speaking, the proposed model may attain higher diagnostic accuracy with a rate of accuracy of above 98%. In addition, two additional analytical experiments were conducted. The findings demonstrate that the method may provide over-task transfer learning and reliable diagnostic results. And, it demonstrates how improved diagnostic models may be built with a good dataset of unlabeled source domains.

Ku-Band SIW Filter with High Fractional Bandwidth Optimized Using Feed Forward Back Propagation ANN

An Artificial Neural Network (ANN) based Substrate Integrated Waveguide Band Pass Filter (SIW BPF) is proposed in this paper. A Feed Forward Back Propagation Neural Network (FF-BP NN) is utilized to optimize the filter dimensions. The Gradient Descent with Momentum and Adaptive Learning Rate algorithm is used to train the network at a learning rate of 0.01. The high pass characteristics of the SIW is converted into band pass by incorporating U slots on its top plane. Defected Ground Structure (DGS) is utilized in the bottom plane to improve the impedance matching. To validate, the prototype is fabricated using RT/Duroid 5880 and tested. The proposed filter has a center frequency at 14.68 GHz with a wide pass band from 12.92 to 16.43 GHz with a 3-dB Fractional Bandwidth (FBW) of 24%, return loss more than 20 dB and insertion loss of about 1.9 dB within the pass band. The filter has a small dimension of about 0.63 , where λ g is the guided wavelength at the center frequency. This filter offers wide passband, smaller size, low insertion loss, good return loss and it is useful in Ku-band satellite communication applications.

Energy-efficiency-oriented optimal control for electrical environmental control system based on advanced neural network

Electric environmental control system (EECS) adopts the unique architecture of electric compressor bleed air and multi-stage expansion refrigeration. The state of EECS is regulated by the electric compressor and multiple regulation valves together, resulting in flow and temperature of each node strongly coupled. Traditional methods use a single valve to control each objective without considering the effect of control scheme on energy efficiency, which will not only lead to poor control effect, but also limit the coefficient of performance (COP) of EECS. This paper proposes an energy-efficiency-oriented optimal control strategy to improve the COP and control performance simultaneously. Part of the regulation valves are regarded as optimization variables and the corresponding control objectives are regarded as constraints to maximize COP. On this basis, an advanced control algorithm is proposed to further improve the control effect. The COP prediction model is developed with an offline trained neural network and the optimal valve openings are solved by the differential evolution algorithm. The controller is developed with another neural network online trained by adaptive learning rate algorithm to adapt to complex conditions. The results show that the COP and control performance of EECS are significantly improved. Compared to other strategies, the optimal control strategy can increase COP by up to 61.14%. This study provides an effective solution to the optimal control problem of systems with similar structure to EECS.

Research on Intelligent Algorithm of the AC Motor Speed Regulation System Based on the Neural Network

One of the most challenging control problems has been the speed control of an alternating current (AC) motor due to the highly nonlinear characteristics and many uncertain parameters including the magnetic flux, temperature-dependent rotor resistance, and the variable load. In this study, an intelligent control algorithm is proposed for the AC motor speed control using a PID controller-based backpropagation neural network (BP-NN). The momentum factor adaptive learning rate algorithm is introduced to improve the PID-based BP-NN. A simulation model representing the complete AC motor speed regulation system is established and tested using the MATLAB program. The simulation results show that this control strategy has strong adaptability and robustness when the controlled object is unknown or the parameters change. Compared with other PID and neural network parameter adjustment methods, the model has the potential to intelligently regulate the speed of the AC motor.

An adaptive federated learning scheme with differential privacy preserving

Driven by the upcoming development of the sixth-generation communication system (6G), the distributed machine learning schemes represented by federated learning has shown advantages in data utilization and multi-party cooperative model training. The total communication costs of federated learning is related to the number of communication rounds, the communication consumption of each participants, the setting of reasonable learning rate and the guarantee of calculation fairness. In addition, the isolating data strategy in the federated learning framework cannot completely guarantee the privacy security of users. Motivated by the above problems, this paper proposes a federated learning scheme combined with the adaptive gradient descent strategy and differential privacy mechanism, which is suitable for multi-party collaborative modeling scenarios. To ensure that federated learning scheme can train efficiently with limited communications costs, the adaptive learning rate algorithm is innovatively used to adjust the gradient descent process and avoid the model overfitting and fluctuation phenomena, so as to improve the modeling efficiency and performance in multi-party calculation scenarios. Furthermore, in order to adapt to the ultra-large-scale distributed secure computing scenario, this research introduces differential privacy mechanism to resist various background knowledge attacks. Experimental results demonstrate that the proposed adaptive federated learning model performs better than the traditional models under fixed communication costs. This novel modeling scheme also has strong robustness to different super-parameter settings and provides stronger quantifiable privacy preserving for federated learning process.

TIME SERIES FORECASTING USING ARTIFICIAL NEURAL NETWORK WITH EXTENDED ADAPTIVE LEARNING

Artificial neural network (ANN) mainly consists of learning algorithms, which are require to optimize the convergence of neural networks. We need to optimize the convergence of neural networks in order to improve the speed and accuracy of decision making process. To enable the optimization process one of the widely used algorithm is back propagation learning algorithm. Objective of study is to applied backpropagation algorithm for solving multivariate time series problem. To better the accuracy of neural network it is important to find optimized architecture for the problem under consideration. The learning rate is also an important factor which affects the performance of result. In this study, we proposed extended adaptive learning approach in which learning rate is adapted from number of previous iteration error trend in first half of training. In next half of training learning rate is adapted as per adaptive learning rate algorithm. Compare performance of three variation of backpropagationalgorithm. All these variation experimented on two standard dataset. Experimental result shows that during validation and training ANN with extended adaptive learning rate outperforms other than two variations.

Deep Learning Based Antenna Design and Beam-Steering Capabilities for Millimeter-Wave Applications

In this study, a deep neural network (DNN) is implemented to soft computation of the dual-band circularly polarized bone-shaped patch antenna (BSPA) at 28 GHz and 38 GHz for 5G applications. Via a simulated database of 150 BSPAs, a DNN model is constructed on a 5-layer system using an adaptive learning rate algorithm. The framework and hyper-parameters of the DNN model are optimized in the training phase of a hybrid algorithm combining strengths of both particle swarm optimization (PSO) and a modified version of the gravitational search algorithm (MGSA-PSO). To generate the database for training and testing the model, 150 BSPAs with different geometrical are simulated in terms of the resonant frequency using a precise electromagnetic analysis platform. A fabricated BSPA operating at 28 GHz and 38 GHz is used to test and verify the DNN model. Then, the application of DNN with back-propagation algorithm and weighted MGSA-PSO algorithm is used for beam-steering the main beam pattern of the designed uniform circular antenna array with side-lobe level <= −30 dB by estimating the appropriate feeding phases of the 16 elements. Several illustrative examples are placed to beam-steer the pattern in the desired direction to check the validity of the technique.

A novel system identification algorithm for quad tilt-rotor based on neural network with foraging strategy

Quad tilt-rotor(QTR) UAV is a nonlinear time-varying system in full flight mode. It is difficult and inaccurate to model the nonlinear time-varying system, which cannot fully reflect the problem of controlling input and system response output in the full flight mode. In order to solve the above problems, a novel neural network model was adopt to identify the nonlinear time-varying system of quad tilt-rotor in full flight mode. An adaptive learning rate algorithm based on foraging strategy is proposed based on the global error BP neural network. Corresponding to the nonlinear time-varying system, BP neural network is set as the time-invariant system structure with constant network structure and continuously changing weights at multiple times, and the nonlinear input-output relationship under the time-varying system is jointly described by fitting the network at all times. The extended Kalman filtering algorithm is used to track the network connection weights by modifying the network weights at the current moment with the input and output data at the next moment. The final identification result shows that the smaller mean square error of both only transition process and full flight mode, shows that using this optimization algorithm can well describe the input and output characteristics of the nonlinear time-varying systems. When the same network structure is adopted, no matter for transition mode or full mode, the BP optimization algorithm based on foraging strategy is better than the global BP algorithm for system identification of the full mode quad tilt-rotor. Therefore, when the BP neural network based on foraging strategy is adopted, the same network structure can be adopted to systematically identify the full mode of quad tilt-rotor by changing the weight.

A multiple multilayer perceptron neural network with an adaptive learning algorithm for thyroid disease diagnosis in the internet of medical things

Medical information systems such as Internet of Medical Things (IoMT) are gained special attention over recent years. X-ray and MRI images are important sources of information to be examined for a particular type of anomalies. Reports based on the images and laboratory examination results could be mined with machine learning techniques as well. Thyroid disease diagnosis is an important capability of medical information systems. The main objective of this study is to improve the diagnosis accuracy of thyroid diseases from semantic reports and examination results using artificial neural network (ANN) in IoMT systems. In order to improve generalization and avoid over-fitting of ANN during the training process, a set of multiple multilayer perceptron (MMLP) neural network with the back-propagation error ability is proposed in this paper. Moreover, an adaptive learning rate algorithm is used to deal with the slow convergence and the local minima problem of the back-propagation error algorithm. The proposed MMLP significantly increased the overall accuracy of thyroid disease classification. With MMLP with a set of 6 networks, an improvement of 0.7% accuracy is achieved compared to a single network. In addition, comparing to the standard back-propagation, by using an adaptive learning rate algorithm in the proposed MMLP, an improvement of 4.6% accuracy and the final accuracy of 99% have been obtained in IoMT systems. The proposed MMLP is compared to recent researches reported for thyroid disease diagnosis, and its superiority is shown.

Mutual Information Based Learning Rate Decay for Stochastic Gradient Descent Training of Deep Neural Networks.

This paper demonstrates a novel approach to training deep neural networks using a Mutual Information (MI)-driven, decaying Learning Rate (LR), Stochastic Gradient Descent (SGD) algorithm. MI between the output of the neural network and true outcomes is used to adaptively set the LR for the network, in every epoch of the training cycle. This idea is extended to layer-wise setting of LR, as MI naturally provides a layer-wise performance metric. A LR range test determining the operating LR range is also proposed. Experiments compared this approach with popular alternatives such as gradient-based adaptive LR algorithms like Adam, RMSprop, and LARS. Competitive to better accuracy outcomes obtained in competitive to better time, demonstrate the feasibility of the metric and approach.

Barzilai–Borwein-based adaptive learning rate for deep learning

Learning rate is arguably the most important hyper-parameter to tune when training a neural network. As manually setting right learning rate remains a cumbersome process, adaptive learning rate algorithms aim at automating such a process. Motivated by the success of the Barzilai–Borwein (BB) step-size method in many gradient descent methods for solving convex problems, this paper aims at investigating the potential of the BB method for training neural networks. With strong motivation from related convergence analysis, the BB method is generalized to adaptive learning rate of mini-batch gradient descent. The experiments showed that, in contrast to many existing methods, the proposed BB method is highly insensitive to initial learning rate, especially in terms of generalization performance. Also, the BB method showed its advantages on both learning speed and generalization performance over other available methods.

Online Embedding Compression for Text Classification Using Low Rank Matrix Factorization

Deep learning models have become state of the art for natural language processing (NLP) tasks, however deploying these models in production system poses significant memory constraints. Existing compression methods are either lossy or introduce significant latency. We propose a compression method that leverages low rank matrix factorization during training, to compress the word embedding layer which represents the size bottleneck for most NLP models. Our models are trained, compressed and then further re-trained on the downstream task to recover accuracy while maintaining the reduced size. Empirically, we show that the proposed method can achieve 90% compression with minimal impact in accuracy for sentence classification tasks, and outperforms alternative methods like fixed-point quantization or offline word embedding compression. We also analyze the inference time and storage space for our method through FLOP calculations, showing that we can compress DNN models by a configurable ratio and regain accuracy loss without introducing additional latency compared to fixed point quantization. Finally, we introduce a novel learning rate schedule, the Cyclically Annealed Learning Rate (CALR), which we empirically demonstrate to outperform other popular adaptive learning rate algorithms on a sentence classification benchmark.

Multi-Objective Optimization Design through Machine Learning for Drop-on-Demand Bioprinting

Abstract Drop-on-demand (DOD) bioprinting has been widely used in tissue engineering due to its high-throughput efficiency and cost effectiveness. However, this type of bioprinting involves challenges such as satellite generation, too-large droplet generation, and too-low droplet speed. These challenges reduce the stability and precision of DOD printing, disorder cell arrays, and hence generate further structural errors. In this paper, a multi-objective optimization (MOO) design method for DOD printing parameters through fully connected neural networks (FCNNs) is proposed in order to solve these challenges. The MOO problem comprises two objective functions: to develop the satellite formation model with FCNNs; and to decrease droplet diameter and increase droplet speed. A hybrid multi-subgradient descent bundle method with an adaptive learning rate algorithm (HMSGDBA), which combines the multi-subgradient descent bundle (MSGDB) method with Adam algorithm, is introduced in order to search for the Pareto-optimal set for the MOO problem. The superiority of HMSGDBA is demonstrated through comparative studies with the MSGDB method. The experimental results show that a single droplet can be printed stably and the droplet speed can be increased from 0.88 to 2.08 m·s−1 after optimization with the proposed method. The proposed method can improve both printing precision and stability, and is useful in realizing precise cell arrays and complex biological functions. Furthermore, it can be used to obtain guidelines for the setup of cell-printing experimental platforms.

Deep neural network–based soft computing the resonant frequency of E–shaped patch antennas

In this study, deep neural network (DNN) is implemented to soft computation of the resonant frequency of E–shaped patch antenna (ESPA). A DNN model is built on a 5-layer framework using an adaptive learning rate algorithm though a simulated database of 144 ESPAs. The framework and hyperparameters of the DNN model are optimized by exploiting K–fold cross validation method in the training phase. In order to generate the database for training and testing the model, 144 ESPAs with different geometrical and electrical parameters are simulated in terms of the resonant frequency using a precise electromagnetic analysis platform. The DNN model is trained on the dataset#130 with an average percentage error (APE) of 0.269 between the simulated and computed frequency values. For corroboration of the proposed DNN model is tested and verified by comparing with previously applied traditional neural networks on the remainder dataset#14, and even verified on a fabricated ESPA operating at 2.4 GHz. The DNN model computes the resonant frequency of the testing dataset#14 with the highest accuracy of 0.285 APE. The results show that the proposed deep neuro-computing model for the resonant frequency is a powerful and helpful technique as an easy alternative to prohibitive measurement and simulation.

The Neural Network of One-Dimensional Convolution-An Example of the Diagnosis of Diabetic Retinopathy

Diabetes is a serious threat to health development, because diabetes is a disease that caused most other diseases (complications). Diabetic retinopathy is the most important manifestation of diabetic microangiopathy and is also one of the most common complications in people with diabetes. At present, the diagnosis of diabetic retinal complications mainly depends on the pictures for diagnosis. The fundus images are the main ways to diagnose retinal diseases at present, but the diagnosis process is complicated. Based on this, this paper uses the electronic medical record information of 301 hospitalized patients with diabetes from 2009 to 2013, mainly using the diabetes diagnostic data, diabetes glycosylation data and diabetes biochemical test data, the depth of learning methods and medical diabetes combined with the application of convolution Neural Network Method (CNN) to build a diagnostic model, and thus draw the diagnosis. The main contribution of this study is twofold: 1) In this paper, we apply the CNN method to one-dimensional unrelated data sets and solve the problem of how to do one-dimensional irrelevant data convolution. 2) In this paper, the CNN model is combined with the BN layer to prevent the dispersion of the gradient, speed up the training speed and improve the accuracy of the model. In addition, this model incorporates an adaptive learning rate algorithm and optimizes the model. The experiments show that this method can achieve a training accuracy of 99.85% and a testing accuracy of 97.56%, which is more than 2% higher than that of using logistic regression. The model methods involved in this study can not only be used for the diagnosis of diabetic retinopathy, but also for the diagnosis of other diseases, such as chronic kidney disease, cardiovascular, and cerebrovascular diseases.

An End-to-End Model Based on Improved Adaptive Deep Belief Network and Its Application to Bearing Fault Diagnosis

Effective machinery prognostics and health management play a crucial role in ensuring the safe and continuous operation of equipment, and satisfactory characteristics' expression of machine health status plays a key role in the ability to diagnose faults with high accuracy. At present, most methods based on signal processing and the shallow learning model rely on artificial feature extraction to identify the machine fault type. In practical applications, however, meaningful health management requires correct recognition of not only the health type but also the fault degree, if any occurs. Such recognition is useful for determining the priority level of mechanical maintenance and minimizing economic losses. Deep learning techniques, such as deep belief network (DBN), have demonstrated great potential in exploring characteristic information from machine status signals. In this paper, an end-to-end fault diagnosis model based on an adaptive DBN optimized by the Nesterov moment (NM) is proposed to extract deep representative features from rotating machinery and recognize bearing fault types and degrees simultaneously. Frequency-domain signals are inputted into the model for feature learning, and NM is introduced to the training process of the DBN model. Individual adaptive learning rate algorithms are then applied to optimize parameter updating. The performance of the proposed method is validated using a self-made bearing fault test platform, and the model is shown to achieve satisfactory convergence and a testing accuracy higher than those obtained from standard DBN and support vector machine.

A Novel Method in Two-Step-Ahead Weight Adjustment of Recurrent Neural Networks: Application in Market Forecasting

Gold price prediction is a very complex nonlinear problem which is severely difficult. Real-time price prediction, as a principle of many economic models, is one of the most challenging tasks for economists since the context of the financial agents are often dynamic. Since in financial time series, direction prediction is important, in this work, an innovative Recurrent Neural Network (RNN) is utilized to obtain accurate Two-Step- Ahead (2SA) prediction results and ameliorate forecasting per- formances of gold market. The training method of the proposed network has been combined with an adaptive learning rate algorithm and a linear combination of Directional Symmetry (DS) is utilized in the training phase. The proposed method has been developed for online and offline applications. Simulations and experiments on the daily Gold market data and the benchmark time series of Lorenz and Rossler shows the high efficiency of proposed method which could forecast future gold price precisely.

English

In this paper, new multilayer perceptron's feed forward back propagation Neural Network (NN) performance using BFGS quasi newton , Levenberg -Marquardt (LM), Gradient descent back propagation with adaptive learning rate(GDA) Algorithms are being anticipated with the project detached to develop a lossless image solidity technique using NN and to design and contrivance image compression using Neural network to achieve maximum peak signal to noise ratio (PSNR), and low mean square error (MSE) and compression levels. This paper presents a NN based procedure that may be practical to data compression and breaks down large images into smaller blocks (1x64) and eradicates redundant information. Lastly, this technique uses a NN training functions like and conversion of block codes to vector codes and vice versa. Results obtained with proposed techniques leads to better compression material. Finally, this technique uses a NN training functions like and adaptation of block codes to vector codes and vice versa. Consequencesacquired with proposed techniques leads to better compression ratio at the same time conserving the image quality. The investigational result shows that the BFG quasi newton algorithm is best among the three proposed algorithm which offers better PSNR value and also reduces the MSE value.

A Comparative Study of MLP Networks Using Backpropagation Algorithms in Electrical Equipment Thermography

Nowadays, infrared thermographic inspection has increasingly been utilized for fault diagnosis tools of electrical equipment, because it maintains non-interrupting operation of power system and ensures early diagnosis of faults. The article discusses the performance comparison of multilayer perceptron (MLP) networks using various backpropagation algorithms for thermal imaging-based condition monitoring of electrical hotspots. The training algorithms are Levenberg–Marquardt, Broyden–Fletcher–Goldfarb–Shanno quasi-Newton, resilient backpropagation, gradient descent with momentum and adaptive learning rate and standard gradient descent training algorithms. The performance of the training algorithms are evaluated in terms of percentage of accuracy, sensitivity, specificity, false-positive and false-negative results. In the beginning, thermal images of electrical equipment are captured using infrared camera. Six statistical intensity features, namely mean, maximum, minimum, median, standard deviation and variance extracted from each hotspot of equipment thermal image are used as the inputs of multilayered perceptron networks to classify the thermal conditions of hotspots into two classes, namely normal and defective. The comparison of MLP networks using training algorithms proves that the MLP network trained with resilient backpropagation algorithm gives the best performance in thermal condition recognition.