Optimal Learning Rate Research Articles

Efficient learning rate adaptation based on hierarchical optimization approach

This paper proposes a new hierarchical approach to learning rate adaptation in gradient methods, called learning rate optimization (LRO). LRO formulates the learning rate adaption problem as a hierarchical optimization problem that minimizes the loss function with respect to the learning rate for current model parameters and gradients. Then, LRO optimizes the learning rate based on the alternating direction method of multipliers (ADMM). In the process of this learning rate optimization, LRO does not require any second-order information and probabilistic model, so it is highly efficient. Furthermore, LRO does not require any additional hyperparameters when compared to the vanilla gradient method with the simple exponential learning rate decay. In the experiments, we integrated LRO with vanilla SGD and Adam. Then, we compared their optimization performance with the state-of-the-art learning rate adaptation methods and also the most commonly-used adaptive gradient methods. The SGD and Adam with LRO outperformed all the competitors on the benchmark datasets in image classification tasks.

Learning as filtering: Implications for spike-based plasticity.

Most normative models in computational neuroscience describe the task of learning as the optimisation of a cost function with respect to a set of parameters. However, learning as optimisation fails to account for a time-varying environment during the learning process and the resulting point estimate in parameter space does not account for uncertainty. Here, we frame learning as filtering, i.e., a principled method for including time and parameter uncertainty. We derive the filtering-based learning rule for a spiking neuronal network-the Synaptic Filter-and show its computational and biological relevance. For the computational relevance, we show that filtering improves the weight estimation performance compared to a gradient learning rule with optimal learning rate. The dynamics of the mean of the Synaptic Filter is consistent with spike-timing dependent plasticity (STDP) while the dynamics of the variance makes novel predictions regarding spike-timing dependent changes of EPSP variability. Moreover, the Synaptic Filter explains experimentally observed negative correlations between homo- and heterosynaptic plasticity.

Efficient method for accelerating line searches in adjoint optimization of photonic devices by combining Schur complement domain decomposition and Born series expansions.

A line search in a gradient-based optimization algorithm solves the problem of determining the optimal learning rate for a given gradient or search direction in a single iteration. For most problems, this is determined by evaluating different candidate learning rates to find the optimum, which can be expensive. Recent work has provided an efficient way to perform a line search with the use of the Shanks transformation of a Born series derived from the Lippman-Schwinger formalism. In this paper we show that the cost for performing such a line search can be further reduced with the use of the method of the Schur complement domain decomposition, which can lead to a 10-fold total speed-up resulting from the reduced number of iterations to convergence and reduced wall-clock time per iteration.

Riemannian stochastic fixed point optimization algorithm

This paper considers a stochastic optimization problem over the fixed point sets of quasinonexpansive mappings on Riemannian manifolds. The problem enables us to consider Riemannian hierarchical optimization problems over complicated sets, such as the intersection of many closed convex sets, the set of all minimizers of a nonsmooth convex function, and the intersection of sublevel sets of nonsmooth convex functions. We focus on adaptive learning rate optimization algorithms, which adapt step-sizes (referred to as learning rates in the machine learning field) to find optimal solutions quickly. We then propose a Riemannian stochastic fixed point optimization algorithm, which combines fixed point approximation methods on Riemannian manifolds with the adaptive learning rate optimization algorithms. We also give convergence analyses of the proposed algorithm for nonsmooth convex and smooth nonconvex optimization. The analysis results indicate that, with small constant step-sizes, the proposed algorithm approximates a solution to the problem. Consideration of the case in which step-size sequences are diminishing demonstrates that the proposed algorithm solves the problem with a guaranteed convergence rate. This paper also provides numerical comparisons that demonstrate the effectiveness of the proposed algorithms with formulas based on the adaptive learning rate optimization algorithms, such as Adam and AMSGrad.

Current Situation and Strategy Formulation of College Sports Psychology Teaching Following Adaptive Learning and Deep Learning Under Information Education.

This study aims to explore the current situation and strategy formulation of sports psychology teaching in colleges and universities following adaptive learning and deep learning under information education. The informatization in physical education, teaching methods, and teaching processes make psychological education more scientific and efficient. First, the relevant theories of adaptive learning and deep learning are introduced, and an adaptive learning analysis model is implemented. Second, based on the deep learning automatic encoder, college students’ sports psychology is investigated and the test results are predicted. Finally, the current situation and development strategy of physical education in colleges and universities are analyzed. The results show that when the learning rate is 1, 0.1, and 0.01, there is no significant change in the analysis factors of recall, ndcg, item_coverage, and sps. When the learning rate is 1, their analysis factors change obviously, and it is calculated that the optimal learning rate of the model is 1. And the difficulty of the recommended test questions by using the sports psychology teaching method based on adaptive learning and deep learning is relatively stable. The test questions include various language points of sports psychology. Compared with others methods, adaptive learning and deep learning can provide comprehensive test questions for sports psychology teaching. This study provides technical support for the reform of sports psychology teaching in colleges and universities and contributes to optimizing the information-based teaching mode.

SPECIFICS OF THE LEARNING ERROR DEPENDENCE OF MULTILAYERED NEURAL NETWORKS FROM THE ACTIVATION FUNCTION DURING THE PROCESS OF PRINTED DIGITS IDENTIFICATION

In this paper, we investigated the learning process during an identification of printed digits from the type of an activation function. The study of the activation function type and the number of iterations in the learning process of the neural system was carried out using the Fourier spectra analysis of the learning error function and branching diagrams. For this purpose, a program for the multilayer neural network was developed in the Python software, which involves setting the number of hidden layers as well as the number of neurons inside them and changes in the learning rate. The learning rate was considered as a constant, and its optimal value, where the best learning rate is observed, was determined. To analyze the learning rate effect on the educational process, we used a logistic function describing the frequency doubling process. It is shown that the learning error function is characterized by bifurcation processes leading to a chaotic state when η >0.8. The optimal learning rate value that determines the emergence of the doubling process of the local minima number is determined. It was found that the sigmoidal activation function (as compared to the activation functions ReLU and hyperbolic tangent) best satisfies the learning process of the three-layer neural network for recognizing digits, given an array of 4x7 zeros and ones. Compared to other activation functions, there is an insignificant change in the learning error during the transition from one digit to another. It is shown that an increase in the number of hidden layers does not lead to a sharp increase during the learning error. An increase in the learning iterations number is accompanied by the appearance of periodic dependences of the logistic function value of the learning rate, the period of which is a variable of the number of iterations and the learning rate. Using Fourier spectra of the error function from the learning rate value, it can be argued that an increase in the number of iterations leads to an increase in the number of harmonics, which eventually leads to the appearance of a chaotic state of the neural network. Keywords : Multilayered neural network, activation function, optimal learning rate, digital identification.

Development of Trees Management System Using Radial Basis Function Neural Network for Rain Forecast

Agriculture and farming are mainly dependent on weather especially in Malaysia as it received heavy rainfall throughout the years. An efficient crop or tree management system with a weather forecast needed for suitable planning of farming operation. Radial Basis Function Neural Network (RBFNN) algorithm was used in this study to predict rainfall and the main focus of this study is to analyze the factor that affects the performance of neural model. This study found that the model works better the more hidden nodes and the optimum learning rate is 0.01 with the RMSE 49% and the percentage accuracy is 57%. Besides that, it is found that the meteorology data also affect the model performance. Future research can be conducted to improve the rainfall forecast of this study and improve the tree management system.

Research on Intelligent Fault Diagnosis of Rolling Bearing Based on Adaptive Resource Allocation Deep Neural Network

The existing deep learning based bearing fault diagnosis models often adopts the default numeric value or manual adjustment for the learning rate of the optimizer, which leads to the low adaptive performance of the diagnosis model and can not adapt to the changeable working conditions of bearing fault diagnosis; Moreover, the trained model may not be used in other scenarios due to the difference of data distribution, and the learning rate of the optimizer has to be manually adjusted again. An adaptive resource allocation depth neural network (PDC-LR-HCNN) is proposed, which is composed of improved extended convolution, Lookahead-Radam (LR) optimizer and hyperband regulator. Firstly, LR optimizer is used to update parameters instead of Adam optimizer; Secondly, the superposition structure of expanded convolution is improved, and the convolution layers with expansion rates of 1, 3 and 5 are used for superposition, and an expanded convolution module based on the characteristics of human receptive field is proposed; Finally, the hyperband regulator is used for adaptive resource allocation to speed up the random search and dynamically find the optimal learning rate of the optimizer, which solves the disadvantage that manually adjusting the learning rate of the optimizer is time-consuming, laborious and not optimal. Through the experimental data sets of CWRU and XJTU-SY bearings, the proposed PDC-LR-HCNN neural network model is verified. The accuracy can reach more than 90%, and the classification accuracy convergence can be achieved within 10 iterations, which verifies the effectiveness of the model.

Multiscale regression on unknown manifolds

<abstract><p>We consider the regression problem of estimating functions on $ \mathbb{R}^D $ but supported on a $ d $-dimensional manifold $ \mathcal{M} ~~\subset \mathbb{R}^D $ with $ d \ll D $. Drawing ideas from multi-resolution analysis and nonlinear approximation, we construct low-dimensional coordinates on $ \mathcal{M} $ at multiple scales, and perform multiscale regression by local polynomial fitting. We propose a data-driven wavelet thresholding scheme that automatically adapts to the unknown regularity of the function, allowing for efficient estimation of functions exhibiting nonuniform regularity at different locations and scales. We analyze the generalization error of our method by proving finite sample bounds in high probability on rich classes of priors. Our estimator attains optimal learning rates (up to logarithmic factors) as if the function was defined on a known Euclidean domain of dimension $ d $, instead of an unknown manifold embedded in $ \mathbb{R}^D $. The implemented algorithm has quasilinear complexity in the sample size, with constants linear in $ D $ and exponential in $ d $. Our work therefore establishes a new framework for regression on low-dimensional sets embedded in high dimensions, with fast implementation and strong theoretical guarantees.</p></abstract>

Sentiment Analysis of Tweets on Prakerja Card using Convolutional Neural Network and Naive Bayes

The Indonesian government launched the Prakerja (pre-employment) card in the midst of the COVID-19 pandemic, andthe local citizens have voiced their opinions about this controversial program through social media such as Twitter. People’scomments on it can be useful information, and this research tries to analyze the sentiment regarding the Prakerja Card programusing the Convolutional Neural Network and Naive Bayes methods. The main task in this sentiment analysis is analyzing the dataand then classifying them into one of the following classes: positive, negative or neutral. Naive Bayes is an algorithm that is often usedin sentiment analysis research, and the results have been very good. Convolutional neural network (CNN) is a deep learning algorithmthat uses one or more layers commonly used for pattern recognition and image recognition. Having applied these methods, thisresearch found that the CNN model with the GlobalMaxPooling layer is the best model of the other two CNN models. Sentimentanalysis has the best accuracy of 78.5% on the CNN method, and NBC of 76.2% accuracy. The best accuracy result in K-fold withfive classes is 85.4% on the CNN model with a learning rate optimization of 0.00158. While the average accuracy on NBC only reached75.3%

Construction of College Students' Employment Quality Evaluation Model System under the Background of Digitalization.

In the era of knowledge economy, human resources are being valued by various countries and regions. The report of the 19th National Congress of the Communist Party of China pointed out that “talent is a strategic resource for realizing national rejuvenation and winning the initiative of international competition. We must adhere to the principle of the party's management of talents, gather talents from all over the world and use them, and accelerate the construction of a strong country with talents.” College students are an important part of talents, and their employment intentions directly affect employment behavior. With the development of education in our country, the enrollment quota of most colleges and universities in the country has gradually increased, and the number of graduates has also increased. Social and economic development has different needs for different professional and technical personnel, and the employment situation in different regions is uneven. Under the increasingly complex employment environment, college students have to face greater employment pressure and compete with each other in a narrower employment field. Therefore, it is necessary to conduct better employment guidance and employment quality evaluation for college students. Based on the improved algorithm of BP network, an artificial intelligence-based employment quality evaluation model is constructed. The design model is optimized by introducing a momentum variable factor, adjusting the learning rate and quasi-Newton method, and training and recommending each optimization model through the training data. The experimental results show that the iterations of the gradient descent algorithm and the additional momentum optimization algorithm are far more than 1000 times. Second, the optimal validation errors of the two algorithms are large and the model performance is poor. The quasi-Newton ring algorithm also has faster coordination speed, stronger stability, and better overall performance. The adaptive learning rate optimization algorithm is performed in these 4 algorithms. In terms of accuracy, the accuracy of the adaptive learning rate BP optimization algorithm is 76.4%, followed by the Newton algorithm and the additional momentum algorithm, and the gradient descent algorithm is the worst.

Effects of depth, width, and initialization: A convergence analysis of layer-wise training for deep linear neural networks

Deep neural networks have been used in various machine learning applications and achieved tremendous empirical successes. However, training deep neural networks is a challenging task. Many alternatives have been proposed in place of end-to-end back-propagation. Layer-wise training is one of them, which trains a single layer at a time, rather than trains the whole layers simultaneously. In this paper, we study a layer-wise training using a block coordinate gradient descent (BCGD) for deep linear networks. We establish a general convergence analysis of BCGD and found the optimal learning rate, which results in the fastest decrease in the loss. We identify the effects of depth, width, and initialization. When the orthogonal-like initialization is employed, we show that the width of intermediate layers plays no role in gradient-based training beyond a certain threshold. Besides, we found that the use of deep networks could drastically accelerate convergence when it is compared to those of a depth 1 network, even when the computational cost is considered. Numerical examples are provided to justify our theoretical findings and demonstrate the performance of layer-wise training by BCGD.

Learning Rate Optimization for Federated Learning Exploiting Over-the-Air Computation

Federated learning (FL) as a promising edge-learning framework can effectively address the latency and privacy issues by featuring distributed learning at the devices and model aggregation in the central server. In order to enable efficient wireless data aggregation, over-the-air computation (AirComp) has recently been proposed and attracted immediate attention. However, fading of wireless channels can produce aggregate distortions in an AirComp-based FL scheme. To combat this effect, the concept of dynamic learning rate (DLR) is proposed in this work. We begin our discussion by considering multiple-input-single-output (MISO) scenario, since the underlying optimization problem is convex and has closed-form solution. We then extend our studies to more general multiple-input-multiple-output (MIMO) case and an iterative method is derived. Extensive simulation results demonstrate the effectiveness of the proposed scheme in reducing the aggregate distortion and guaranteeing the testing accuracy using the MNIST and CIFAR10 datasets. In addition, we present the asymptotic analysis and give a near-optimal receive beamforming design solution in closed form, which is verified by numerical simulations.

Optimal learning with anisotropic Gaussian SVMs

This paper investigates the nonparametric regression problem using SVMs with anisotropic Gaussian RBF kernels. Under the assumption that the target functions are resided in certain anisotropic Besov spaces, we establish the almost optimal learning rates, more precisely, optimal up to some logarithmic factor, presented by the effective smoothness. By taking the effective smoothness into consideration, our almost optimal learning rates are faster than those obtained with the underlying RKHSs being certain anisotropic Sobolev spaces. Moreover, if the target function depends only on fewer dimensions, faster learning rates can be further achieved.

SPSO-DBN based compensation algorithm for lackness of electric energy metering in micro-grid

Accurate electric energy metering in micro-grid is one of the most urgent problems in the field of electric energy metering. In order to solve the problem that the lackness of electric energy metering can't be accurately calculated with the existing electric energy compensation algorithm when the electric energy metering device is completely faulty, this paper proposes an electric energy compensation algorithm that is limited in micro-grid and is based on standard particle swarm to optimize the deep belief network. Taking the historical electric energy data of each micro source metering point in micro-grid as an example, it is divided into the training set and the test set, the standard particle swarm optimization algorithm is used to determine the optimal learning layers and learning rate of deep belief network, and the electric energy algorithm model is established. The model verification results show that the proposed algorithm is nearly five times lower than the algorithm based on deep belief network solely. Further engineering case comparison shows that lackness of electric energy metering compensated by the algorithm studied in the paper is closer to the theoretical value, and the error is also less than the traditional average electricity algorithm. The real-time power curve is closer to the theoretical value, and has the higher accuracy.

Identification of Covid’19 Vaccinator by Deep Learning Approach Using Contactless Palmprints

The invention of the first vaccine has also raised several anti-vaccination views among people. Vaccine reluctance may be exacerbated by the growing reliance on social media, which is considered as a source of health information. During this COVID'19 scenario, the verification of non-vaccinators via the use of biometric characteristics has received greater attention, especially in areas such as vaccination monitoring and other emergency medical services, among other things. The traditional digital camera utilizes the middle-resolution images for commercial applications in a regulated or contact-based environment with user participation, while the latter uses high-resolution latent palmprints. This research study attempts to utilize convolutional neural networks (CNN) for the first time to perform contactless recognition. To identify the COVID '19 vaccine using the CNN technique, this research work has used the contactless palmprint method. Further, this research study utilizes the PalmNet structure of convolutional neural network to resolve the issue. First, the ROI region of the palmprint was extracted from the input picture based on the geometric form of the print. After image registration, the ROI region is sent into a convolutional neural network as an input. The softmax activation function is then used to train the network so that it can choose the optimal learning rate and super parameters for the given learning scenario. The neural networks of the deep learning platform were then compared and summarized.

ZipLine: an optimized algorithm for the elastic bulk synchronous parallel model

The bulk synchronous parallel (BSP) is a celebrated synchronization model for general-purpose parallel computing that has successfully been employed for distributed training of deep learning models. A shortcoming of the BSP is that it requires workers to wait for the straggler at every iteration. Therefore, employing BSP increases the waiting time of the faster workers of a cluster and results in an overall prolonged training time. To ameliorate this shortcoming of BSP, we propose ElasticBSP, a model that aims to relax its strict synchronization requirement with an elastic synchronization by allowing delayed synchronization to minimize the waiting time. ElasticBSP offers more flexibility and adaptability during the training phase, without sacrificing the accuracy of the trained model. ElasticBSP is realized by the algorithm named ZipLine, which consists of two phases. First, it estimates for each worker the end time points of its future iterations at run time, and then a one-pass algorithm over the estimated time points of all workers is employed to fast compute an optimal future time point for synchronization. We provide theoretical results about the correctness and performance of the ZipLine algorithm. Furthermore, we propose algorithmic and implementation optimizations of ZipLine, namely ZipLineOpt and ZipLineOptBS, which reduce the time complexity of ZipLine to linearithmic time. A thorough experimental evaluation demonstrates that our proposed ElasticBSP model, materialized by the proposed optimized ZipLine variants, converges faster and to a higher accuracy than the predominant BSP. The focus of the paper is on optimizing the synchronization scheduling over a parameter server architecture. It is orthogonal to other types of optimizations, such as the learning rate optimization.

Deep learning approach for Sentinel-1 surface water mapping leveraging Google Earth Engine

Satellite remote sensing plays an important role in mapping the location and extent of surface water. A variety of approaches are available for mapping surface water, but deep learning approaches are not commonplace as they are ‘data hungry’ and require large amounts of computational resources. However, with the availability of various satellite sensors and rapid development in cloud computing, the remote sensing scientific community is adapting modern deep learning approaches. The new integration of cloud-based Google AI platform and Google Earth Engine enables users to deploy calculations at scale. In this paper, we investigate two methods of automatic data labeling: 1. the Joint Research Centre (JRC) surface water maps; 2. an Edge-Otsu dynamic threshold approach. We deployed a U-Net convolutional neural network to map surface water from Sentinel-1 Synthetic Aperture Radar (SAR) data and tested the model performance using different hyperparameter tuning combinations to identify the optimal learning rate and loss function. The performance was then evaluated using an independent validation data set. We tested 12 models overall and found that the models utilizing the JRC data labels showed a better model performance, with F1-scores ranging from 0.972 to 0.986 for the training test and validation efforts. Additionally, an independently sampled high-resolution data set was used to further evaluate model performance. From this independent validation effort we observed models leveraging JRC data labels produced F1-Scores ranging from 0.9130.922. A pairwise comparison of models, through varying input data, learning rates, and loss functions constituents, revealed the JRC Adjusted Binary Cross Entropy Dice model to be statistically different than the 66 other model combinations and displayed the highest relative evaluations metrics including accuracy, precision score, Cohen Kappa coefficient, and F1-score. These results are in the same range as many of the conventional methods. We observed that the integration of Google AI Platform into Google Earth Engine can be a powerful tool to deploy deep-learning algorithms at scale and that automatic data labeling can be an effective strategy in the development of deep-learning models, however independent data validation remains an important step in model evaluation.

Engineering Calculation And Algorithm Of Adaptation Of Parameters Of A Neuro-Fuzzy Controller

As part of the study, a control scheme with the adaptation of the coefficients of the neuron-fuzzy regulator implemented. The area difference method used as a training method for the network. It improved by adding a rule base, which allows choosing the optimal learning rate for individual neurons of the neural network. The neural network controller applied as a superstructure of the PID controller in the process control scheme. The dynamic object can function in different modes. This technological process operates in different modes in terms of loading and temperature setpoints. Because of experiments, the power consumption and the amount of time required maintaining the same absorption process, using a conventional PID controller and a neural-network controller evaluated. It concluded that the neuro-fuzzy controller with a superstructure reduced the transient time by 19%.

Enhancement of multilayer perceptron model training accuracy through the optimization of hyperparameters: a case study of the quality prediction of injection-molded parts

Injection molding has been broadly used in the mass production of plastic parts and must meet the requirements of efficiency and quality consistency. Machine learning can effectively predict the quality of injection-molded part. However, the performance of machine learning models largely depends on the accuracy of the training. Hyperparameters such as activation functions, momentum, and learning rate are crucial to the accuracy and efficiency of model training. This research aims to analyze the influence of hyperparameters on testing accuracy, explore the corresponding optimal learning rate, and provide the optimal training model for predicting the quality of injection-molded parts. In this study, stochastic gradient descent (SGD) and stochastic gradient descent with momentum (SGDM) are used to optimize the artificial neural network model. Through optimization of these training model hyperparameters, the width testing accuracy of the injection product is improved. The experimental results indicate that in the absence of momentum effects, all five activation functions can achieve more than 90% of the training accuracy with a learning rate of 0.1. Moreover, when optimized with the SGD, the learning rate of the Sigmoid activation function is 0.1, and the testing accuracy reaches 95.8%. Although momentum has the least influence on accuracy, it affects the convergence speed of the Sigmoid function, which reduces the number of required learning iterations (82.4% reduction rate). In summary, optimizing hyperparameter settings can improve the accuracy of model testing and markedly reduce training time.