Choice Of Learning Rate Research Articles

Non-ergodic linear convergence property of the delayed gradient descent under the strongly convexity and the Polyak–Łojasiewicz condition

In this work, we establish the linear convergence estimate for the gradient descent involving the delay [Formula: see text] when the cost function is [Formula: see text]-strongly convex and [Formula: see text]-smooth. This result improves upon the well-known estimates in [Y. Arjevani, O. Shamir and N. Srebro, A tight convergence analysis for stochastic gradient descent with delayed updates, Proc. Mach. Learn. Res. 117 (2020) 111–132; S. U. Stich and S. P. Karimireddy, The error-feedback framework: Better rates for SGD with delayed gradients and compressed updates, J. Mach. Learn. Res. 21(1) (2020) 9613–9648] in the sense that it is non-ergodic and is still established in spite of weaker constraint of cost function. Also, the range of learning rate [Formula: see text] can be extended from [Formula: see text] to [Formula: see text] for [Formula: see text] and [Formula: see text] for [Formula: see text], where [Formula: see text] is the Lipschitz continuity constant of the gradient of cost function. In a further research, we show the linear convergence of cost function under the Polyak–Łojasiewicz[Formula: see text](PL) condition, for which the available choice of learning rate is further improved as [Formula: see text] for the large delay [Formula: see text]. The framework of the proof for this result is also extended to the stochastic gradient descent with time-varying delay under the PL condition. Finally, some numerical experiments are provided in order to confirm the reliability of the analyzed results.

Selecting and Composing Learning Rate Policies for Deep Neural Networks

The choice of learning rate (LR) functions and policies has evolved from a simple fixed LR to the decaying LR and the cyclic LR, aiming to improve the accuracy and reduce the training time of Deep Neural Networks (DNNs). This article presents a systematic approach to selecting and composing an LR policy for effective DNN training to meet desired target accuracy and reduce training time within the pre-defined training iterations. It makes three original contributions. First, we develop an LR tuning mechanism for auto-verification of a given LR policy with respect to the desired accuracy goal under the pre-defined training time constraint. Second, we develop an LR policy recommendation system (LRBench) to select and compose good LR policies from the same and/or different LR functions through dynamic tuning, and avoid bad choices, for a given learning task, DNN model, and dataset. Third, we extend LRBench by supporting different DNN optimizers and show the significant mutual impact of different LR policies and different optimizers. Evaluated using popular benchmark datasets and different DNN models (LeNet, CNN3, ResNet), we show that our approach can effectively deliver high DNN test accuracy, outperform the existing recommended default LR policies, and reduce the DNN training time by 1.6-6.7× to meet a targeted model accuracy.

Mini-Batch Metropolis–Hastings With Reversible SGLD Proposal

Traditional Markov chain Monte Carlo (MCMC) algorithms are computationally intensive and do not scale well to large data. In particular, the Metropolis–Hastings (MH) algorithm requires passing over the entire dataset to evaluate the likelihood ratio in each iteration. We propose a general framework for performing MH-MCMC using mini-batches of the whole dataset and show that this gives rise to approximately a tempered stationary distribution. We prove that the algorithm preserves the modes of the original target distribution and derive an error bound on the approximation with mild assumptions on the likelihood. To further extend the utility of the algorithm to high-dimensional settings, we construct a proposal with forward and reverse moves using stochastic gradient and show that the construction leads to reasonable acceptance probabilities. We demonstrate the performance of our algorithm in both low dimensional models and high dimensional neural network applications. Particularly in the latter case, compared to popular optimization methods, our method is more robust to the choice of learning rate and improves testing accuracy. Supplementary materials for this article are available online.

Technology learning and diffusion at the global and local scales: A modeling exercise in the REMIND model

Empirical studies on technology advancement and regional diffusion indicate that technology learning is a multi-scale process driven by factors such as globally traded equipment and local accumulation of experience. This understanding should be incorporated in energy-economy models to improve the model representation and policy recommendations. The present study augments the large-scale, integrated assessment model REMIND for this purpose. To answer the research question on how multi-level learning affects technology diffusion and the regional costs of mitigation policies, alternative variants of multi-scale learning are implemented and calibrated to one set of regional-specific cost data. The results show that purely local learning leads to similar technology diffusion patterns as fully global learning, since the learning rates are set equal, and all regions are calibrated to their cost levels and specific capacity. Relative to these two, the combination of global and local learning leads to slower deployment of learning technologies and increases the mitigation cost if the cost disparity persists across regions, e.g. due to incomplete spillover. Our modelling exercise suggests that the choice of learning rates at different levels matter and calls for better data quality.

Similarity learning for person reidentification using projected gradient method

The problem of learning a similarity measure for cross-view person identification in a disjointed camera network is addressed. Our learning framework is based on the projected gradient approach and is suitable for large-scale applications. The hinge loss with the triplet constraints is used as the objective function. Contrary to other approaches where a dual formulation is used, we propose to optimize the objective function in the primal form, with the help of a minibatch gradient descent algorithm. The choice of learning rate and its schedule are nontrivial tasks in such optimization schemes. We have studied empirically the effect of using different strategies for learning rate schedules. Our experimental study includes conventional strategies such as fixed or diminishing learning rates and recently developed adaptive gradient methods such as root-mean-square propagation (RMSProp) for learning rate schedule. The experimental results are presented on three benchmark datasets of person reidentification, namely VIPeR, CUHK Campus, and CUHK03. Our experimental results demonstrate that the RMSProp is well suited for the proposed learning framework. A comparison with recent methods in the domain presents the justification of the proposed approach.

Research on Medical Data Feature Extraction and Intelligent Recognition Technology Based on Convolutional Neural Network

In order to mine information from medical health data and develop intelligent application-related issues, the multi-modal medical health data feature representation learning related content was studied, and several feature learning models were proposed for disease risk assessment. In the aspect of medical text feature learning, a medical text feature learning model based on convolutional neural network is proposed. The convolutional neural network text analysis technology is applied to the disease risk assessment application. The medical data feature representation adopts the deep learning method. The learning and extraction of different disease characteristics use the same method to realize the versatility of the model. A simple preprocessing of the experimental data samples, including its power frequency denoising and lead convolution regularization, constructs a convolutional neural network for medical data feature advancement and intelligent recognition. On the basis of it, several sets of experiments were carried out to discuss the influence of the convolution kernel and the choice of learning rate on the experimental results. In addition, comparative experiments with support vector machine, BP neural network and RBF neural network are carried out. The results show that the convolutional neural network used in this paper shows obvious advantages in recognition rate and training speed compared with other methods. In the aspect of time series data feature learning, a multi-channel convolutional self-encoding neural network is proposed. Analyze the connection between fatigue and emotional abnormalities and define the concept of emotional fatigue. The proposed multi-channel convolutional neural network is used to learn the data features, and the convolutional self-encoding neural network is used to learn the facial image data features. These two characteristics and the collected physiological data are combined to perform emotional fatigue detection. An emotional fatigue detection demonstration platform for multi-modal data feature fusion is established to realize data acquisition, emotional fatigue detection and emotional feedback. The experimental results verify the validity, versatility and stability of the model.

Faster Convergent Artificial Neural Networks

Proposed in this paper is a novel fast-convergence algorithm applied to neural networks (ANNs) with a learning rate based on the eigenvalues of the associated Hessian matrix of the input data. That is, the learning rate applied to the backpropagation algorithm changes dynamically with the input data used for training. The best choice of learning rate to converge to an accurate value quickly is derived. This newly proposed fast-convergence algorithm is applied to a traditional multilayer ANN architecture with feed-forward and backpropagation techniques. The proposed strategy is applied to various functions learned by the ANN through training. Learning curves obtained using calculated learning rates according to the novel method proposed are compared to learning curves utilizing an arbitrary learning rate to demonstrate the usefulness of this novel technique. This study shows that convergence to accurate values can be achieved much more quickly (a reduction in iterations by a factor of hundred) using the techniques proposed here. This approach is illustrated in this research work with derivations and pertinent examples to illustrate the method and learning curves obtained.

Online learning algorithms for principal component analysis applied on single-lead ECGs

This article evaluates several adaptive approaches to solve the principal component analysis (PCA) problem applied on single-lead ECGs. Recent studies have shown that the principal components can indicate morphologically or environmentally induced changes in the ECG signal and can be used to extract other vital information such as respiratory activity. Special interest is focused on the convergence behavior of the selected gradient algorithms, which is a major criterion for the usability of the gained results. As the right choice of learning rates is very data dependant and subject to movement artifacts, a new measurement system was designed, which uses acceleration data to improve the performance of the online algorithms. As the results of PCA seem very promising, we propose to apply a single-channel independent component analysis (SCICA) as a second step, which is investigated in this paper as well.