Adaptive Gradient Method Research Articles

A random matrix theory approach to damping in deep learning

We conjecture that the inherent difference in generalisation between adaptive and non-adaptive gradient methods in deep learning stems from the increased estimation noise in the flattest directions of the true loss surface. We demonstrate that typical schedules used for adaptive methods (with low numerical stability or damping constants) serve to bias relative movement towards flat directions relative to sharp directions, effectively amplifying the noise-to-signal ratio and harming generalisation. We further demonstrate that the numerical damping constant used in these methods can be decomposed into a learning rate reduction and linear shrinkage of the estimated curvature matrix. We then demonstrate significant generalisation improvements by increasing the shrinkage coefficient, closing the generalisation gap entirely in both logistic regression and several deep neural network experiments. Extending this line further, we develop a novel random matrix theory based damping learner for second order optimisers inspired by linear shrinkage estimation. We experimentally demonstrate our learner to be very insensitive to the initialised value and to allow for extremely fast convergence in conjunction with continued stable training and competitive generalisation. We also find that our derived method works well with adaptive gradient methods such as Adam.

An Adaptive Gradient Method with Energy and Momentum

We introduce a novel algorithm for gradient-based optimization of stochastic objective functions. The method may be seen as a variant of SGD with momentum equipped with an adaptive learning rate automatically adjusted by an 'energy' variable. The method is simple to implement, computationally efficient, and well suited for large-scale machine learning problems. The method exhibits unconditional energy stability for any size of the base learning rate. We provide a regret bound on the convergence rate under the online convex optimization framework. We also establish the energy-dependent convergence rate of the algorithm to a stationary point in the stochastic non-convex setting. In addition, a sufficient condition is provided to guarantee a positive lower threshold for the energy variable. Our experiments demonstrate that the algorithm converges fast while generalizing better than or as well as SGD with momentum in training deep neural networks, and compares also favorably to Adam.

Efficient learning rate adaptation based on hierarchical optimization approach

This paper proposes a new hierarchical approach to learning rate adaptation in gradient methods, called learning rate optimization (LRO). LRO formulates the learning rate adaption problem as a hierarchical optimization problem that minimizes the loss function with respect to the learning rate for current model parameters and gradients. Then, LRO optimizes the learning rate based on the alternating direction method of multipliers (ADMM). In the process of this learning rate optimization, LRO does not require any second-order information and probabilistic model, so it is highly efficient. Furthermore, LRO does not require any additional hyperparameters when compared to the vanilla gradient method with the simple exponential learning rate decay. In the experiments, we integrated LRO with vanilla SGD and Adam. Then, we compared their optimization performance with the state-of-the-art learning rate adaptation methods and also the most commonly-used adaptive gradient methods. The SGD and Adam with LRO outperformed all the competitors on the benchmark datasets in image classification tasks.

Adaptive primal-dual stochastic gradient method for expectation-constrained convex stochastic programs

Stochastic gradient methods (SGMs) have been widely used for solving stochastic optimization problems. A majority of existing works assume no constraints or easy-to-project constraints. In this paper, we consider convex stochastic optimization problems with expectation constraints. For these problems, it is often extremely expensive to perform projection onto the feasible set. Several SGMs in the literature can be applied to solve the expectation-constrained stochastic problems. We propose a novel primal-dual type SGM based on the Lagrangian function. Different from existing methods, our method incorporates an adaptiveness technique to speed up convergence. At each iteration, our method inquires an unbiased stochastic subgradient of the Lagrangian function, and then it renews the primal variables by an adaptive-SGM update and the dual variables by a vanilla-SGM update. We show that the proposed method has a convergence rate of $$O(1/\sqrt{k})$$ in terms of the objective error and the constraint violation. Although the convergence rate is the same as those of existing SGMs, we observe its significantly faster convergence than an existing non-adaptive primal-dual SGM and a primal SGM on solving the Neyman–Pearson classification and quadratically constrained quadratic programs. Furthermore, we modify the proposed method to solve convex–concave stochastic minimax problems, for which we perform adaptive-SGM updates to both primal and dual variables. A convergence rate of $$O(1/\sqrt{k})$$ is also established to the modified method for solving minimax problems in terms of primal-dual gap. Our code has been released at https://github.com/RPI-OPT/APriD .

Calibrating the adaptive learning rate to improve convergence of ADAM

Adaptive gradient methods (AGMs) have been widely used to optimize nonconvex problems in the deep learning area. We identify two aspects of AGMs that can be further improved. First, we observe that the adaptive learning rate (A-LR) used by AGMs varies significantly across the dimensions of the optimization problem and over epochs, which we call anisotropic scale of the A-LR. It can slow down the convergence and make the algorithm trap into a sharp local minimizer. Actually, all existing modified AGMs represent efforts in revising the A-LR. Second, we theoretically prove that the convergence rate of AGMs depends on its hyper-parameter used in the A-LR formula, such as the ∊ used in ADAM, which has not been examined previously. We then propose new AGMs that calibrate the A-LR with an activation function, such as, the softplus function. Particularly, the Sadam and SAMSGrad methods are two instances of our method. We further prove that SAMSGrad enjoys a better convergence speed than the AMSGrad method under separate conditions including the nonconvex, non-strongly convex, and Polyak-Łojasiewicz conditions. Empirical studies are used to demonstrate the anisotropic A-LR issue and show that the proposed methods outperform existing AGMs and generalize better in multiple deep learning tasks.

Bistatic SAR Clutter-Ridge Matched STAP Method for Nonstationary Clutter Suppression

Clutter suppression is a challenging task in synthetic aperture radar-ground moving target indication (SAR-GMTI). In general, sufficient secondary samples are not easily acquired due to the nonstationary and nonhomogeneous characteristics of bistatic SAR (BiSAR) clutter, resulting in worse clutter suppression results. Recently, space–time adaptive processing based on sparse recovery (SR-STAP) has been developed since its better clutter suppression performance with less samples. However, since the off-grid problem in space–time domain caused by BiSAR’s separate configuration, existing SR-STAP would suffer from severe performance degradation. To address this problem, a clutter-ridge matched STAP (CRM-STAP) method for BiSAR nonstationary clutter suppression is proposed. First, clutter distribution modeling with arbitrary BiSAR configuration is applied to accurately obtain the clutter ridge in space–time domain. Then, keystone transform and time-division processing are applied to correct range cell migration and eliminate Doppler frequency migration, respectively. Next, to solve the off-grid problem, the CRM dictionary is reconstructed via adaptive gradient method, which is established along the direction of clutter ridge and its orthogonal direction. Then, with the constructed CRM dictionary, the clutter covariance matrix (CCM) estimation process is transformed to a multimeasured vector optimization problem, and it can be directly solved by the sparse Bayesian learning algorithm. Finally, based on the estimated CCM, the CRM-STAP filter is built to suppress the nonstationary clutter effectively. Compared with the existing STAP and SR-STAP methods, this method can avoid the performance degradation in clutter suppression caused by the off-grid problem and overcomes the strong nonstationary problem of BiSAR clutter in heterogeneous environments. In October 2020, we have successfully carried out the world’s first airborne BiSAR-GMTI experiment, and the experimental results are given to verify the effectiveness of this method.

DAdam: A Consensus-Based Distributed Adaptive Gradient Method for Online Optimization

Adaptive optimization methods, such as <sc xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">AdaGrad</small> , <sc xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">RMSProp</small> , and <sc xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">Adam</small> , are widely used in solving large-scale machine learning problems. A number of schemes have been proposed in the literature aiming at parallelizing them, based on communications between peripheral nodes with a central node, but incur high communications cost. To address this issue, we develop a novel consensus-based distributed adaptive moment estimation method ( <sc xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">DAdam</small> ) for online optimization over a decentralized network that enables data parallelization, as well as decentralized computation. The method is particularly useful, since it can accommodate settings where access only to local data is permitted. Further, as established theoretically in this work, it can outperform centralized adaptive algorithms, for certain classes of loss functions used in machine learning applications. We analyze the convergence properties of the proposed algorithm and provide a regret bound on the convergence rate of adaptive moment estimation methods in both online convex and non-convex settings. Empirical results demonstrate that <sc xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">DAdam</small> exhibits also good performance in practice and compares favorably to competing online optimization methods.

Energy saving of buildings for reducing carbon dioxide emissions using novel dendrite net integrated adaptive mean square gradient

Artificial neural networks have been widely applied in construction industries. Due to dendrites of biological neurons participating in the pre-calculation of neural networks, the structure of the traditional artificial neural network needs to be adjusted subjectively. Thus, a novel dendrite net based on the adaptive mean square gradient is proposed in this paper. The energy consumption of buildings is predicted and analyzed by the proposed method to cut the carbon dioxide emissions. The adaptive mean square gradient method is used to update the weight matrix of the dendrite net method, which can avoid errors caused by selecting hidden layer nodes to improve the prediction accuracy of the proposed method. Finally, the proposed method is used to energy saving and emission reducing of the construction industry. Compared with other methods, the experimental results show the availability of the proposed method. Through predicting the heating and cooling loads based on the proposed method, the construction plan is adjusted to decrease the energy consumption of buildings. Moreover, the appliances energy consumption is predicted and analyzed by the proposed method to improve energy efficiency and cut carbon dioxide emissions.

Quantized Adam with Error Feedback

In this article, we present a distributed variant of an adaptive stochastic gradient method for training deep neural networks in the parameter-server model. To reduce the communication cost among the workers and server, we incorporate two types of quantization schemes, i.e., gradient quantization and weight quantization, into the proposed distributed Adam. In addition, to reduce the bias introduced by quantization operations, we propose an error-feedback technique to compensate for the quantized gradient. Theoretically, in the stochastic nonconvex setting, we show that the distributed adaptive gradient method with gradient quantization and error feedback converges to the first-order stationary point, and that the distributed adaptive gradient method with weight quantization and error feedback converges to the point related to the quantized level under both the single-worker and multi-worker modes. Last, we apply the proposed distributed adaptive gradient methods to train deep neural networks. Experimental results demonstrate the efficacy of our methods.

ACMo: Angle-Calibrated Moment Methods for Stochastic Optimization

Stochastic gradient descent (SGD) is a widely used method for its outstanding generalization ability and simplicity. Adaptive gradient methods have been proposed to further accelerate the optimization process. In this paper, we revisit existing adaptive gradient optimization methods with a new interpretation. Such new perspective leads to a refreshed understanding of the roles of second moments in stochastic optimization. Based on this, we propose Angle-Calibration Moment method (ACMo), a novel stochastic optimization method. It enjoys the benefits of second moments with only first moment updates. Theoretical analysis shows that ACMo is able to achieve the same convergence rate as mainstream adaptive methods. Experiments on a variety of CV and NLP tasks demonstrate that ACMo has a comparable convergence to state-of-the-art Adam-type optimizers, and even a better generalization performance in most cases. The code is available at https://github.com/Xunpeng746/ACMo.

Adaptive Gradient Methods for Constrained Convex Optimization and Variational Inequalities

We provide new adaptive first-order methods for constrained convex optimization. Our main algorithms AdaACSA and AdaAGD+ are accelerated methods, which are universal in the sense that they achieve nearly-optimal convergence rates for both smooth and non-smooth functions, even when they only have access to stochastic gradients. In addition, they do not require any prior knowledge on how the objective function is parametrized, since they automatically adjust their per-coordinate learning rate. These can be seen as truly accelerated Adagrad methods for constrained optimization. We complement them with a simpler algorithm AdaGrad+ which enjoys the same features, and achieves the standard non-accelerated convergence rate. We also present a set of new results involving adaptive methods for unconstrained optimization and variational inequalities arising from monotone operators.

Joint Estimation of Target Parameters and System Deviations in MIMO Radar With Widely Separated Antennas on Moving Platforms

A multiple-input multiple-output (MIMO) radarwith widely separated antennas on moving platforms suffers from the effects of platform deviations on the target parameter estimation since the trajectories of the moving platforms are sensitive to environmental factors, such as strong wind. This article addresses the joint estimation of multiple target positions and velocities as well as radar system deviations to minimize the impact of platform deviations. The proposed algorithm can also be regarded as a self-calibration technique. First, the grid search dimensions for parameter estimation are reduced via a generalized maximum likelihood (GML) algorithm. Second, the adaptive gradient (AdaGrad) method is used to implement the GML estimation for multitarget echo delays and Doppler shifts. Finally, to address the nonlinear estimation problem of interest, the iterative least squares method is used to estimate the multiple target positions and velocities as well as radar system deviations based on the estimated delays and Doppler shifts. Prior information, such as target positions or the probability distribution of the echo coefficient, is not needed in the proposed method. The parameter identifiability is also derived in this article. Numerical simulations show that compared to a method without the estimation of system deviations, the proposed method is more efficient with respect to the derived Cramér–Rao bound.

A modified Mask region‐based convolutional neural network approach for the automated detection of archaeological sites on high‐resolution light detection and ranging‐derived digital elevation models in the North German Lowland

AbstractDue to complicated backgrounds and unclear target orientation, automated object detection is difficult in the field of archaeology. Most of the current convolutional neural network (CNN) object‐oriented detection techniques are based on a faster region‐based CNN (R‐CNN) and other one‐stage detectors that often lack adequate processing speeds and detection accuracies. Recently, the two‐stage detector Mask R‐CNN technique achieved impressive results in object detection and instance segmentation problems and was successfully applied in the analysis of archaeological airborne laser scanning (ALS) data. In this study, we outline a modified Mask R‐CNN technique that reliably and efficiently detects relict charcoal hearth (RCH) sites on light detection and ranging (LiDAR) data‐based digital elevation models (DEMs). Using image augmentation and image preprocessing steps combined with the deep learning‐based adaptive gradient method with a dynamic bound on the learning rate (AdaBound) optimization technique, we could improve the model's accuracy and significantly reduce its training time. We use DEMs based on high‐resolution LiDAR data and the visualization for archaeological topography (VAT) technique that give images with a very strong contrast of the terrain and the outline of the sites of interest in the North German Lowland. Therefore, the model can identify RCH sites with an average recall of 83% and an average precision of 87%. Techniques such as the modified Mask R‐CNN method outlined here will help to greatly improve our knowledge about archaeological site densities in the realm of historical charcoal production and past human‐landscape interactions. This method provides an accurate, time‐efficient and bias‐free large‐scale site mapping option not only for the North German Lowland but potentially for other landscapes as well.

Robustness of Adaptive Neural Network Optimization Under Training Noise

Adaptive gradient methods such as adaptive moment estimation (Adam), RMSProp, and adaptive gradient (AdaGrad) use the temporal history of the gradient updates to improve the speed of convergence and reduce reliance on manual learning rate tuning, making them a popular choice for off-the-shelf Deep Neural Network (DNN) optimizers. In this article, we study the robustness of neural network optimizers in the presence of training perturbations. We show that popular adaptive optimization methods exhibit poor generalization while learning from noisy training data, compared to vanilla Stochastic Gradient Descent (SGD) and its variants, which manifest better implicit regularization properties. We construct an illustrative example of a family of two-class linearly separable toy-data such that models trained under noise using adaptive optimizers show only 52% test accuracy (random classifier), whereas SGD-based methods can achieve 100% test accuracy. We strengthen our hypothesis by empirical analysis using Convolutional Neural Networks (CNNs) on publicly available image datasets. For this purpose, our method trains neural network models with various optimizers on noisy training data, and we compute test accuracy on clean test data. Our results further highlight the robustness of SGD optimization against such noisy training data compared to its adaptive counterparts. Based on the results, our paper suggests a reconsideration of the extensive use of adaptive gradient methods for neural network optimization, especially when the training data is noisy.

Devanagari Handwritten Character Recognition using fine-tuned Deep Convolutional Neural Network on trivial dataset

In order to rapidly build an automatic and precise system for image recognition and categorization, deep learning is a vital technology. Handwritten character classification also gaining more attention due to its major contribution in automation and specially to develop applications for helping visually impaired people. Here, the proposed work highlighting on fine-tuning approach and analysis of state-of-the-art Deep Convolutional Neural Network (DCNN) designed for Devanagari Handwritten characters classification. A new Devanagari handwritten characters dataset is generated which is publicly available. Datasets consist of total 5800 isolated images of 58 unique character classes: 12 vowels, 36 consonants and 10 numerals. In addition to this database, a two-stage VGG16 deep learning model is implemented to recognize those characters using two advanced adaptive gradient methods. A two-stage approach of deep learning is developed to enhance overall success of the proposed Devanagari Handwritten Character Recognition System (DHCRS). The first model achieves 94.84% testing accuracy with training loss of 0.18 on new dataset. Moreover, the second fine-tuned model requires very fewer trainable parameters and notably less training time to achieve state-of-the-art performance on a very small dataset. It achieves 96.55% testing accuracy with training loss of 0.12. We also tested the proposed model on four different benchmark datasets of isolated characters as well as digits of Indic scripts. For all the datasets tested, we achieved the promising results.

Global and local optimization in identification of parabolic systems

Abstract The problem of identification of coefficients and initial conditions for a boundary value problem for parabolic equations that reduces to a minimization problem of a misfit function is investigated. Firstly, the tensor train decomposition approach is presented as a global convergence algorithm. The idea of the proposed method is to extract the tensor structure of the optimized functional and use it for multidimensional optimization problems. Secondly, for the refinement of the unknown parameters, three local optimization approaches are implemented and compared: Nelder–Mead simplex method, gradient method of minimum errors, adaptive gradient method. For gradient methods, the evident formula for the continuous gradient of the misfit function is obtained. The identification problem for the diffusive logistic mathematical model which can be applied to social sciences (online social networks), economy (spatial Solow model) and epidemiology (coronavirus COVID-19, HIV, etc.) is considered. The numerical results for information propagation in online social network are presented and discussed.

A Heuristic Adaptive Fast Gradient Method in Stochastic Optimization Problems

In this paper, we present a heuristic adaptive fast gradient method. We show that in practice our method has a better convergence rate than popular today optimization methods. Moreover, we justify our method and point out some problems that do not allow us to obtain theoretical estimates.

Numerical and experimental study on coherent beam combining using an improved stochastic parallel gradient descent algorithm

The adaptive gradient (AdaGrad) method is an optimization algorithm widely used in the field of artificial intelligence. An adaptive gradient stochastic parallel gradient descent (SPGD) algorithm (AdaSPGD algorithm), combining an AdaGrad algorithm with an SPGD algorithm, is innovatively introduced and implemented in coherent beam synthesis. The performance of a coherent beam combination system utilizing the AdaSPGD method is validated by numerical simulation of straightening static phase aberrations. The results of the simulations indicate that the AdaSPGD algorithm not only can effectively solve the trouble of difficulty in selecting the gain coefficient in the actual beam combining system, but also can accelerate the convergence of the phase control algorithm. Furthermore, the effectiveness of the proposed algorithm is demonstrated by means of the experimental investigation on coherent beam synthesis of a two-channel fiber array. The AdaSPGD algorithm is a satisfactory modification of the conventional SPGD algorithm.

Optimization for Deep Learning: An Overview

Optimization is a critical component in deep learning. We think optimization for neural networks is an interesting topic for theoretical research due to various reasons. First, its tractability despite non-convexity is an intriguing question and may greatly expand our understanding of tractable problems. Second, classical optimization theory is far from enough to explain many phenomena. Therefore, we would like to understand the challenges and opportunities from a theoretical perspective and review the existing research in this field. First, we discuss the issue of gradient explosion/vanishing and the more general issue of undesirable spectrum and then discuss practical solutions including careful initialization, normalization methods and skip connections. Second, we review generic optimization methods used in training neural networks, such as stochastic gradient descent and adaptive gradient methods, and existing theoretical results. Third, we review existing research on the global issues of neural network training, including results on global landscape, mode connectivity, lottery ticket hypothesis and neural tangent kernel.

A Comparison of Optimization Algorithms for Deep Learning

In recent years, we have witnessed the rise of deep learning. Deep neural networks have proved their success in many areas. However, the optimization of these networks has become more difficult as neural networks going deeper and datasets becoming bigger. Therefore, more advanced optimization algorithms have been proposed over the past years. In this study, widely used optimization algorithms for deep learning are examined in detail. To this end, these algorithms called adaptive gradient methods are implemented for both supervised and unsupervised tasks. The behavior of the algorithms during training and results on four image datasets, namely, MNIST, CIFAR-10, Kaggle Flowers and Labeled Faces in the Wild are compared by pointing out their differences against basic optimization algorithms.