Adaptive Gradient Method Research Articles

A distributed adaptive policy gradient method based on momentum for multi-agent reinforcement learning

Policy Gradient (PG) method is one of the most popular algorithms in Reinforcement Learning (RL). However, distributed adaptive variants of PG are rarely studied in multi-agent. For this reason, this paper proposes a distributed adaptive policy gradient algorithm (IS-DAPGM) incorporated with Adam-type updates and importance sampling technique. Furthermore, we also establish the theoretical convergence rate of O(1/T)\\documentclass[12pt]{minimal} \\usepackage{amsmath} \\usepackage{wasysym} \\usepackage{amsfonts} \\usepackage{amssymb} \\usepackage{amsbsy} \\usepackage{mathrsfs} \\usepackage{upgreek} \\setlength{\\oddsidemargin}{-69pt} \\begin{document}$$\\mathcal {O}(1/\\sqrt{T})$$\\end{document}, where T represents the number of iterations, it can match the convergence rate of the state-of-the-art centralized policy gradient methods. In addition, many experiments are conducted in a multi-agent environment, which is a modification on the basis of Particle world environment. By comparing with some other distributed PG methods and changing the number of agents, we verify the performance of IS-DAPGM is more efficient than the existing methods.

Communication-Efficient Zeroth-Order Adaptive Optimization for Federated Learning

Federated learning has become a prevalent distributed training paradigm, in which local devices collaboratively train learning models without exchanging local data. One of the most dominant frameworks of federated learning (FL) is FedAvg, since it is efficient and simple to implement; here, the first-order information is generally utilized to train the parameters of learning models. In practice, however, the gradient information may be unavailable or infeasible in some applications, such as federated black-box optimization problems. To solve the issue, we propose an innovative zeroth-order adaptive federated learning algorithm without using the gradient information, referred to as ZO-AdaFL, which integrates the zeroth-order optimization algorithm into the adaptive gradient method. Moreover, we also rigorously analyze the convergence behavior of ZO-AdaFL in a non-convex setting, i.e., where ZO-AdaFL achieves convergence to a region close to a stationary point at a speed of O(1/T) (T represents the total iteration number). Finally, to verify the performance of ZO-AdaFL, simulation experiments are performed using the MNIST and FMNIST datasets. Our experimental findings demonstrate that ZO-AdaFL outperforms other state-of-the-art zeroth-order FL approaches in terms of both effectiveness and efficiency.

A control theoretic framework for adaptive gradient optimizers

Gradient-based optimization and control frameworks have been utilized in several applications. The learning rate parameter is typically chosen following a schedule or using methods such as line search to enhance the convergence rate. Recently, the machine learning community has developed methodologies for automated tuning of the learning rate, known as adaptive gradient methods. This paper develops a control theory-inspired framework for modeling adaptive gradient methods that solve non-convex optimization problems. We first model the adaptive gradient methods in a state–space framework, which allows us to present simpler convergence proofs of prominent adaptive optimizers, such as AdaGrad, Adam, and AdaBelief. The proposed framework is constructive because it allows synthesizing new adaptive optimizers. To illustrate this fact, we then utilize the transfer function paradigm from classical control to propose a new variant of Adam, coined AdamSSM, and prove its convergence. We add an appropriate pole-zero pair in the transfer function from squared gradients to the second moment estimate. Applications on benchmark machine learning tasks of image classification using CNN architectures and language modeling using LSTM architecture demonstrate that the AdamSSM algorithm improves the gap between generalization accuracy and faster convergence than the recent adaptive gradient methods.

On Faster Convergence of Scaled Sign Gradient Descent

Communication has been seen as a significant bottleneck in industrial applications over large-scale networks. To alleviate the communication burden, sign-based optimization algorithms have gained popularity recently in both industrial and academic communities, which is shown to be closely related to adaptive gradient methods, such as Adam. Along this line, this paper investigates faster convergence for a variant of sign-based gradient descent, called scaled SIGNGD, in three cases: 1) the objective function is strongly convex; 2) the objective function is nonconvex but satisfies the Polyak-Łojasiewicz (PL) inequality; 3) the gradient is stochastic, called scaled SIGNSGD in this case. For the first two cases, it can be shown that the scaled SIGNGD converges at a linear rate. For case 3), the algorithm is shown to converge linearly to a neighborhood of the optimal value when a constant learning rate is employed, and the algorithm converges at a rate of <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"><tex-math notation="LaTeX">$O(1/k+1/k^{2}+1/k^{3})$</tex-math></inline-formula> when using a diminishing learning rate, where <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"><tex-math notation="LaTeX">$k$</tex-math></inline-formula> is the iteration number. The results are also extended to the distributed setting by majority vote in a parameter-server framework. Finally, numerical experiments are performed to corroborate the theoretical findings.

Escaping saddle points efficiently with occupation-time-adapted perturbations

Motivated by the super-diffusivity of self-repelling random walk, which has roots in statistical physics, this paper develops a new perturbation mechanism for optimization algorithms. In this mechanism, perturbations are adapted to the history of states via the notion of occupation time. After integrating this mechanism into the framework of perturbed gradient descent (PGD) and perturbed accelerated gradient descent (PAGD), two new algorithms are proposed: perturbed gradient descent adapted to occupation time (PGDOT) and its accelerated version (PAGDOT). PGDOT and PAGDOT are guaranteed to avoid getting stuck at non-degenerate saddle points, and are shown to converge to second-order stationary points at least as fast as PGD and PAGD, respectively. The theoretical analysis is corroborated by empirical studies in which the new algorithms consistently escape saddle points and outperform not only their counterparts, PGD and PAGD, but also other popular alternatives including stochastic gradient descent, Adam, and several state-of-the-art adaptive gradient methods.

An Exploration of Online Interactive Teaching Mode of Vocational Undergraduate Business English Based on Deep Learning Networks

Abstract English teaching is highly dependent on interactive communication, and for a long time, interactive teaching methods and tools have often had difficulty fully showing their advantages in the classroom due to subjective and objective conditions. In this study, based on self-coding networks, KL entropy is introduced for sparse limitation. Then, an adaptive gradient method with dynamic momentum and an essential learning rate is used to optimize the deep self-coding network. At the same time, an online interactive teaching model of vocational undergraduate business English is constructed from the aspects of pre-course preparation, platform selection, and in-class content, and teaching practice is carried out to explore the effect of the model. The results show that the significant difference between the posttest English scores of the experimental class and the control class in University Z is 0.000&lt;0.05, and the mean value of the English scores of the experimental class is slightly higher than the mean value of the English scores of the control class. This study is based on Adamdyn’s online interactive teaching model, which is effective in teaching vocational undergraduate business English.

Adaptive fixed-time proximal gradient method for non-smooth optimization: the fractional approach

Non-smooth optimization has now played an important role in many fields, such as l1-regulation problem and sparse recovery problem. In this paper, a novel fractional fixed-time proximal gradient method is proposed for solving the composite optimization problems with non-smooth terms. At first, according to the distribution of zeroes of Mittag-Leffler functions, a novel fixed-time convergence theory with a fractional adaptive gain is proposed. Based on the optimal condition analysis of the transformed problem using the augmented Lagrangian function, a novel fixed-time proximal gradient method is then proposed and the fixed-time convergence property is rigorously proven. Furthermore, the proposed algorithm is applied to solve the sparse recovery problem in compressive sensing, and the restricted isometry condition required for the application of the algorithm is analyzed to achieve fixed-time recovery of the signal. Finally, simulation examples are provided to validate all the results.

A Derivative-Incorporated Adaptive Gradient Method for Federated Learning

As a new machine learning technique, federated learning has received more attention in recent years, which enables decentralized model training across data silos or edge intelligent devices in the Internet of Things without exchanging local raw data. All kinds of algorithms are proposed to solve the challenges in federated learning. However, most of these methods are based on stochastic gradient descent, which undergoes slow convergence and unstable performance during the training stage. In this paper, we propose a differential adaptive federated optimization method, which incorporates an adaptive learning rate and the gradient difference into the iteration rule of the global model. We further adopt the first-order moment estimation to compute the approximate value of the differential term so as to avoid amplifying the random noise from the input data sample. The theoretical convergence guarantee is established for our proposed method in a stochastic non-convex setting under full client participation and partial client participation cases. Experiments for the image classification task are performed on two standard datasets by training a neural network model, and experiment results on different baselines demonstrate the effectiveness of our proposed method.

An adaptive modified three-term conjugate gradient method with global convergence

In this paper, an adaptive three-term conjugate gradient method is proposed based on the Perry family by using a modified secant condition. The new method depends on a positive parameter which has a critical role in the efficiency of the algorithm. We propose an adaptive choice for this parameter to control the condition number of the direction matrix by minimizing the distance between the smallest and largest eigenvalue. Global convergence of the new algorithm is proved for general functions, under standard assumptions. Experimental results verify the expected numerical behavior of our algorithm.

Parallel and distributed asynchronous adaptive stochastic gradient methods

Parallel and distributed asynchronous adaptive stochastic gradient methods

Adaptive learning rate optimization algorithms with dynamic bound based on Barzilai-Borwein method

The training effect of the neural network model is directly influenced by optimization algorithms. The Barzilai-Borwein(BB) method is used in the stochastic gradient descent (SGD) and other deep learning optimization algorithms because of its outstanding performance in terms of convergence speed. In order to improve the stability of BB step size and the output quality of network model in deep learning, this paper presents two improved optimization algorithms based on the BB method: BBbound and AdaBBbound. BBbound reduces the floating BB step size by generating the upper bound of the step in the current iteration. It avoids the occasional occurrence of long steps; AdaBBbound is an adaptive gradient method based on BB method with dynamic bound. The algorithm has a fast convergence effect in the early stage, then calculates learning rate and smoothly transitions to SGD by setting the better conditions of the BB method. We analyzed the performance of the two algorithms based on the initial conditions and the learning rate change curve. Meanwhile, we tested our algorithms on popular network models such as ResNet and DenseNet. The results showed that the new optimization algorithms have achieved high stability and significantly improved the performance of the network model.

Adopting improved Adam optimizer to train dendritic neuron model for water quality prediction.

As one of continuous concern all over the world, the problem of water quality may cause diseases and poisoning and even endanger people's lives. Therefore, the prediction of water quality is of great significance to the efficient management of water resources. However, existing prediction algorithms not only require more operation time but also have low accuracy. In recent years, neural networks are widely used to predict water quality, and the computational power of individual neurons has attracted more and more attention. The main content of this research is to use a novel dendritic neuron model (DNM) to predict water quality. In DNM, dendrites combine synapses of different states instead of simple linear weighting, which has a better fitting ability compared with traditional neural networks. In addition, a recent optimization algorithm called AMSGrad (Adaptive Gradient Method) has been introduced to improve the performance of the Adam dendritic neuron model (ADNM). The performance of ADNM is compared with that of traditional neural networks, and the simulation results show that ADNM is better than traditional neural networks in mean square error, root mean square error and other indicators. Furthermore, the stability and accuracy of ADNM are better than those of other conventional models. Based on trained neural networks, policymakers and managers can use the model to predict the water quality. Real-time water quality level at the monitoring site can be presented so that measures can be taken to avoid diseases caused by water quality problems.

Variable admittance control based on iterative learning scheme with adaptive gradient methods for human-robot collaborative manipulation

Variable admittance control based on iterative learning scheme with adaptive gradient methods for human-robot collaborative manipulation

SVRG meets AdaGrad: painless variance reduction

Variance reduction (VR) methods for finite-sum minimization typically require the knowledge of problem-dependent constants that are often unknown and difficult to estimate. To address this, we use ideas from adaptive gradient methods to propose AdaSVRG, which is a more-robust variant of SVRG, a common VR method. AdaSVRG uses AdaGrad, a common adaptive gradient method, in the inner loop of SVRG, making it robust to the choice of step-size. When minimizing a sum of n smooth convex functions, we prove that a variant of AdaSVRG requires \(\tilde{O}(n + 1/\epsilon )\) gradient evaluations to achieve an \(O(\epsilon )\)-suboptimality, matching the typical rate, but without needing to know problem-dependent constants. Next, we show that the dynamics of AdaGrad exhibit a two-phase behavior – the step-size remains approximately constant in the first phase, and then decreases at a \(O\left( {1}/{\sqrt{t}}\right)\) rate. This result maybe of independent interest, and allows us to propose a heuristic that adaptively determines the length of each inner-loop in AdaSVRG. Via experiments on synthetic and real-world datasets, we validate the robustness and effectiveness of AdaSVRG, demonstrating its superior performance over standard and other “tune-free” VR methods.

A Fully Natural Gradient Scheme for Improving Inference of the Heterogeneous Multioutput Gaussian Process Model.

A recent novel extension of multioutput Gaussian processes (GPs) handles heterogeneous outputs, assuming that each output has its own likelihood function. It uses a vector-valued GP prior to jointly model all likelihoods' parameters as latent functions drawn from a GP with a linear model of coregionalization (LMC) covariance. By means of an inducing points' framework, the model is able to obtain tractable variational bounds amenable to stochastic variational inference (SVI). Nonetheless, the strong conditioning between the variational parameters and the hyperparameters burdens the adaptive gradient optimization methods used in the original approach. To overcome this issue, we borrow ideas from variational optimization introducing an exploratory distribution over the hyperparameters, allowing inference together with the posterior's variational parameters through a fully natural gradient (NG) optimization scheme. Furthermore, in this work, we introduce an extension of the heterogeneous multioutput model, where its latent functions are drawn from convolution processes. We show that our optimization scheme can achieve better local optima solutions with higher test performance rates than adaptive gradient methods for both the LMC and the convolution process model. We also show how to make the convolutional model scalable by means of SVI and how to optimize it through a fully NG scheme. We compare the performance of the different methods over the toy and real databases.

A novel temperature prediction method without using energy equation based on physics-informed neural network (PINN): A case study on plate- circular/square pin-fin heat sinks

This study introduces a new physics-informed neural networks (PINN)-based prediction method to determine the temperature pattern of fluid and fins when flow passes over plate-circular/ plate-square pin fin heat sinks (PCPFHS / PSPFHS). The proposed method is based on calculating the velocity pattern on the fins' surface. For this target, a training algorithm based on the feed-forward neural network (FNN) (110 layers of learnable weights and 16 neurons in each layer), a nonlinear activation function (rectified linear unit, "ReLU") and the Adam method for optimization is used. The training algorithm is fed by transient large eddy simulation (LES) results at every 0.01 s time step for the total physical time of 100 s. According to the input parameter type, the training:validation ratio is varied between 70:30 and 90:10 in order to keep the coefficient of determination (R2) at its maximum. The automatic differentiation employed the forward accumulation approach to reduce calculation costs, while the transient training matrix fed the neural network. The adaptive gradient (AdaGrad) method is also to improve convergence process and its speed up. Based on the developed calculation tools, the temperature pattern for the flow and over the fins are calculated according to the energy balance on the fin surface and the transient pattern of velocity predicted by PINN. After careful validation with experimental data and sensitivity analysis on the number of neurons and layers, the thermal behavior of PCPFHS and PSPFHS are determined using the conduction heat transfer equation inside the fins via the finite element method and by assuming a heat balance between the fins' surface and airflow. As a result of the proposed method, it is possible to reduce the number of equations in the calculation process of these parameters. According to the results, it is found that PSPFHS has an average Nusselt number which is 9.63% greater than the one in PCPFHS. However, compared to PCPFHS, PSPFHS shows a -17.78% reduced vorticity ratio at Re = 4,865. The results indicated that, for long calculation times (for instance, at 2,000 s physical time), the PINN method reduces calculation costs up to 35% compared to the technique that directly solves the energy conservation in the whole domain.

An Adaptive Projection Gradient Method for Solving Nonlinear Fractional Programming

In this study, we focus on solving the nonlinear fractional optimization problem in which the numerator is smooth convex and the denominator is smooth concave. To achieve this goal, we develop an algorithm called the adaptive projection gradient method. The main advantage of this method is that it allows the computations for the gradients of the considered functions and the metric projection to take place separately. Moreover, an interesting property that distinguishes the proposed method from some of the existing methods is the nonincreasing property of its step-size sequence. In this study, we also prove that the sequence of iterates that is generated by the method converges to a solution for the considered problem and we derive the rate of convergence. To illustrate the performance and efficiency of our algorithm, some numerical experiments are performed.

AdaCB: An Adaptive Gradient Method with Convergence Range Bound of Learning Rate

Adaptive gradient descent methods such as Adam, RMSprop, and AdaGrad achieve great success in training deep learning models. These methods adaptively change the learning rates, resulting in a faster convergence speed. Recent studies have shown their problems include extreme learning rates, non-convergence issues, as well as poor generalization. Some enhanced variants have been proposed, such as AMSGrad, and AdaBound. However, the performances of these alternatives are controversial and some drawbacks still occur. In this work, we proposed an optimizer called AdaCB, which limits the learning rates of Adam in a convergence range bound. The bound range is determined by the LR test, and then two bound functions are designed to constrain Adam, and two bound functions tend to a constant value. To evaluate our method, we carry out experiments on the image classification task, three models including Smallnet, Network IN Network, and Resnet are trained on CIFAR10 and CIFAR100 datasets. Experimental results show that our method outperforms other optimizers on CIFAR10 and CIFAR100 datasets with accuracies of (82.76%, 53.29%), (86.24%, 60.19%), and (83.24%, 55.04%) on Smallnet, Network IN Network and Resnet, respectively. The results also indicate that our method maintains a faster learning speed, like adaptive gradient methods, in the early stage and achieves considerable accuracy, like SGD (M), at the end.

Dynamic regret of adaptive gradient methods for strongly convex problems

Adaptive gradient algorithms such as ?>AdaGrad and its variants have gained popularity in the training of deep neural networks. While many works as for adaptive methods have focused on the static regret as a performance metric to achieve a good regret guarantee, the dynamic regret analyses of these methods remain unclear. As opposed to the static regret, dynamic regret is considered to be a stronger concept of performance measurement in the sense that it explicitly elucidates the non-stationarity of the environment. In this paper, we go through a variant of AdaGrad (referred to as M-AdaGrad) in a strong convex setting via the notion of dynamic regret, which measures the performance of an online learner against a reference (optimal) solution that may change over time. We demonstrate a regret bound in terms of the path-length of the minimizer sequence that essentially reflects the non-stationarity of environments. In addition, we enhance the dynamic regret bound by exploiting the multiple accesses of the gradient to the learner in each round. Empirical results indicate that M-AdaGrad works also well in practice.

DP-RBAdaBound: A differentially private randomized block-coordinate adaptive gradient algorithm for training deep neural networks

In order to rapidly train deep learning models, many adaptive gradient methods have been proposed in recent years, such as Adam and AMSGrad. However, the computation of the full gradient vectors in the above methods becomes expensive prohibitively at each iteration for handling high dimensional data. Moreover, the private information may be leaked in the process of training. For these reasons, we propose a differentially private randomized block-coordinate adaptive gradient algorithm, called DP-RBAdaBound, for training deep learning model. To reduce the computation of the full gradient vectors, we randomly choose a block-coordinate to update the model parameter at each iteration. Meanwhile, we add the Laplace noise to the block-coordinate of the gradient vector per iteration for preserving the privacy of users. Furthermore, we rigorously show that the proposed algorithm can preserve ϵ-differential privacy, where ϵ>0 denotes the privacy level. Moreover, we also rigorously prove that the square-root regret bound is also achieved in convex settings, i.e., O(T), where T is a time horizon. Besides, we offer a tradeoff between regret bound and privacy, i.e., the regret bound has order of O(1/ϵ4) by fixing other parameters when ϵ-differential privacy is achieved. Finally, we confirm the computational benefit by training DenseNet-121 and ResNet-34 models on CIFAR-10 dataset, respectively. Meanwhile, the effectiveness of DP-RBAdaBound is also validated through training the DenseNet-121 model on CIFAR-100 dataset and LSTM model on Penn TreeBank dataset, respectively.