Calibrating the adaptive learning rate to improve convergence of ADAM

Abstract

Adaptive gradient methods (AGMs) have been widely used to optimize nonconvex problems in the deep learning area. We identify two aspects of AGMs that can be further improved. First, we observe that the adaptive learning rate (A-LR) used by AGMs varies significantly across the dimensions of the optimization problem and over epochs, which we call anisotropic scale of the A-LR. It can slow down the convergence and make the algorithm trap into a sharp local minimizer. Actually, all existing modified AGMs represent efforts in revising the A-LR. Second, we theoretically prove that the convergence rate of AGMs depends on its hyper-parameter used in the A-LR formula, such as the ∊ used in ADAM, which has not been examined previously. We then propose new AGMs that calibrate the A-LR with an activation function, such as, the softplus function. Particularly, the Sadam and SAMSGrad methods are two instances of our method. We further prove that SAMSGrad enjoys a better convergence speed than the AMSGrad method under separate conditions including the nonconvex, non-strongly convex, and Polyak-Łojasiewicz conditions. Empirical studies are used to demonstrate the anisotropic A-LR issue and show that the proposed methods outperform existing AGMs and generalize better in multiple deep learning tasks.