AdaDB: An adaptive gradient method with data-dependent bound

Abstract

Adaptive gradient learning methods such as Adam, RMSProp, and AdaGrad play an essential role in training a very deep neural network. The learning rates of these optimizers are adaptively changed to accelerate the training process. And the convergence speed is much faster than SGD in many deep learning tasks such as classification and NLP tasks. However, recent works have pointed out that adaptive learning methods can not converge to a critical point under some situations and suffer poor generalization in many deep learning tasks. In our study, we propose AdaDB, an adaptive learning optimizer with data-dependent bound on the learning rate. Every element in the learning rate vector is constrained between a dynamic upper bound and a constant lower bound. And the upper bound is dependent on the data. We also give a theoretical proof of the convergence of AdaDB in the non-convex setting. Our experiments show that AdaDB is capable of eliminating the generalization gap between Adam and SGD. Experiments also reveal the convergence speed of AdaDB is much faster than Adam.