DALU: Adaptive Learning Rate Update in Distributed Deep Learning

Abstract

In the last decade, deep learning has attracted much attention from both the academia and industry. As one of the important hyperparameters, learning rate greatly affects the speed and convergence of deep learning. In the existing approaches, the learning rate is commonly adjusted with the training process, however, does not consider the model accuracy after each training epoch. This problem is even more complex in distributed deep learning approach, which involves multiple machines for learning. In this study, we propose an adaptive learning rate updating strategy (DALU) for distributed deep learning. After completing each training epoch, the classification accuracy on training data set is recorded. Learning rate is adjusted according to the accuracy of current epoch and previous epochs. We also design several special machines having more aggressive learning rate update than regular machines in the distributed system. Compared with current commonly used learning rate adjusting approaches, exponential decay learning rate and cosine decay learning rate, DALU improves the accuracy of deep learning by 3.0% to 3.9%.