Toward Optimal Learning Rate Schedule in Scene Classification Network

Abstract

Stochastic gradient descent (SGD) and adaptive methods, including ADAM, RMSProp, AdaDelta, and AdaGrad, are two dominant optimization algorithms for training convolution neural network (CNN) in scene classification tasks. Recent work reveals that these adaptive methods lead to degraded performance in image classification tasks compared with SGD. In this letter, a learning rate schedule named switching from constant to step decay (SCTSD) is proposed to further improve the classification accuracy of SGD. SCTSD begins with a constant learning rate and switches to step decay learning rates when appropriate. Theoretical evidence is provided on the superiority of SCTSD compared with other manually tuned schedules. Comparison experiments have been conducted among adaptive methods, SCTSD, and other manual schedules with three CNN architectures. The experiment results on various scene classification data sets show that SCTSD has the highest accuracy on the test set and it is state of the art. In the end, some suggestions on hyperparameters selection of SCTSD are given for scene classification.