Comparative Design Analysis of Optimized Learning Rate for Convolutional Neural Network

Abstract

Learning Rate is the most critical hyper-parameter to tune while training neural networks. It determines the amount of weight that should be adjusted for loss gradient. Therefore, choosing an optimal learning rate is a challenging task for neural networks. In this paper, various learning rate schedules like constant learning rates, step decay and exponential decay algorithm are implemented. Exponential decay comes out to be the best learning method among the constant, step and exponential decays. Further, best learning rate is used to optimize Adam, RMSProp and SGDM algorithm. These are implemented on the CIFAR-10 data set. Various combinations of different parameters of exponential decay such as decay steps, decay factor, initial learning rates are used to train the model, and their optimal learning rate and losses are observed. The parameters have minimum loss considered the best one. The comparative analysis of the model is performed based on accuracy, loss function and a number of iterations. The experimental results show that Adam with exponential decay based learning method achieving higher accuracy at batch size 128, decay steps 400, and drop out rate 0.3 with minimum loss 0.5. So, it is proven the best optimization method for training convolutional neural networks.