Performance comparison of different momentum techniques on deep reinforcement learning

Abstract

ABSTRACTRecently, the popularity of deep artificial neural networks has increased considerably. Generally, the method used in the training of these structures is simple gradient descent. However, training a deep structure with simple gradient descent can take quite a long time. Some additional approaches have been utilized to solve this problem. One of these techniques is the momentum that accelerates gradient descent learning. Momentum techniques can be used for supervised learning as well as for reinforcement learning. However, its efficiency may vary due to the dissimilarities in two learning processes. While the expected values of inputs are clearly known in supervised learning, it may take long-running iterations to reach the exact expected values of the states in reinforcement learning. In an online learning approach, a deep neural network should not memorize and continue to converge with the more precise values that exist over time during these iterations. For this reason, it is necessary to use a momentum technique that both adapt to reinforcement learning and accelerate the learning process. In this paper, the performance of different momentum techniques is compared with the Othello game benchmark. Test results show that the Nesterov momentum technique provided a more effective generalization with an online reinforcement learning approach.