An efficient MLP-learning algorithm using parallel tangent gradient and improved adaptive learning rates

Abstract

Poor convergence is a common problem of gradient-based multi-layer perceptron (MLP)-learning algorithms. It is claimed that using a deflecting direction like momentum and adaptive learning rates (ALRs) can improve the convergence performance. For a more reliable and faster MLP learning, we introduce the parallel tangent gradient with adaptive learning rates (PTGALR) algorithm that uses parallel tangent deflecting direction instead of the momentum. Moreover, we use two independent variable learning rates, one for the gradient descent and the other for accelerating direction through the parallel tangent. Also, we propose an improved ALR computation algorithm that calculates the learning rates with a dispensable error oscillation. This adaptation algorithm has two outputs: one is the applied learning rate and the other is used for a better learning rate estimation in the next iteration. Moreover, the proposed ALR computation algorithm models the error function as a one-dimensional quadratic function of the learning rate when it is needed. The implementation results of PTGALR for some well-known binary and real MLP problems show higher and faster convergence with lower oscillations than the similar adaptive learning algorithms.