Nonlinear Optimization Method Based on Stochastic Gradient Descent for Fast Convergence

Abstract

Nonlinear optimization problems where an objective function is given as the sum of multiple partial functions are important because they include many machine learning problems. Although some extensions of stochastic gradient descent are proposed for solving the problems, they have their respective disadvantages. In order to overcome the disadvantages, we previously proposed a method based on stochastic gradient descent. However, this method does not converge at a good solution when overestimating the average gradient of the partial functions. In addition, the initial step size is small, and therefore the convergence speed is reduced. In order to resolve these problems and realize faster convergence, in this paper, we propose a method which is an extension of our previous method.