Nonconvex optimization with inertial proximal stochastic variance reduction gradient

Abstract

Recent research has shown the significant performance of stochastic gradient descent (SGD) coupled with a momentum trick in solving convex empirical risk minimization (ERM) problems. However, most machine learning problems are nonconvex. An inertial accelerated Prox-SVRG (IAPSVRG) algorithm is proposed to minimize ERM problems with no convexity assumption by incorporating Nesterov's momentum trick into the stochastic variance reduction gradient (SVRG). Certain convergence properties are established for the subsequence are generated by the IAPSVRG algorithm using the supermartingale convergence theorem. Building on the powerful Kurdyka-Łojasiewicz (KL) property, it is shown that each bounded sequence generated by the IAPSVRG algorithm converges globally to a critical point of the objective function if an appropriate threshold of the inertial coefficient is provided. A regularized nonlinear least squares is used to check the effectiveness of the proposed method in terms of classification, convergence, and sensitivity to the choice of the momentum parameter. The experimental results show that the testing accuracy of the proposed algorithm on each of the four selected datasets exceeds at least half that of the other four compared algorithms. Furthermore, the proposed algorithm is less sensitive to the choice of the momentum parameter within a certain threshold compared to the four compared algorithms.