Variance-Reduced Shuffling Gradient Descent With Momentum for Finite-Sum Minimization

Abstract

Finite-sum minimization is a fundamental optimization problem in signal processing and machine learning. This paper proposes a variance-reduced shuffling gradient descent with Nesterov’s momentum for smooth convex finite-sum optimization. We integrate an explicit variance reduction into the shuffling gradient descent to deal with the variance introduced by shuffling gradients. The proposed algorithm with a unified shuffling scheme converges at a rate of O(1T), where T is the number of epochs. The convergence rate independent of gradient variance is better than most existing shuffling gradient algorithms for convex optimization. Finally, numerical simulations demonstrate the convergence performance of the proposed algorithm.