An accelerated stochastic variance-reduced method for machine learning problems

Abstract

Variance reduction techniques provide simple and fast algorithms for solving machine learning problems. In this paper, we present a novel stochastic variance-reduced method. The proposed method relies on the mini-batch version of stochastic recursive gradient algorithm (MB-SARAH), which updates stochastic gradient estimates by using a simple recursive scheme. However, facing the challenge of the step size sequence selection in MB-SARAH, we introduce an online step size sequence based on the hypergradient descent (HD) method, which only requires little additional computation. For the proposed method, referred to as MB-SARAH-HD, we provide a general convergence analysis and prove linear convergence for strongly convex problems in expectation. Specifically, we prove that the proposed method has sublinear convergence rate in a single outer loop. We also prove that the iteration complexity outperforms several variants of the state-of-the-art stochastic gradient descent (SGD) method under suitable conditions. Numerical experiments on standard datasets are provided to demonstrate the efficacy and superiority of our MB-SARAH-HD method over existing approaches in the literature.