Mini-batch algorithms with online step size

Abstract

Mini-batch algorithms have been proposed as a way to speed-up stochastic optimization methods and good results for mini-batch algorithms have been reported previously. A major issue with mini-batch algorithms is how to timely and readily acquire step size while running the algorithm. Usually, mini-batch algorithms employ a diminishing step size, or a best-tuned step size by mentor, which, in practice, are time consuming. To solve this problem, we propose using a hypergradient to compute an online step size (OSS) for mini-batch algorithms. Specifically, we incorporate online step size into advanced mini-batch algorithms, mini-batch nonconvex stochastic variance reduced gradient (MSVRG), thereby generating a new method, MSVRG-OSS. When computing step size in MSVRG-OSS, mini-batch samples are used. In addition, MSVRG-OSS, which needs little additional computation, requires only one extra copy of the original gradient to be stored in memory. We prove that MSVRG-OSS converges linearly in expectation and analyze its complexity. We present numerical results on problems arising with machine learning that indicate the proposed method shows great promise. We also show that, with slightly large batch samples, MSVRG-OSS is insensitive to the initial parameters, which are the key factor for controlling the performance of the algorithm.