Convergence analysis of asynchronous stochastic recursive gradient algorithms

Abstract

Asynchronous stochastic algorithms with variance reduction techniques have been empirically shown to be useful for many large-scale machine learning problems. By making a parallel optimization algorithm asynchronous, one can reduce the synchronization cost and improve the practical efficiency. Recently, the stochastic recursive gradient algorithm has shown superior theoretical performance; however, it is not scalable enough in the current big data era. To make it more practical, we propose a class of asynchronous stochastic recursive gradient methods and analyze them in the shared memory model. The analysis results show that our asynchronous algorithms can linearly converge to the solution in the strongly convex case and complete the iteration faster. In addition, we analyze the “price of asynchrony” and give the sufficient conditions required for linear speedup. To the best of our knowledge, our speedup conditions match the optimal bounds of asynchronous stochastic algorithms known thus far. Finally, we demonstrate our theoretical analyses from a practical implementation on a multicore machine.