Stochastic gradient descent for linear systems with sequential matrix entry accumulation

Abstract

Conventional stochastic iterative methods are often employed for solving linear systems of equations involving large matrix sizes using low memory footprint. However, their performances are often limited by the unavailability of all the matrix entries, which is often termed as the problem of missing data. Although Ma and Needell [1] have recently proposed a method, termed as mSGD, assuming a model for data missing that results in improved convergence, their result is also affected by constant large variance of the stochastic gradient. In this paper we propose a SGD type method termed as cumulative information SGD (CISGD) for solving a linear system with missing data with an additional provision to accumulate a very small number of matrix entries sequentially per iteration, termed as the sequential matrix entry accumulation (SEMEA) mechanism. CISGD uses the data collected by SEMEA mechanism along with the prior model for data missing mechanism of [1] to gradually reduce variance of the stochastic gradient. The convergence of the proposed CISGD is theoretically analyzed and some interesting implications of the result are investigated under a specific SEMEA mechanism. Finally, numerical experiments are performed along with simulations that corroborate the theoretical findings regarding the efficacy of the proposed CISGD method.