Comments on the Randomized Kaczmarz Method

Abstract

In this short note we respond to some concerns raised by Y. Censor, G. Herman, and M. Jiang about the randomized Kaczmarz method that we proposed in [5]. The Kaczmarz method is a well-known iterative algorithm for solving a linear system of equations Ax = b. For more than seven decades, this method was useful in practical applications, and it was studied in many research papers. Despite of this, little is known about the rate of convergence of this method. The classical scheme of Kaczmarz’s method sweeps through the rows of A in a cyclic manner, projecting in each substep the last iterate orthogonally onto a hyperplane associated with a row of A. One variation of Kaczmarz’s method consists of randomly choosing in each iteration the row for the projection. Our algorithm in [5] (labeled Algorithm 1 there) is based on this approach. The idea of choosing the rows randomly is certainly not new. It has been mentioned for instance by Natterer [4], and later also by Feichtinger et al. [1] and by G. Herman and L. Meyer [3]. In these papers, improvement of performance is observed in numerical experiments, but none of these papers contain any proof of the rate of convergence. We consider the main contribution of [5] that (i) it contains the first proof for a rate of convergence for the Kaczmarz’s method that is applicable to general matrices (and not just to very restricted special cases); (ii) the algorithm achieves an exponential rate of convergence; and (iii) the rate