Kaczmarz's anomaly: A surprising feature of Kaczmarz's method

Abstract

This paper exposes one of the more fascinating features of Kaczmarz's method. Let Ax=b denote the linear system to solve, where A∈Rm×n and b=(b1,b2,…,bm)T∈Rm. Let the rows of A be denoted as aiT,i=1,…,m. Then the ith step of the basic iteration achieves projection on the hyperplane {x|aiTx=bi}. Normalizing the equations allows us to assume that aiTai=1, so the vectors ai,i=1,…,m, are points on the unit sphere in Rn. The “Kaczmarz anomaly” occurs when these points are randomly scattered over a large area of the unit sphere. Assume first that the number of equations, m, is considerably smaller than the number of unknowns, n. In this case Kaczmarz's method enjoys a fast rate of convergence. Then, as m increases toward n, the rate of convergence slows down. That is, the more equations we have more iterations are needed. In particular, as m approaches n there is dramatic increase in the number of iterations. The closer are m and n the slower is the convergence. However, as m passes n the situation is reversed. From now on, the more equations we have less iterations are needed. Finally, when solving overdetermined linear systems in which m is considerably larger than n the method returns to enjoy rapid convergence.The paper illustrates this phenomenon and explains its reasons. Linear systems with randomly scattered row's directions are common in practical applications. The exposed behavior helps to see the reasons behind success (or failure) of Kaczmarz's method when applied to solve such systems.