Randomized Kaczmarz Method Research Articles

A Randomized Algorithm for Multivariate Function Approximation

The randomized Kaczmarz (RK) method is a randomized iterative algorithm for solving (overdetermined) linear systems of equations. In this paper, we extend the RK method to function approximation in a bounded domain. We demonstrate that by conducting the approximation randomly one sample at a time the method converges. Convergence analysis is conducted in terms of expectation, where we establish sharp upper and lower bounds for both the convergence rate of the algorithm and the error of the resulting approximation. The analysis also establishes the optimal sampling probability measure to achieve the optimal rate of convergence. Various numerical examples are provided to validate the theoretical results.

A Sampling Kaczmarz--Motzkin Algorithm for Linear Feasibility

We combine two iterative algorithms for solving large-scale systems of linear inequalities: the relaxation method of Agmon, Motzkin, et al. and the randomized Kaczmarz method. We obtain a family of algorithms that generalize and extend both projection-based techniques. We prove several convergence results, and our computational experiments show our algorithms often outperform the original methods.

Convergence Properties of the Randomized Extended Gauss--Seidel and Kaczmarz Methods

The Kaczmarz and Gauss--Seidel methods both solve a linear system $\boldsymbol{X}{\boldsymbol{\beta}} = \boldsymbol{y}$ by iteratively refining the solution estimate. Recent interest in these methods has been sparked by a proof of Strohmer and Vershynin which shows the randomized Kaczmarz method converges linearly in expectation to the solution. Lewis and Leventhal then proved a similar result for the randomized Gauss--Seidel algorithm. However, the behavior of both methods depends heavily on whether the system is underdetermined or overdetermined, and whether it is consistent or not. Here we provide a unified theory of both methods, their variants for these different settings, and draw connections between both approaches. In doing so, we also provide a proof that an extended version of randomized Gauss--Seidel converges linearly to the least norm solution in the underdetermined case (where the usual randomized Gauss--Seidel fails to converge). We detail analytically and empirically the convergence properties of both methods and their extended variants in all possible system settings. With this result, a complete and rigorous theory of both methods is furnished.

Randomized Iterative Methods for Linear Systems

We develop a novel, fundamental, and surprisingly simple randomized iterative method for solving consistent linear systems. Our method has six different but equivalent interpretations: sketch-and-project, constrain-and-approximate, random intersect, random linear solve, random update, and random fixed point. By varying its two parameters---a positive definite matrix (defining geometry), and a random matrix (sampled in an independent and identically distributed fashion in each iteration)---we recover a comprehensive array of well-known algorithms as special cases, including the randomized Kaczmarz method, randomized Newton method, randomized coordinate descent method, and random Gaussian pursuit. We naturally also obtain variants of all these methods using blocks and importance sampling. However, our method allows for a much wider selection of these two parameters, which leads to a number of new specific methods. We prove exponential convergence of the expected norm of the error in a single theorem, from which existing complexity results for known variants can be obtained. However, we also give an exact formula for the evolution of the expected iterates, which allows us to give lower bounds on the convergence rate.

Iterative Hard Thresholding Based on Randomized Kaczmarz Method

The iterative hard thresholding methods have been recently developed to deal with the sparse regularization problems arising in compressed sensing and other sparse signal processing methods. The methods are attractive due to their simplicity. In this paper, we propose acceleration schemes for iterative hard thresholding methods based on randomized Kaczmarz method. The speedup is achieved by replacing the gradient step in the iterative hard thresholding methods with a randomized Kaczmarz algorithm. Numerical experiments with the sparse signal recovery show the simplicity and effectiveness of the proposed algorithms.

Block Kaczmarz Method with Inequalities

The randomized Kaczmarz method is an iterative algorithm that solves systems of linear equations. Recently, the randomized method was extended to systems of equalities and inequalities by Leventhal and Lewis. Even more recently, Needell and Tropp provided an analysis of a block version of this randomized method for systems of linear equations. This paper considers the use of a block type method for systems of mixed equalities and inequalities, bridging these two bodies of work. We show that utilizing a matrix paving over the equalities of the system can lead to significantly improved convergence, and prove a linear convergence rate as in the standard block method. We also demonstrate that using blocks of inequalities offers similar improvement only when the system satisfies a certain geometric property. We support the theoretical analysis with several experimental results.

Randomized Extended Kaczmarz for Solving Least Squares

We present a randomized iterative algorithm that exponentially converges in the mean square to the minimum $\ell_2$-norm least squares solution of a given linear system of equations. The expected number of arithmetic operations required to obtain an estimate of given accuracy is proportional to the squared condition number of the system multiplied by the number of nonzero entries of the input matrix. The proposed algorithm is an extension of the randomized Kaczmarz method that was analyzed by Strohmer and Vershynin.

Two-Subspace Projection Method for Coherent Overdetermined Systems

We present a Projection onto Convex Sets (POCS) type algorithm for solving systems of linear equations. POCS methods have found many applications ranging from computer tomography to digital signal and image processing. The Kaczmarz method is one of the most popular solvers for overdetermined systems of linear equations due to its speed and simplicity. Here we introduce and analyze an extension of the Kaczmarz method that iteratively projects the estimate onto a solution space given by two randomly selected rows. We show that this projection algorithm provides exponential convergence to the solution in expectation. The convergence rate improves upon that of the standard randomized Kaczmarz method when the system has correlated rows. Experimental results confirm that in this case our method significantly outperforms the randomized Kaczmarz method.

Acceleration of randomized Kaczmarz method via the Johnson–Lindenstrauss Lemma

The Kaczmarz method is an algorithm for finding the solution to an overdetermined consistent system of linear equations Ax?=?b by iteratively projecting onto the solution spaces. The randomized version put forth by Strohmer and Vershynin yields provably exponential convergence in expectation, which for highly overdetermined systems even outperforms the conjugate gradient method. In this article we present a modified version of the randomized Kaczmarz method which at each iteration selects the optimal projection from a randomly chosen set, which in most cases significantly improves the convergence rate. We utilize a Johnson---Lindenstrauss dimension reduction technique to keep the runtime on the same order as the original randomized version, adding only extra preprocessing time. We present a series of empirical studies which demonstrate the remarkable acceleration in convergence to the solution using this modified approach.

Randomized Kaczmarz solver for noisy linear systems

The Kaczmarz method is an iterative algorithm for solving systems of linear equations Ax=b. Theoretical convergence rates for this algorithm were largely unknown until recently when work was done on a randomized version of the algorithm. It was proved that for overdetermined systems, the randomized Kaczmarz method converges with expected exponential rate, independent of the number of equations in the system. Here we analyze the case where the system Ax=b is corrupted by noise, so we consider the system where Ax is approximately b + r where r is an arbitrary error vector. We prove that in this noisy version, the randomized method reaches an error threshold dependent on the matrix A with the same rate as in the error-free case. We provide examples showing our results are sharp in the general context.

Comments on the Randomized Kaczmarz Method

In this short note we respond to some concerns raised by Y. Censor, G. Herman, and M. Jiang about the randomized Kaczmarz method that we proposed in [5]. The Kaczmarz method is a well-known iterative algorithm for solving a linear system of equations Ax = b. For more than seven decades, this method was useful in practical applications, and it was studied in many research papers. Despite of this, little is known about the rate of convergence of this method. The classical scheme of Kaczmarz’s method sweeps through the rows of A in a cyclic manner, projecting in each substep the last iterate orthogonally onto a hyperplane associated with a row of A. One variation of Kaczmarz’s method consists of randomly choosing in each iteration the row for the projection. Our algorithm in [5] (labeled Algorithm 1 there) is based on this approach. The idea of choosing the rows randomly is certainly not new. It has been mentioned for instance by Natterer [4], and later also by Feichtinger et al. [1] and by G. Herman and L. Meyer [3]. In these papers, improvement of performance is observed in numerical experiments, but none of these papers contain any proof of the rate of convergence. We consider the main contribution of [5] that (i) it contains the first proof for a rate of convergence for the Kaczmarz’s method that is applicable to general matrices (and not just to very restricted special cases); (ii) the algorithm achieves an exponential rate of convergence; and (iii) the rate