Kaczmarz Method Research Articles

A class of residual-based extended Kaczmarz methods for solving inconsistent linear systems

In this paper, we propose four extended Kaczmarz methods: two partially randomized methods like probability proportional to residual or residual homogenizing, and two deterministic strategies with the maximal-residual control and the maximum-distance control. Without the full column rank and overdetermined assumptions on linear systems, we provide a thorough convergence analysis in terms of expectation, and derive upper bounds for the expected convergence rates of the new extended Kaczmarz methods. Numerical experiments on Gaussian models as well as 2D image reconstruction problems demonstrate that the new extended Kaczmarz methods can be much more effective than the existing ones. Especially, the improvements of two deterministic strategies are very prominent.

On Convergence of the Partially Randomized Extended Kaczmarz Method

On Convergence of the Partially Randomized Extended Kaczmarz Method

Linear Convergence of Randomized Kaczmarz Method for Solving Complex-Valued Phaseless Equations

A randomized Kaczmarz method was recently proposed for phase retrieval, which has been shown numerically to exhibit empirical performance over other state-of-the-art phase retrieval algorithms both in terms of the sampling complexity and in terms of computation time. While the rate of convergence has been studied well in the real case where the signals and measurement vectors are all real-valued, there is no guarantee for the convergence in the complex case. In fact, the linear convergence of the randomized Kaczmarz method for phase retrieval in the complex setting is left as a conjecture by Tan and Vershynin. In this paper, we provide the first theoretical guarantees for it. We show that for random measurements $\mathbf{a}_j \in \mathbb{C}^n, j=1,\ldots,m $ which are drawn independently and uniformly from the complex unit sphere, or equivalent are independent complex Gaussian random vectors, when $m \ge Cn$ for some universal positive constant $C$, the randomized Kaczmarz scheme with a good initialization converges linearly to the target solution (up to a global phase) in expectation with high probability. This gives a positive answer to that conjecture.

Kaczmarz Anomaly in Tomography Problems

The Kaczmarz method is an important tool for solving large sparse linear systems that arise in computerized tomography. The Kaczmarz anomaly phenomenon has been observed recently when solving certain types of random systems. This raises the question of whether a similar anomaly occurs in tomography problems. The aim of the paper is to answer this question, to examine the extent of the phenomenon and to explain its reasons. Another tested issue is the ability of random row shuffles to sharpen the anomaly and to accelerate the rate of convergence. The results add important insight into the nature of the Kaczmarz method.

On greedy randomized average block Kaczmarz method for solving large linear systems

Inspired by the greedy randomized Kaczmarz method, we propose a probability criterion which can capture subvectors of the residual whose norms are relatively large. According to this probability criterion we select a submatrix randomly from the coefficient matrix, then average the projections of the current iteration vector onto each individual row of this chosen submatrix, constructing the greedy randomized average block Kaczmarz method for solving the consistent system of linear equations, which can be implemented in a distributed environment. When the size of each block is one, the probability criterion in the greedy randomized average block Kaczmarz method is a generalization of that in the greedy randomized Kaczmarz method. The greedy randomized Kaczmarz method is also a special case of the greedy randomized average block Kaczmarz method. Two kinds of extrapolated stepsizes for the greedy randomized average block Kaczmarz method are analyzed. The experimental results show the advantage of the greedy randomized average block Kaczmarz method over the greedy randomized Kaczmarz method and several existing randomized block Kaczmarz methods.

Adaptively sketched Bregman projection methods for linear systems

The sketch-and-project, as a general archetypal algorithm for solving linear systems, unifies a variety of randomized iterative methods such as the randomized Kaczmarz and randomized coordinate descent. However, since it aims to find a least-norm solution from a linear system, the randomized sparse Kaczmarz can not be included. This motivates us to propose a more general framework, called sketched Bregman projection (SBP) method, in which we are able to find solutions with certain structures from linear systems. To generalize the concept of adaptive sampling to the SBP method, we show how the progress, measured by Bregman distance, of single step depends directly on a sketched loss function. Theoretically, we provide detailed global convergence results for the SBP method with different adaptive sampling rules. At last, for the (sparse) Kaczmarz methods, a group of numerical simulations are tested, with which we verify that the methods utilizing sampling Kaczmarz–Motzkin rule demands the fewest computational costs to achieve a given error bound comparing to the corresponding methods with other sampling rules.

On the Kaczmarz methods based on relaxed greedy selection for solving matrix equation [formula omitted

For solving a consistent system of linear equations, the Kaczmarz method is a popular representative among iterative algorithms due to its simplicity and efficiency. Based on the Petrov–Galerkin conditions and the relaxed greedy selection strategy, we introduce two kinds of Kaczmarz-type methods to solve matrix equation AXB=C, including randomized and deterministic versions and establish the corresponding convergence theories. We prove that our algorithms all have exponential convergence rates. Then we apply our algorithms to real-world application, such as the tensor product surface fitting in computer-aided geometric design (CAGD), it is shown that our algorithms are more effective than some often used conventional matrix equation solvers.

Quantile-based Random Kaczmarz for corrupted linear systems of equations

Abstract We consider linear systems $Ax = b$ where $A \in \mathbb{R}^{m \times n}$ consists of normalized rows, $\|a_i\|_{\ell ^2} = 1$, and where up to $\beta m$ entries of $b$ have been corrupted (possibly by arbitrarily large numbers). Haddock, Needell, Rebrova & Swartworth propose a quantile-based Random Kaczmarz method and show that for certain random matrices $A$ it converges with high likelihood to the true solution. We prove a deterministic version by constructing, for any matrix $A$, a number $\beta _A$ such that there is convergence for all perturbations with $\beta < \beta _A$. Assuming a random matrix heuristic, this proves convergence for tall Gaussian matrices with up to $\sim 0.5\%$ corruption (a number that can likely be improved).

On extended Kaczmarz methods with random sampling and maximum-distance for solving large inconsistent linear systems

In this paper, combining random sampling and maximum-distance, we propose two new extended Kaczmarz methods for solving inconsistent linear systems with full column rank. Based on some inequalities, we prove the convergences of the new extended Kaczmarz methods and give the upper bounds of the expected convergence rates. Numerical experiments show that the two new extended Kaczmarz methods are more effective than the existing methods. Especially, the new method with Z-test has certain advantage over that without it.

Incremental Nonnegative Tucker Decomposition with Block-Coordinate Descent and Recursive Approaches

Nonnegative Tucker decomposition (NTD) is a robust method used for nonnegative multilinear feature extraction from nonnegative multi-way arrays. The standard version of NTD assumes that all of the observed data are accessible for batch processing. However, the data in many real-world applications are not static or are represented by a large number of multi-way samples that cannot be processing in one batch. To tackle this problem, a dynamic approach to NTD can be explored. In this study, we extend the standard model of NTD to an incremental or online version, assuming volatility of observed multi-way data along one mode. We propose two computational approaches for updating the factors in the incremental model: one is based on the recursive update model, and the other uses the concept of the block Kaczmarz method that belongs to coordinate descent methods. The experimental results performed on various datasets and streaming data demonstrate high efficiently of both algorithmic approaches, with respect to the baseline NTD methods.

Greedy randomized and maximal weighted residual Kaczmarz methods with oblique projection

<abstract><p>For solving large-scale consistent linear system, a greedy randomized Kaczmarz method with oblique projection and a maximal weighted residual Kaczmarz method with oblique projection are proposed. By using oblique projection, these two methods greatly reduce the number of iteration steps and running time to find the minimum norm solution, especially when the rows of matrix are highly linearly correlated. Theoretical proof and numerical results show that the greedy randomized Kaczmarz method with oblique projection and the maximal weighted residual Kaczmarz method with oblique projection are more effective than the greedy randomized Kaczmarz method and the maximal weighted residual Kaczmarz method respectively.</p></abstract>

A Count Sketch Maximal Weighted Residual Kaczmarz Method with Oblique Projection for Highly Overdetermined Linear Systems

Motivated by the count sketch maximal weighted residual Kaczmarz (CS-MWRK) method presented by Zhang and Li (Appl. Math. Comput., 410, 126486), we combine the count sketch tech with the maximal weighted residual Kaczmarz Method with Oblique Projection (MWRKO) constructed by Wang, Li, Bao and Liu (arXiv: 2106.13606) to develop a new method for solving highly overdetermined linear systems. The convergence rate of the new method is analyzed. Numerical results demonstrate that our method performs better in computing time compared with the CS-MWRK and MWRKO methods.

A greedy average block Kaczmarz method for the large scaled consistent system of linear equations

<abstract><p>This paper presents a greedy average block Kaczmarz (GABK) method to solve the large scaled consistent system of linear equations. The GABK method introduces the strategy of extrapolation process to improve the GBK algorithm and to avoid computing the Moore-Penrose inverse of a submatrix of the coefficient matrix determined by the block index set. The GABK method is proved to converge linearly to the least-norm solution of the consistent system of linear equations. Numerical examples show that the GABK method has the best efficiency and effectiveness among all methods compared.</p></abstract>

On Relaxed Greedy Randomized Iterative Methods for the Solution of Factorized Linear Systems

RK-RK and REK-RK methods are the two latest while very effective randomized iteration solvers for factorized linear system $UV \beta = y$ by interlacing randomized Kaczmarz (RK) and randomized extended Kaczmarz (REK) updates. This paper considers two latest randomized iterative methods for solving large-scale linear systems and linear least-squares problems—greedy randomized Kaczmarz (GRK) method and greedy randomized Gauss–Seidel (GRGS) method. By introducing a relaxation parameter $\omega$ into the iterates of GRK and GRGS, we construct relaxed GRK and GRGS methods, respectively. In addition, by interlacing their updates, we propose relaxed GRK-GRK and GRGS-GRK methods to solve consistent and inconsistent factorized linear systems, respectively. We prove the exponential convergence of these two interlaced methods and show that relaxed GRK-GRK and GRGS-GRK can be more efficient than RK-RK and REK-RK, respectively, if the relaxation parameters are chosen appropriately.

On global randomized block Kaczmarz method for image reconstruction

<abstract><p>Image reconstruction represents an important technique applied in various fields such as medicine, biology, materials science, nondestructive testing, and so forth. In this paper, we transform the problem of image reconstruction into the problem of solving linear systems with multiple right-hand sides. Based on the idea of K-means clustering, we propose the global randomized block Kaczmarz method, so as to solve the problem of the linear systems with multiple right-hand sides effectively and use this method to image reconstruction. Theoretical analysis proves the convergence of this method, and the simulation results demonstrate the performance of this method in image reconstruction.</p></abstract>

On stochastic Kaczmarz type methods for solving large scale systems of ill-posed equations

In this article we investigate a family of stochastic gradient type methods for solving systems of linear ill-posed equations. The method under consideration is a stochastic version of the projective Landweber–Kaczmarz method in Leitão and Svaiter (2016 Inverse Problems 32 025004) (see also Leitão and Svaiter (2018 Numer. Funct. Anal. Optim. 39 1153–80)). In the case of exact data, mean square convergence to zero of the iteration error is proven. In the noisy data case, we couple our method with an a priori stopping rule and characterize it as a regularization method for solving systems of linear ill-posed operator equations. Numerical tests are presented for two linear ill-posed problems: (i) a Hilbert matrix type system with over 108 equations; (ii) a big data linear regression problem with real data. The obtained results indicate superior performance of the proposed method when compared with other well-established random iterations. Our preliminary investigation indicates that the proposed iteration is a promising alternative for computing stable approximate solutions of large scale systems of linear ill-posed equations.

Greedy Motzkin–Kaczmarz methods for solving linear systems

AbstractThe famous greedy randomized Kaczmarz (GRK) method uses the greedy selection rule on maximum distance to determine a subset of the indices of working rows. In this paper, with the greedy selection rule on maximum residual, we propose the greedy randomized Motzkin–Kaczmarz (GRMK) method for linear systems. The block version of the new method is also presented. We analyze the convergence of the two methods and provide the corresponding convergence factors. The computational complexities of the proposed methods are also given. Extensive numerical experiments show that the GRMK method has almost the same performance as the GRK method for dense matrices and the former performs better in computing time for some sparse matrices and some realistic practical problems, and their block versions have almost the same performance for most of cases.

Sparse sampling Kaczmarz–Motzkin method with linear convergence

The randomized sparse Kaczmarz method was recently proposed to recover sparse solutions of linear systems. In this work, we introduce a greedy variant of the randomized sparse Kaczmarz method by employing the sampling Kaczmarz–Motzkin method and prove its linear convergence in expectation with respect to the Bregman distance in the noiseless and noisy cases. This greedy variant can be viewed as a unification of the sampling Kaczmarz–Motzkin method and the randomized sparse Kaczmarz method, and hence inherits the merits of these two methods. Numerically, we report a couple of experimental results to demonstrate its superiority.

On the generally randomized extended Gauss-Seidel method

The randomized extended Gauss-Seidel method is a popular representative among the iterative algorithm due to its simplicity for solving the inconsistent and consistent systems of linear equations, which builds the connection between the randomized Kaczmarz and Gauss-Seidel methods. In this work we develop a general version of the randomized extended Gauss-Seidel method, as well as some new iterative schemes. We prove that our algorithm can exponentially converge in expectation to the solutions of the consistent or inconsistent linear systems under two different sampling strategies. Numerical examples show that the proposed algorithm is feasible and effective, where the block method performs significantly better than the corresponding original form.

Randomized block Kaczmarz methods with [formula omitted]-means clustering for solving large linear systems

Following the philosophy of the block Kaczmarz methods, we propose a randomized block Kaczmarz method with the blocks determined by the k-means clustering (RBK(k)). It can be considered as an efficient variant of the relaxed greedy randomized Kaczmarz algorithm by using a practical probability criterion for selecting the working block submatrix per iteration. The new algorithm is proved to be convergent when the linear system is consistent. A practical variant of the new method is also given. Some numerical examples are given to verify the effectiveness of the proposed methods.