Sparse QR Factorization Research Articles

Experiments with Active-Set LP Algorithms Allowing Basis Deficiency

An interesting question for linear programming (LP) algorithms is how to deal with solutions in which the number of nonzero variables is less than the number of rows of the matrix in standard form. An approach is that of basis deficiency-allowing (BDA) simplex variations, which work with a subset of independent columns of the coefficient matrix in standard form, wherein the basis is not necessarily represented by a square matrix. We describe one such algorithm with several variants. The research question deals with studying the computational behaviour by using small, extreme cases. For these instances, we must wonder which parameter setting or variants are more appropriate. We compare the setting of two nonsimplex active-set methods with Holmström’s TomLab LpSimplex v3.0 commercial sparse primal simplex commercial implementation. All of them update a sparse QR factorization in Matlab. The first two implementations require fewer iterations and provide better solution quality and running time.

Large-Scale Optimization with Linear Equality Constraints Using Reduced Compact Representation

For optimization problems with linear equality constraints, we prove that the (1,1) block of the inverse KKT matrix remains unchanged when projected onto the nullspace of the constraint matrix. We develop reduced compact representations of the limited-memory inverse BFGS Hessian to compute search directions efficiently when the constraint Jacobian is sparse. Orthogonal projections are implemented by a sparse QR factorization or a preconditioned LSQR iteration. In numerical experiments two proposed trust-region algorithms improve in computation times, often significantly, compared to previous implementations of related algorithms and compared to IPOPT.

A regularized interior-point method for constrained linear least squares

We propose an infeasible interior-point algorithm for constrained linear least-squares problems based on the primal-dual regularization of convex programs of Friedlander and Orban. Regularization allows us to dispense with the assumption that the active gradients are linearly independent. At each iteration, a linear system with a symmetric quasi-definite (SQD) matrix is solved. This matrix is shown to be uniformly bounded and nonsingular. While the linear system may be solved using sparse LDL factorization, we observe that other approaches may be used. In particular, we build on the connection between SQD linear systems and unconstrained linear least-squares problems to solve the linear system with sparse QR factorization. We establish conditions under which a solution of the original, constrained least-squares problem is recovered. We report computational experience with the sparse QR factorization and illustrate the potential advantages of our approach.

A refinement of an iterative orthogonal projection method

The Kaczmarz algorithm is an iterative orthogonal projection method for solving linear systems of equations. As compared to direct methods such as Gaussian elimination or sparse QR-factorization, this algorithm is efficient for problems with sparse matrices, as they appear in constraint-based User Interface (UI) layout specifications. We present a variant of the Kaczmarz method for solving non-square systems that can be applied to Graphical User Interface (GUI) layout problems. In its original form the Kaczmarz algorithm cannot handle soft constraints. Therefore, we propose two algorithms for handling specifications containing soft constraints using prioritized irreducible infeasible subsystem (IIS) detection and prioritized grouping constraints. If we use Kaczmarz during resizing of a window in a GUI then the system can also be under-determined. In this case, space is not distributed in an aesthetically pleasing way. To distribute the space according to the preferred size of the layout, we introduce the least squares Kaczmarz method to get the desired results. The performance and convergence of the proposed algorithms are evaluated empirically using randomly generated UI layout specifications of various sizes. The results show that these methods outperform Matlab’s LINPROG, a well-known efficient linear programming solver.

Sensitivity Analysis for Nonlinear Programming in CasADi

We present an extension of the CasADi numerical optimization framework that allows arbitrary order NLP sensitivities to be calculated automatically and efficiently. The approach, which can be used together with any NLP solver available in CasADi, is based on a sparse QR factorization and an implementation of a primal-dual active set method. The whole toolchain is freely available as open-source software and allows generation of thread-safe, self-contained C code with small memory footprint. We illustrate the toolchain using three examples; a sparse QP, an optimal control problem and a parameter estimation problem.

Wilkinson's inertia‐revealing factorization and its application to sparse matrices

SummaryWe propose a new inertia‐revealing factorization for sparse symmetric matrices. The factorization scheme and the method for extracting the inertia from it were proposed in the 1960s for dense, banded, or tridiagonal matrices, but they have been abandoned in favor of faster methods. We show that this scheme can be applied to any sparse symmetric matrix and that the fill in the factorization is bounded by the fill in the sparse QR factorization of the same matrix (but is usually much smaller). We describe our serial proof‐of‐concept implementation and present experimental results, studying the method's numerical stability and performance.

Hybrid multilevel solution of sparse least-squares linear systems

PurposeLarge sparse least-squares problems arise in different scientific disciplines such as optimization, data analysis, machine learning and simulation. This paper aims to propose a two-level hybrid direct-iterative scheme, based on novel block independent column reordering, for efficiently solving large sparse least-squares linear systems.Design/methodology/approachHerewith, a novel block column independent set reordering scheme is used to separate the columns in two groups: columns that are block independent and columns that are coupled. The permutation scheme leads to a two-level hierarchy. Using this two-level hierarchy, the solution of the least-squares linear system results in the solution of a reduced size Schur complement-type square linear system, using the preconditioned conjugate gradient (PCG) method as well as backward substitution using the upper triangular factor, computed through sparse Q-less QR factorization of the columns that are block independent. To improve the convergence behavior of the PCG method, the upper triangular factor, computed through sparse Q-less QR factorization of the coupled columns, is used as a preconditioner. Moreover, to further reduce the fill-in, then the column approximate minimum degree (COLAMD) algorithm is used to permute the block consisting of the coupled columns.FindingsThe memory requirements for solving large sparse least-squares linear systems are significantly reduced compared to Q-less QR decomposition of the original as well as the permuted problem with COLAMD. The memory requirements are reduced further by choosing to form larger blocks of independent columns. The convergence behavior of the iterative scheme is improved due to the chosen preconditioning scheme. The proposed scheme is inherently parallel due to the introduction of block independent column reordering.Originality/valueThe proposed scheme is a hybrid direct-iterative approach for solving sparse least squares linear systems based on the implicit computation of a two-level approximate pseudo-inverse matrix. Numerical results indicating the applicability and effectiveness of the proposed scheme are given.

Algorithm 980

Sparse matrix factorization involves a mix of regular and irregular computation, which is a particular challenge when trying to obtain high-performance on the highly parallel general-purpose computing cores available on graphics processing units (GPUs). We present a sparse multifrontal QR factorization method that meets this challenge and is significantly faster than a highly optimized method on a multicore CPU. Our method factorizes many frontal matrices in parallel and keeps all the data transmitted between frontal matrices on the GPU. A novel bucket scheduler algorithm extends the communication-avoiding QR factorization for dense matrices by exploiting more parallelism and by exploiting the staircase form present in the frontal matrices of a sparse multifrontal method.

Towards using direct methods in seismic tomography: computation of the full resolution matrix using high-performance computing and sparse QR factorization

Towards using direct methods in seismic tomography: computation of the full resolution matrix using high-performance computing and sparse QR factorization

A Fast Semidirect Least Squares Algorithm for Hierarchically Block Separable Matrices

We present a fast algorithm for linear least squares problems governed by hierarchically block separable (HBS) matrices. Such matrices are generally dense but data-sparse and can describe many important operators including those derived from asymptotically smooth radial kernels that are not too oscillatory. The algorithm is based on a recursive skeletonization procedure that exposes this sparsity and solves the dense least squares problem as a larger, equality-constrained, sparse one. It relies on a sparse QR factorization coupled with iterative weighted least squares methods. In essence, our scheme consists of a direct component, comprised of matrix compression and factorization, followed by an iterative component to enforce certain equality constraints. At most two iterations are typically required for problems that are not too ill-conditioned. For an $M \times N$ HBS matrix with $M \geq N$ having bounded off-diagonal block rank, the algorithm has optimal $\mathcal{O} (M + N)$ complexity. If the rank increases with the spatial dimension as is common for operators that are singular at the origin, then this becomes $\mathcal{O} (M + N)$ in 1D, $\mathcal{O} (M + N^{3/2})$ in 2D, and $\mathcal{O} (M + N^{2})$ in 3D. We illustrate the performance of the method on both over- and underdetermined systems in a variety of settings, with an emphasis on radial basis function approximation and efficient updating and downdating.

Algorithm 915, SuiteSparseQR

SuiteSparseQR is a sparse QR factorization package based on the multifrontal method. Within each frontal matrix, LAPACK and the multithreaded BLAS enable the method to obtain high performance on multicore architectures. Parallelism across different frontal matrices is handled with Intel's Threading Building Blocks library. The symbolic analysis and ordering phase pre-eliminates singletons by permuting the input matrix A into the form [ R 11 R 12 ; 0 A 22 ] where R 11 is upper triangular with diagonal entries above a given tolerance. Next, the fill-reducing ordering, column elimination tree, and frontal matrix structures are found without requiring the formation of the pattern of A TA . Approximate rank-detection is performed within each frontal matrix using Heath's method. While Heath's method is not always exact, it has the advantage of not requiring column pivoting and thus does not interfere with the fill-reducing ordering. For sufficiently large problems, the resulting sparse QR factorization obtains a substantial fraction of the theoretical peak performance of a multicore computer.

A sparse counterpart of Reichel and Gragg’s package QRUP

We describe how to maintain the triangular factor of a sparse QR factorization when columns are added and deleted and Q cannot be stored for sparsity reasons. The updating procedures could be thought of as a sparse counterpart of Reichel and Gragg’s package QRUP. They allow us to solve a sequence of sparse linear least squares subproblems in which each matrix B k is an independent subset of the columns of a fixed matrix A , and B k + 1 is obtained by adding or deleting one column. Like Coleman and Hulbert [T. Coleman, L. Hulbert, A direct active set algorithm for large sparse quadratic programs with simple bounds, Math. Program. 45 (1989) 373–406], we adapt the sparse direct methodology of Björck and Oreborn of the late 1980s, but without forming A T A , which may be only positive semidefinite. Our Matlab 5 implementation works with a suitable row and column numbering within a static triangular sparsity pattern that is computed in advance by symbolic factorization of A T A and preserved with placeholders.

Schur Factorization of the Impedance Matrix in a Localizing Basis

A Schur-based factorization procedure is described for impedance matrices transformed into a localizing basis. A recently reported factorization strategy relying on a sparse QR factorization of the transformed impedance matrix was shown to exhibit improved efficiency relative to standard direct factorizations. It is shown here that a Schur factorization provides additional efficiencies.

Updating and downdating an upper trapezoidal sparse orthogonal factorization†

We describe how to update and downdate an upper trapezoidal sparse orthogonal factorization, namely the sparse QR factorization of A T k where A k is a 'tall and thin' full column rank matrix formed with a subset of the columns of a fixed matrix A. In order to do this, we have adapted Saunders' techniques of the early 1970s for square matrices, to rectangular matrices (with fewer columns than rows) by using the static data structure of George and Heath of the early 1980s but allowing row downdating on it. An implicitly determined column permutation allows us to dispense with computing a new ordering after each update/downdate; it fits well into the LINPACK downdating algorithm and ensures that the updated trapezoidal factor will remain sparse. We give all the necessary formulae even if the orthogonal factor is not available, and we comment on our implementation using the sparse toolbox of MATLAB 5.

Algorithm 836

Two codes are discussed, COLAMD and SYMAMD, that compute approximate minimum degree orderings for sparse matrices in two contexts: (1) sparse partial pivoting, which requires a sparsity preserving column pre-ordering prior to numerical factorization, and (2) sparse Cholesky factorization, which requires a symmetric permutation of both the rows and columns of the matrix being factorized. These orderings are computed by COLAMD and SYMAMD, respectively. The ordering from COLAMD is also suitable for sparse QR factorization, and the factorization of matrices of the form A T A and AA T , such as those that arise in least-squares problems and interior point methods for linear programming problems. The two routines are available both in MATLAB and C-callable forms. They appear as built-in routines in MATLAB Version 6.0.

A software package for sparse orthogonal factorization and updating

Although there is good software for sparse QR factorization, there is little support for updating and downdating, something that is absolutely essential in some linear programming algorithms, for example. This article describes an implementation of sparse LQ factorization, including block triangularization, approximate minimum degree ordering, symbolic factorization, multifrontal factorization, and updating and downdating. The factor Q is not retained. The updating algorithm expands the nonzero pattern of the factor L, which is reflected in dynamic representation of L. The block triangularization is used as an "ordering for sparsity" rather than as a prerequisite for block backward substitution. In symbolic factorization, something called "element counters" is introduced to reduce the overestimation of the number of nonzeros that the commonly used methods do. Both the approximate minimum degree ordering and the symbolic factorization are done without explicitly forming the nonzero pattern of the symmetric matrix in the corresponding normal equations. Tests show that the average time used for a single update or downdate is essentially the same as the time used for a single forward or backward substitution. Other parts of the implementation show the same range of performance as existing code, but cannot be replaced because of the special character of the systems that are solved.

Computing row and column counts for sparse QR and LU factorization

We present algorithms to determine the number of nonzeros in each row and column of the factors of a sparse matrix, for both the QR factorization and the LU factorization with partial pivoting. The algorithms use only the nonzero structure of the input matrix, and run in time nearly linear in the number of nonzeros in that matrix. They may be used to set up data structures or schedule parallel operations in advance of the numerical factorization. The row and column counts we compute are upper bounds on the actual counts. If the input matrix is strong Hall and there is no coincidental numerical cancellation, the counts are exact for QR factorization and are the tightest bounds possible for LU factorization. These algorithms are based on our earlier work on computing row and column counts for sparse Cholesky factorization, plus an efficient method to compute the column elimination tree of a sparse matrix without explicitly forming the product of the matrix and its transpose.

Exact Prediction of QR Fill-In by Row-Merge Trees

Row-merge trees for forming the QR factorization of a sparse matrix A are closely related to elimination trees for the Cholesky factorization of AT A. Row-merge trees predict the exact fill-in (assuming no numerical cancellation) provided A satisfies the strong Hall property, but overestimate the fill-in in general. However, here a fast and simple postprocessing step for row-merge trees is presented that predicts the exact fill-in for sparse QR factorization using Householder reflectors for general matrices.

A coarse-grained parallel QR-factorization algorithm for sparse least squares problems

A sparse QR-factorization algorithm SPARQR for coarse-grained parallel computations is described. The coefficient matrix, which is assumed to be general sparse, is reordered in an attempt to bring as many zero elements in the lower left corner as possible. The reordered matrix is then partitioned into block rows, and Givens plane rotations are applied in each block-row. These are independent tasks and can be done in parallel. Row and column permutations are carried out within the diagonal blocks in an attempt to preserve better the sparsity of the matrix. The algorithm can be used for solving least squares problems either directly or combined with an iterative method (preconditioned conjugate gradients are used). Small non-zero elements can optionally be dropped in the latter case. This leads to a better preservation of the sparsity and, therefore, to a faster factorization. The price which has to be paid is some loss of accuracy. The iterative method is used to regain the accuracy lost during the factorization. Numerical results from several experiments with matrices from the well-known Harwell–Boeing collection as well as with some larger sparse matrices are presented in this work. An SGI Power Challenge computer with 16 processors has been used in the experiments.

Parallel solution of sparse linear least squares problems on distributed-memory multiprocessors

This paper studies the parallel solution of large-scale sparse linear least squares problems on distributed-memory multiprocessors. The key components required for solving a sparse linear least squares problem are sparse QR factorization and sparse triangular solution. A block-oriented parallel algorithm for sparse QR factorization has already been described in the literature. In this paper, new block-oriented parallel algorithms for sparse triangular solution are proposed. The arithmetic and communication complexities of the new algorithms applied to regular grid problems are analyzed. The proposed parallel sparse triangular solution algorithms together with the block-oriented parallel sparse QR factorization algorithm result in a highly efficient approach to the parallel solution of sparse linear least squares problems. Performance results obtained on an IBM Scalable POWERparallel system SP2 are presented. The largest least squares problem solved has over two million rows and more than a quarter million columns. The execution speed for the numerical factorization of this problem achieves over 3.7 gigaflops per second on an IBM SP2 machine with 128 processors.