Low-rank Matrix Approximation Research Articles

Pass-efficient randomized LU algorithms for computing low-rank matrix approximation

Abstract Low-rank matrix approximation is extremely useful in the analysis of data that arises in scientific computing, engineering applications, and data science. However, as data sizes grow, traditional low-rank matrix approximation methods, such as singular value decomposition (SVD) and column pivoting QR decomposition (CPQR), are either prohibitively expensive or cannot provide sufficiently accurate results. A solution is to use randomized low-rank matrix approximation methods such as randomized SVD, and randomized LU decomposition on extremely large data sets. In this paper, we focus on the randomized LU decomposition method. Then we propose a novel randomized LU algorithm, called SubspaceLU, for the fixed low-rank approximation problem. SubspaceLU is based on the sketch of the co-range of input matrices and allows for an arbitrary number of passes of the input matrix, v ≥ 2 {v\geq 2} . Numerical experiments on CPU show that our proposed SubspaceLU is generally faster than the existing randomized LU decomposition, while remaining accurate. Experiments on GPU shows that our proposed SubspaceLU can gain more speedup than the existing randomized LU decomposition. We also propose a version of SubspaceLU, called SubspaceLU_FP, for the fixed precision low-rank matrix approximation problem. SubspaceLU_FP is a post-processing step based on an efficient blocked adaptive rank determination Algorithm 5 proposed in this paper. We present numerical experiments that show that SubspaceLU_FP can achieve close results to SVD but faster in speed. We finally propose a single-pass algorithm based on LU factorization. Tests show that the accuracy of our single-pass algorithm is comparable with the existing single-pass algorithms.

Using low-rank approximations of gridded data for spline surface fitting

The paper is devoted to the problem of finding bivariate tensor-product spline (or similar) functions that approximate gridded data in the least-squares sense. We propose to apply a low rank approximation of matrices to the data, to find the result by solving a sequence of univariate fitting problems that can be handled efficiently. This can be seen as a generalization of the method proposed by Georgieva and Hofreither (2017) that combines cross approximation (which is a particular method for low rank matrix approximation) with spline interpolation.While the algorithm yields the best least-squares approximation after r steps, where r denotes the rank of the data matrix, terminating it earlier yields a low-rank approximation, which often provides a sufficient level of accuracy. We also present a stopping criterion (based on a lower error estimate) that allows to use the method efficiently in the situation when the required number of degrees of freedom is not known in advance.

Sparse low-rank approximation of matrix and local preservation for unsupervised image feature selection

Sparse low-rank approximation of matrix and local preservation for unsupervised image feature selection

Generalized Matrix Local Low Rank Representation by Random Projection and Submatrix Propagation.

Matrix low rank approximation is an effective method to reduce or eliminate the statistical redundancy of its components. Compared with the traditional global low rank methods such as singular value decomposition (SVD), local low rank approximation methods are more advantageous to uncover interpretable data structures when clear duality exists between the rows and columns of the matrix. Local low rank approximation is equivalent to low rank submatrix detection. Unfortunately, existing local low rank approximation methods can detect only submatrices of specific mean structure, which may miss a substantial amount of true and interesting patterns. In this work, we develop a novel matrix computational framework called RPSP (Random Probing based submatrix Propagation) that provides an effective solution for the general matrix local low rank representation problem. RPSP detects local low rank patterns that grow from small submatrices of low rank property, which are determined by a random projection approach. RPSP is supported by theories of random projection. Experiments on synthetic data demonstrate that RPSP outperforms all state-of-the-art methods, with the capacity to robustly and correctly identify the low rank matrices when the pattern has a similar mean as the background, background noise is heteroscedastic and multiple patterns present in the data. On real-world datasets, RPSP also demonstrates its effectiveness in identifying interpretable local low rank matrices.

Effective elastic properties of three-dimensional multiple crack problems with the isogeometric boundary element parallel fast direct solver

We propose an isogeometric boundary element (IGABEM) parallel fast direct solver based on OpenMP to solve three-dimensional multiple crack problems. In this paper, the orderly and randomly distributed cracks are studied to observe the influence of the multiple crack interaction on the effective elastic properties. This paper designs the parallel implementation of the matrix low rank approximation and the matrix assembly for the hierarchical off-diagonal low rank (HODLR) matrix. In the matrix low rank approximation, we propose the m-n tree to improve the accelerated hybrid algorithm to balance the CPU performance and memory usage. The numerical results show that the effective elastic properties can respond to the strong interaction to a certain extent, but the maximum stress intensity factors (SIFs) of multiple cracks still need to be calculated to diagnose the possible crack initiation points. On the other hand, the numerical results also show that the IGABEM parallel fast direct solver can greatly improve the computational efficiency on the premise of ensuring the computational accuracy.

Hybrid Threshold Denoising Framework Using Singular Value Decomposition for Side-Channel Analysis Preprocessing.

The traces used in side-channel analysis are essential to breaking the key of encryption and the signal quality greatly affects the correct rate of key guessing. Therefore, the preprocessing of side-channel traces plays an important role in side-channel analysis. The process of side-channel leakage signal acquisition is usually affected by internal circuit noise, external environmental noise, and other factors, so the collected signal is often mixed with strong noise. In order to extract the feature information of side-channel signals from very low signal-to-noise ratio traces, a hybrid threshold denoising framework using singular value decomposition is proposed for side-channel analysis preprocessing. This framework is based on singular value decomposition and introduces low-rank matrix approximation theory to improve the rank selection methods of singular value decomposition. This paper combines the hard threshold method of truncated singular value decomposition with the soft threshold method of singular value shrinkage damping and proposes a hybrid threshold denoising framework using singular value decomposition for the data preprocessing step of side-channel analysis as a general preprocessing method for non-profiled side-channel analysis. The data used in the experimental evaluation are from the raw traces of the public database of DPA contest V2 and AES_HD. The success rate curve of non-profiled side-channel analysis further confirms the effectiveness of the proposed framework. Moreover, the signal-to-noise ratio of traces is significantly improved after preprocessing, and the correlation with the correct key is also significantly enhanced. Experimental results on DPA v2 and AES_HD show that the proposed noise reduction framework can be effectively applied to the side-channel analysis preprocessing step, and can successfully improve the signal-to-noise ratio of the traces and the attack efficiency.

Hierarchical off-diagonal low-rank approximation of Hessians in inverse problems, with application to ice sheet model initialization

Obtaining lightweight and accurate approximations of discretized objective functional Hessians in inverse problems governed by partial differential equations (PDEs) is essential to make both deterministic and Bayesian statistical large-scale inverse problems computationally tractable. The cubic computational complexity of dense linear algebraic tasks, such as Cholesky factorization, that provide a means to sample Gaussian distributions and determine solutions of Newton linear systems is a computational bottleneck at large-scale. These tasks can be reduced to log-linear complexity by utilizing hierarchical off-diagonal low-rank (HODLR) matrix approximations. In this work, we show that a class of Hessians that arise from inverse problems governed by PDEs are well approximated by the HODLR matrix format. In particular, we study inverse problems governed by PDEs that model the instantaneous viscous flow of ice sheets. In these problems, we seek a spatially distributed basal sliding parameter field such that the flow predicted by the ice sheet model is consistent with ice sheet surface velocity observations. We demonstrate the use of HODLR Hessian approximation to efficiently sample the Laplace approximation of the posterior distribution with covariance further approximated by HODLR matrix compression. Computational studies are performed which illustrate ice sheet problem regimes for which the Gauss–Newton data-misfit Hessian is more efficiently approximated by the HODLR matrix format than the low-rank (LR) format. We then demonstrate that HODLR approximations can be favorable, when compared to global LR approximations, for large-scale problems by studying the data-misfit Hessian associated with inverse problems governed by the first-order Stokes flow model on the Humboldt glacier and Greenland ice sheet.

Denoising for airborne transient electromagnetic data using noise-whitening-based weighted nuclear norm minimization

Abstract Recovering anomalous information covered under noise in late gates can enhance airborne transient electromagnetic (ATEM) detection. Conventional denoising mainly comprises filtering and gate correlation-based decomposition algorithms; the former fails to extract anomalies contaminated by noise and the latter relies on the correlation between gates, which may yield false late gate anomalies caused by early large-amplitude anomalies in early gates. In ATEM profiles, the correlation between anomalies in adjacent gates makes the anomalies to be measured to have low-rank characteristics relative to the noise-contaminated profiles; the noise is uniformly distributed in the profiles, which have nonlocal self-similarity. Therefore, the low-rank matrix approximation algorithm is applicable to ATEM data denoising. In this study, an algorithm—noise-whitening-based weighted nuclear norm minimization (NW-WNNM)—is designed to remove ATEM profile noise. First, we analyze the influence of patch size in block matching on anomalous and noisy patches and estimate the profile patch size adaptively. Then, we combine the estimation of noise variance in weighted nuclear norm minimization (WNNM) with the noise whitening of similar patch matrices to reduce the noise interference on the nuclear norm and add a whitening factor in the weight vector to make the soft-thresholding function applicable to the low-rank reconstruction of the whitened matrix. By analyzing the reconstructed low-rank matrix and its feature distribution, compared with WNNM, NW-WNNM can detect the feature information more accurately and eliminate the influence of noise on the nuclear norm. Simulation and field profile results indicate that NW-WNNM is superior to comparison denoising methods.

Fast truncated SVD of sparse and dense matrices on graphics processors

We investigate the solution of low-rank matrix approximation problems using the truncated singular value decomposition (SVD). For this purpose, we develop and optimize graphics processing unit (GPU) implementations for the randomized SVD and a blocked variant of the Lanczos approach. Our work takes advantage of the fact that the two methods are composed of very similar linear algebra building blocks, which can be assembled using numerical kernels from existing high-performance linear algebra libraries. Furthermore, the experiments with several sparse matrices arising in representative real-world applications and synthetic dense test matrices reveal a performance advantage of the block Lanczos algorithm when targeting the same approximation accuracy.

Robust DOA Estimation Using VSS-LMS With Low-Rank Matrix Approximation

Direction-of-arrival (DOA) estimation performance of adaptive filtering-based methods, <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">e.g.</i> , least mean square (LMS), degrades significantly in adverse conditions ( <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">i.e.,</i> at low signal-to-noise ratio (SNR), small number of antenna array elements and snapshots, and in presence of two closely spaced signal sources). Moreover, these estimation methods require many regulating parameters to efficiently update the step size, which are very difficult to tune manually in practical scenarios. Although the subspace decomposition-based methods, <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">e.g</i> ., multiple signal classification (MUSIC), provide somewhat better performance in adverse conditions, they are based on the statistical properties of the signal such as spatial covariance matrix (SCM) and its eigenvalue decomposition (EVD), resulting in high computational complexity. In this paper, a robust DOA estimation method is proposed based on a variable step size LMS algorithm with low rank matrix approximation (LRMA), where the received signal de-noising problem is formulated first as a LRMA problem directly using the received signal observations instead of the SCM, and then the variable step size is calculated from the predicted instantaneous error and the estimated powers of reference and <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">de-noised</i> signals. The output spatial spectrum is determined by the reciprocal of the antenna array pattern, and the high peaks in the output spatial spectrum give the received signals estimated directions. The proposed method updates the step size efficiently without choosing any regulating parameter, and achieves improved performance even in adverse conditions due to the de-noising feature of LRMA with low computational complexity. Further, we also conduct the convergence and stability analyses of the proposed iterative estimation method with the <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">de-noised</i> signal. Numerical results demonstrate that the proposed method outperforms state-of-the-art methods especially those, which yielded based on adaptive filtering.

Robust Matrix Completion Method Based on TNNR and Total Row Difference for Recovering Optical Image

Matrix completion aims to recover a matrix from an incomplete matrix with many unknown elements and has wide applications in optical image recovery and machine learning, in which the popular method is to formulate it as a general low-rank matrix approximation problem. However, the traditional optimization model for matrix completion is less robust. This paper proposes a robust matrix completion method in which the truncated nuclear norm regularization (TNNR) is used as the approximation of the rank function and the sum of absolute values of the row difference, which is called as the total row difference, is used to constrain the oscillations of the missing matrix. By minimizing the value of total row difference in the objective, the proposed model controls the oscillation and reduces the impact of missing part in the process of matrix completion continuously. Furthermore, we propose a two-step iterative algorithm framework and design an ADMM algorithm for subproblem model that includes minimizing the total row difference. Experiments show that the proposed algorithm has more stable performance and better recovery effect and obviously reduces the sensitivity of the traditional TNNR models to the truncated rank parameter.

An Improved Analysis and Unified Perspective on Deterministic and Randomized Low-Rank Matrix Approximation

An Improved Analysis and Unified Perspective on Deterministic and Randomized Low-Rank Matrix Approximation

Sketching for a low-rank nonnegative matrix approximation: Numerical study

Abstract We propose new approximate alternating projection methods, based on randomized sketching, for the low-rank nonnegative matrix approximation problem: find a low-rank approximation of a nonnegative matrix that is nonnegative, but whose factors can be arbitrary. We calculate the computational complexities of the proposed methods and evaluate their performance in numerical experiments. The comparison with the known deterministic alternating projection methods shows that the randomized approaches are faster and exhibit similar convergence properties.

Accelerating self-consistent field iterations in Kohn-Sham density functional theory using a low-rank approximation of the dielectric matrix

We present an efficient preconditioning technique for accelerating the fixed point iteration in real-space Kohn-Sham density functional theory (DFT) calculations. The preconditioner uses a low rank approximation of the dielectric matrix (LRDM) based on G\^ateaux derivatives of the residual of fixed point iteration along appropriately chosen direction functions. We develop a computationally efficient method to evaluate these G\^ateaux derivatives in conjunction with the Chebyshev filtered subspace iteration procedure, an approach widely used in large-scale Kohn-Sham DFT calculations. Further, we propose a variant of LRDM preconditioner based on adaptive accumulation of low-rank approximations from previous SCF iterations, and also extend the LRDM preconditioner to spin-polarized Kohn-Sham DFT calculations. We demonstrate the robustness and efficiency of the LRDM preconditioner against other widely used preconditioners on a range of benchmark systems with sizes ranging from $\sim$ 100-1100 atoms ($\sim$ 500--20,000 electrons). The benchmark systems include various combinations of metal-insulating-semiconducting heterogeneous material systems, nanoparticles with localized $d$ orbitals near the Fermi energy, nanofilm with metal dopants, and magnetic systems. In all benchmark systems, the LRDM preconditioner converges robustly within 20--30 iterations. In contrast, other widely used preconditioners show slow convergence in many cases, as well as divergence of the fixed point iteration in some cases. Finally, we demonstrate the computational efficiency afforded by the LRDM method, with up to 3.4$\times$ reduction in computational cost for the total ground-state calculation compared to other preconditioners.

An Efficient and Error-Controllable Angular Sweep Algorithm for Electromagnetic Wave Scattering and its Analysis

The angular sweep of electromagnetic wave scattering is formulated as a matrix equation with multiple right-hand sides (RHSs). Although the low-rank approximation of an RHS matrix is a popular choice for reducing the computational costs of multiple RHSs, only a small amount of research has been conducted to explore how this approximation impacts the solution quality. Furthermore, there has not been sufficient research on the quality of the solution as a function of the accuracy of the iterative solver. We present an error analysis of the approximated solution considering both the reduced number of RHSs and the tolerance of the iterative solver. Based on the error analysis, a new angular sweep algorithm is proposed with fine-tuned tolerances of the iterative solver for individual singular vectors. The different tolerances for each singular vector increase the efficiency of the proposed algorithm. Another benefit of the proposed algorithm is that the error can be bounded by a user-defined global tolerance. In addition, a variant of the generalized conjugate residual method for multiple RHSs is introduced to accelerate iterative solvers. Finally, numerical validation is conducted with three examples in which the discontinuous Galerkin surface integral equation method is applied. The experiments support two conclusions: tight upper and lower bounds of the solution error exist, and fine-tuning the tolerances reduces the computational costs.

Gain and phase calibration of sensor arrays from ambient noise by cross-spectral measurements fitting.

We address the problem of blind gain and phase calibration of a sensor array from ambient noise. The key motivation is to ease the calibration process by avoiding a complex procedure setup. We show that computing the sample covariance matrix in a diffuse field is sufficient to recover the complex gains. To do so, we formulate a non-convex least-square problem based on sample and model covariances. We propose to obtain a solution by low-rank matrix approximation, and two efficient proximal algorithms are derived accordingly. The first algorithm solves the problem modified with a convex relaxation to guarantee that the solution is a global minimizer, and the second algorithm directly solves the initial non-convex problem. We investigate the efficiency of the proposed algorithms by numerical and experimental results according to different sensing configurations. These results show that efficient calibration highly depends on how the measurements are correlated. That is, estimation is achieved more accurately when the field is spatially over-sampled.

Randomized Algorithms for Rounding in the Tensor-Train Format

The tensor-train (TT) format is a highly compact low-rank representation for high-dimensional tensors. TT is particularly useful when representing approximations to the solutions of certain types of parametrized partial differential equations. For many of these problems, computing the solution explicitly would require an infeasible amount of memory and computational time. While the TT format makes these problems tractable, iterative techniques for solving the PDEs must be adapted to perform arithmetic while maintaining the implicit structure. The fundamental operation used to maintain feasible memory and computational time is called rounding, which truncates the internal ranks of a tensor already in TT format. We propose several randomized algorithms for this task that are generalizations of randomized low-rank matrix approximation algorithms and provide significant reduction in computation compared to deterministic TT-rounding algorithms. Randomization is particularly effective in the case of rounding a sum of TT-tensors (where we observe speedup), which is the bottleneck computation in the adaptation of GMRES to vectors in TT format. We present the randomized algorithms and compare their empirical accuracy and computational time with deterministic alternatives.

Matrix Completion via Successive Low-rank Matrix Approximation

In this paper, a successive low-rank matrix approximation algorithm is presented for the matrix completion (MC) based on hard thresholding method, which approximate the optimal low-rank matrix from rank-one matrix step by step. The algorithm enables the distance between the matrix with the observed elements and the projection on low-rank manifold to be minimum. The optimal low-rank matrix with observed elements is obtained when the distance is zero. In theory, convergence and convergent error of the new algorithm are analyzed in detail. Furthermore, some numerical experiments show that the algorithm is more effective in CPU time and precision than the orthogonal rank-one matrix pursuit(OR1MP) algorithm and the augmented Lagrange multiplier (ALM) method when the sampling rate is low.

Denoising of Bistatic Ranges for Elliptic Positioning

This letter considers the denoising of possibly unreliable bistatic range (BR) measurements for elliptic target positioning in multistatic systems. Built upon the BRs over all transmitter–target–receiver paths, the concept of BR matrix is introduced here to handle the problem in an algebraic manner. Specifically, the rank of the ideal BR matrix is proven to be at most 2, which constitutes the foundation of a low-rank matrix approximation (LRMA) formulation presented to enhance the quality of raw datasets. An alternating direction method of multipliers (ADMM) is subsequently devised to perform efficient optimization. Simulations validate the proposed LRMA scheme for denoising.

Improved Low Rank Matrix Approximation Algorithm for Non-synchronous Measurement of Sound Source Localization

Improved Low Rank Matrix Approximation Algorithm for Non-synchronous Measurement of Sound Source Localization