Low-rank Approximation Problem Research Articles

An Improved Frequent Directions Algorithm for Low-Rank Approximation via Block Krylov Iteration.

Frequent directions (FDs), as a deterministic matrix sketching technique, have been proposed for tackling low-rank approximation problems. This method has a high degree of accuracy and practicality but experiences a lot of computational cost for large-scale data. Several recent works on the randomized version of FDs greatly improve the computational efficiency but unfortunately sacrifice some precision. To remedy such an issue, this article aims to find a more accurate projection subspace to further improve the efficiency and effectiveness of the existing FDs' techniques. Specifically, by utilizing the power of the block Krylov iteration and random projection technique, this article presents a fast and accurate FDs algorithm named r-BKIFD. The rigorous theoretical analysis shows that the proposed r-BKIFD has a comparable error bound with original FDs, and the approximation error can be arbitrarily small when the number of iterations is chosen appropriately. Extensive experimental results on both synthetic and real data further demonstrate the superiority of r-BKIFD over several popular FDs algorithms both in terms of computational efficiency and accuracy.

On the optimal rank-1 approximation of matrices in the Chebyshev norm

The problem of low rank approximation is ubiquitous in science. Traditionally this problem is solved in unitary invariant norms such as Frobenius or spectral norm due to existence of efficient methods for building approximations. However, recent results reveal the potential of low rank approximations in Chebyshev norm, which naturally arises in many applications. In this paper we tackle the problem of building optimal rank-1 approximations in the Chebyshev norm. We investigate the properties of alternating minimization algorithm for building the low rank approximations and demonstrate how to use it to construct optimal rank-1 approximation. As a result we propose an algorithm that is capable of building optimal rank-1 approximations in Chebyshev norm for moderate matrices.

Pass-efficient randomized LU algorithms for computing low-rank matrix approximation

Abstract Low-rank matrix approximation is extremely useful in the analysis of data that arises in scientific computing, engineering applications, and data science. However, as data sizes grow, traditional low-rank matrix approximation methods, such as singular value decomposition (SVD) and column pivoting QR decomposition (CPQR), are either prohibitively expensive or cannot provide sufficiently accurate results. A solution is to use randomized low-rank matrix approximation methods such as randomized SVD, and randomized LU decomposition on extremely large data sets. In this paper, we focus on the randomized LU decomposition method. Then we propose a novel randomized LU algorithm, called SubspaceLU, for the fixed low-rank approximation problem. SubspaceLU is based on the sketch of the co-range of input matrices and allows for an arbitrary number of passes of the input matrix, v ≥ 2 {v\geq 2} . Numerical experiments on CPU show that our proposed SubspaceLU is generally faster than the existing randomized LU decomposition, while remaining accurate. Experiments on GPU shows that our proposed SubspaceLU can gain more speedup than the existing randomized LU decomposition. We also propose a version of SubspaceLU, called SubspaceLU_FP, for the fixed precision low-rank matrix approximation problem. SubspaceLU_FP is a post-processing step based on an efficient blocked adaptive rank determination Algorithm 5 proposed in this paper. We present numerical experiments that show that SubspaceLU_FP can achieve close results to SVD but faster in speed. We finally propose a single-pass algorithm based on LU factorization. Tests show that the accuracy of our single-pass algorithm is comparable with the existing single-pass algorithms.

Fast truncated SVD of sparse and dense matrices on graphics processors

We investigate the solution of low-rank matrix approximation problems using the truncated singular value decomposition (SVD). For this purpose, we develop and optimize graphics processing unit (GPU) implementations for the randomized SVD and a blocked variant of the Lanczos approach. Our work takes advantage of the fact that the two methods are composed of very similar linear algebra building blocks, which can be assembled using numerical kernels from existing high-performance linear algebra libraries. Furthermore, the experiments with several sparse matrices arising in representative real-world applications and synthetic dense test matrices reveal a performance advantage of the block Lanczos algorithm when targeting the same approximation accuracy.

Robust Matrix Completion Method Based on TNNR and Total Row Difference for Recovering Optical Image

Matrix completion aims to recover a matrix from an incomplete matrix with many unknown elements and has wide applications in optical image recovery and machine learning, in which the popular method is to formulate it as a general low-rank matrix approximation problem. However, the traditional optimization model for matrix completion is less robust. This paper proposes a robust matrix completion method in which the truncated nuclear norm regularization (TNNR) is used as the approximation of the rank function and the sum of absolute values of the row difference, which is called as the total row difference, is used to constrain the oscillations of the missing matrix. By minimizing the value of total row difference in the objective, the proposed model controls the oscillation and reduces the impact of missing part in the process of matrix completion continuously. Furthermore, we propose a two-step iterative algorithm framework and design an ADMM algorithm for subproblem model that includes minimizing the total row difference. Experiments show that the proposed algorithm has more stable performance and better recovery effect and obviously reduces the sensitivity of the traditional TNNR models to the truncated rank parameter.

Sketching for a low-rank nonnegative matrix approximation: Numerical study

Abstract We propose new approximate alternating projection methods, based on randomized sketching, for the low-rank nonnegative matrix approximation problem: find a low-rank approximation of a nonnegative matrix that is nonnegative, but whose factors can be arbitrary. We calculate the computational complexities of the proposed methods and evaluate their performance in numerical experiments. The comparison with the known deterministic alternating projection methods shows that the randomized approaches are faster and exhibit similar convergence properties.

Low-rank signal subspace: parameterization, projection and signal estimation

The paper contains several theoretical results related to the weighted nonlinear least-squares problem for low-rank signal estimation, which can be considered as a Hankel structured low-rank approximation problem. A parameterization of the subspace of low-rank time series connected with generalized linear recurrence relations (GLRRs) is described and its features are investigated. It is shown how the obtained results help to describe the tangent plane, prove optimization problem features and construct stable algorithms for solving low-rank approximation problems. For the latter, a stable algorithm for constructing the projection onto a subspace of time series that satisfy a given GLRR is proposed and justified. This algorithm is used for a new implementation of the known Gauss-Newton method using the variable projection approach. The comparison by stability and computational cost is performed theoretically and with the help of an example.

A Data-Driven State Estimation Framework for Natural Gas Networks With Measurement Noise

The large-scale coverage of natural gas makes the composition structure and operation mode of natural gas network more complex, higher requirements are put forward for the effectiveness and accuracy of state estimation. The existing methods for state estimation of natural gas network with noise are all modeled after processing the data with noise, leading to the real data being distorted to a certain extent. With that in mind, a data-driven method is presented in this paper. While solving the problem of state estimation for natural gas network with measurement noise in the input data, filtering and denoising are unnecessary during state estimation, retaining the complete information of real data. It avoids destruction of real data induced by separating noise from measured data owing to different methods and intensities of noise processing. According to the gas flow characteristic equation of natural gas system, the original problem is converted into a weighted low-rank approximation problem, the search space is shrunk to an orthogonal complement space. The selection of initial values is not merely unrestricted but there will be no accumulation and transmission of iteration error. The effectiveness of the proposed method is demonstrated through simulating 10-node natural gas network. Compared with the Newton’s method, the data-driven method has superior performance, the RMSE achieves 0.2268 and the MAPE achieves 1.63%.

Randomized Quaternion Singular Value Decomposition for Low-Rank Matrix Approximation

This paper presents a randomized quaternion singular value decomposition (QSVD) algorithm for low-rank matrix approximation problems, which are widely used in color face recognition, video compression, and signal processing problems. With quaternion normal distribution-based random sampling, the randomized QSVD algorithm projects high-dimensional data to a low-dimensional subspace and then identifies an approximate range subspace of the quaternion matrix. The key statistical properties of quaternion Wishart distribution are proposed and used to perform the approximation error analysis of the algorithm. Theoretical results show that the randomized QSVD algorithm can trace dominant singular value decomposition triplets of a quaternion matrix with acceptable accuracy. Numerical experiments also indicate the rationality of proposed theories. Applied to color face recognition problems, the randomized QSVD algorithm obtains higher recognition accuracies and behaves more efficient than the known Lanczos-based partial QSVD and a quaternion version of the fast frequent directions algorithm.

Guarantees for Existence of a Best Canonical Polyadic Approximation of a Noisy Low-Rank Tensor

The canonical polyadic decomposition (CPD) of a low-rank tensor plays a major role in data analysis and signal processing by allowing for unique recovery of underlying factors. However, it is well known that the low-rank CPD approximation problem is ill-posed. That is, a tensor may fail to have a best rank $R$ CPD approximation when $R>1$. This article gives deterministic bounds for the existence of best low-rank tensor approximations over ${\mathbb{K}}={\mathbb{R}}$ or ${\mathbb{K}}={\mathbb{C}}$. More precisely, given a tensor ${\mathcal T} \in {\mathbb{K}}^{I \times I \times I}$ of rank $R \leq I$, we compute the radius of a Frobenius norm ball centered at ${\mathcal T}$ in which best ${\mathbb{K}}$-rank $R$ approximations are guaranteed to exist. In addition we show that every ${\mathbb{K}}$-rank $R$ tensor inside of this ball has a unique canonical polyadic decomposition. This neighborhood may be interpreted as a neighborhood of “mathematical truth" in which CPD approximation and computation are well-posed. In pursuit of these bounds, we describe low-rank tensor decomposition as a “joint generalized eigenvalue" problem. Using this framework, we show that, under mild assumptions, a low-rank tensor which has rank strictly greater than border rank is defective in the sense of algebraic and geometric multiplicities for joint generalized eigenvalues. Bounds for existence of best low-rank approximations are then obtained by establishing perturbation theoretic results for the joint generalized eigenvalue problem. In this way we establish a connection between existence of best low-rank approximations and the tensor spectral norm. In addition we solve a “tensor Procrustes problem" which examines orthogonal compressions for pairs of tensors. The main results of the article are illustrated by a variety of numerical experiments.

Low Phase-Rank Approximation

In this paper, we propose and solve low phase-rank approximation problems, which serve as a counterpart to the well-known low-rank approximation problem and the Schmidt-Mirsky theorem. It is well known that a nonzero complex number can be specified by its gain and phase, and while it is generally accepted that the gains of a matrix may be defined by its singular values, there is no widely accepted definition for its phases. In this work, we consider sectorial matrices, whose numerical ranges do not contain the origin, and adopt the canonical angles of such matrices as their phases. Similarly to the rank of a matrix being defined as the number of its nonzero singular values, we define the phase-rank of a sectorial matrix as the number of its nonzero phases. While a low-rank approximation problem is associated with the matrix arithmetic mean, it turns out that a natural parallel for the low phase-rank approximation problem is to use the matrix geometric mean to measure the approximation error. Importantly, we derive a majorization inequality between the phases of the geometric mean and the arithmetic mean of the phases, similarly to the Ky-Fan inequality for eigenvalues of Hermitian matrices. A characterization of the solutions to the proposed problem, with the same flavor as the Schmidt-Mirsky theorem, is then obtained in the case where both the objective matrix and the approximant are restricted to be positive-imaginary. In addition, we provide an alternative formulation of the low phase-rank approximation problem using geodesic distances between sectorial matrices. The two formulations give rise to the exact same set of solutions when the involved matrices are additionally assumed to be unitary.

A nonlocal enhanced Low-Rank tensor approximation framework for 3D Magnetic Resonance image denoising

Magnetic Resonance Imaging (MRI) has become an increasingly essential tool in clinical detection and diagnosis for its ability to disclose the distinctive information of human anatomic structures in vivo. However, the image intensity in magnitude MRI images is frequently corrupted by Rician noise which is inherent to the acquisition process. This study aimed to propose a denoising model for 3D MR image restoration based on low-rank tensor approximation with a Logarithmic-Sum regularization framework. Nonlocal similarity of images was exploited with grouping and block matching techniques, and tensor decomposition approaches were utilized taking consideration of multi-dimensional structural dependence conservation. The low-rank tensor approximation problem was regulated by introducing a Logarithmic-Sum trace norm, which thresholded the singular value spectrum adaptively with different thresholds. Noise removal was processed through the low-rank approximation, and the denoised tensor blocks were aggregated to restore the clear images. Numerical experiments were conducted under comprehensive noise conditions for 3D MR volumetric datasets. Through denoising comparison, the results demonstrated that the proposed algorithm achieves state-of-the-art performance for Rician noise removal with excellent detail preservation..

Bias correction for Vandermonde low-rank approximation

The low-rank approximation problem, that is the problem of approximating a given matrix with a matrix of lower rank, appears in many scientific fields. In some applications the given matrix is structured and the approximation is required to have the same structure. Examples of linear structures are Hankel, Toeplitz, and Sylvester. Currently, there are only a few results for nonlinearly structured low-rank approximation problems. The problem of Vandermonde structured low-rank approximation is considered. The high condition number of the Vandermonde matrix, in combination with the noise in the data, makes the problem challenging. A numerical method based on a bias correction procedure is proposed and its properties are demonstrated by simulation. The performance of the method is illustrated on numerical results.

Approximate GCD by relaxed NewtonSLRA algorithm

We propose a better algorithm for approximate greatest common divisor (approximate GCD) of univariate polynomials in terms of robustness and distance, based on the NewtonSLRA algorithm that is a solver for the structured low rank approximation (SLRA) problem. Our algorithm mainly enlarges the tangent space in the NewtonSLRA algorithm and adapts it to a certain weighted Frobenius norm. Moreover, we propose some improvement in computing time.

Effective implementation to reduce execution time of a low-rank matrix approximation problem

This paper proposes a new method to compute generalized low-rank matrix approximation (GLRMA). The GLRMA is a general case of the well-known low-rank approximation problem proposed by Eckart–Young in 1936. This new method, so-called the fast-GLRMA method, is based on tensor product and Tikhonov’s regularization to approximate the pseudoinverse and bilateral random projections to estimate, in turn, the low-rank approximation. The fast-GLRMA method significantly reduces the execution time to compute the optimal solution, while preserving the accuracy of the classical method of solving the GLRMA. Computational experiments to measure execution time and speedup confirmed the efficiency of the proposed method.

PLANC

We consider the problem of low-rank approximation of massive dense nonnegative tensor data, for example, to discover latent patterns in video and imaging applications. As the size of data sets grows, single workstations are hitting bottlenecks in both computation time and available memory. We propose a distributed-memory parallel computing solution to handle massive data sets, loading the input data across the memories of multiple nodes, and performing efficient and scalable parallel algorithms to compute the low-rank approximation. We present a software package called Parallel Low-rank Approximation with Nonnegativity Constraints, which implements our solution and allows for extension in terms of data (dense or sparse, matrices or tensors of any order), algorithm (e.g., from multiplicative updating techniques to alternating direction method of multipliers), and architecture (we exploit GPUs to accelerate the computation in this work). We describe our parallel distributions and algorithms, which are careful to avoid unnecessary communication and computation, show how to extend the software to include new algorithms and/or constraints, and report efficiency and scalability results for both synthetic and real-world data sets.

A novel MR image denoising via LRMA and NLSS

Nonlocal self-similarity has been proven to be a useful tool for image denoising. For MR image denoising, the method combining the nonlocal self-similarity with the low-rank approximation has been recently attracting considerable attentions, due to its favorable performance. Since the original low-rank approximation problem is difficult to be solved, the frequently used method is to use the nuclear norm minimization for the matrix low-rank approximation. However, the solution obtained by nuclear norm minimization generally deviates from the solution of the original problem. In this paper, an approach for MR image denoising is proposed by combining a novel nonlocal self-similarity scheme with a novel low-rank approximation scheme. In proposed approach, a similarity evaluation with respect to the noise is proposed in the patch matching stage. To approximate the original low-rank minimization problem, the propose approach minimizes trace-based operator at each step. Every minimization is solvable and used to approximate the original low-rank minimization. An algorithm is established for this approximation, as well. Experimental results show that the proposed approach has a superior performance, comparing with some of the low-rank approximation methods, in both the objective quality metrics and visual inspections.

Seismic denoising via truncated nuclear norm minimization

Because there are many similar geologic structures underground, seismic profiles have an abundance of self-repeating patterns. Thus, we can divide a seismic profile into groups of blocks with a similar seismic structure. The matrix formed by stacking together similar blocks in each group should be of low rank. Hence, we can transfer the seismic denoising problem to a series of low-rank matrix approximation (LRMA) problems. The LRMA-based model commonly adopts the nuclear norm as a convex substitute of the rank of a matrix. However, the nuclear norm minimization (NNM) shrinks the different rank components equally and may cause some biases in practice. The recently introduced truncated nuclear norm (TNN) has been proven to more accurately approximate the rank of a matrix, which is given by the sum of the set of the smallest singular values. Based on this, we have adopted a novel denoising method using truncated nuclear norm minimization (TNNM). The objective function of this method consists of two terms, the F-norm data fidelity and a TNN regularization. We develop an efficient two-step iterative algorithm to solve this objective function. Then, we apply the proposed TNNM algorithm to groups of blocks with similar seismic structure, and we aggregate all resulting denoised blocks to get the denoised seismic data. We update the denoised results during each iteration to gradually attenuate the heavy noise. Numerical experiments demonstrate that, compared with f-x deconvolution and the curvelet- and NNM-based methods, TNNM not only attenuates noise more effectively even when the signal-to-noise ratio is as low as −10 dB and seismic data have complex structures, but it also accurately preserves the seismic structures without inducing Gibbs artifacts.

Sampling-based dimension reduction for subspace approximation with outliers

The subspace approximation problem with outliers, for given n points in d dimensions x1,x2,…,xn∈Rd, an integer 1≤k≤d, and an outlier parameter 0≤α≤1, is to find a k-dimensional linear subspace of Rd that minimizes the sum of squared distances to its nearest (1−α)n points. More generally, the ℓp subspace approximation problem with outliers minimizes the sum of p-th powers of distances instead of the sum of squared distances. Even the case of p=2 or robust PCA is non-trivial, and previous work requires additional assumptions on the input or generative models for it. Any multiplicative approximation algorithm for the subspace approximation problem with outliers must solve the robust subspace recovery problem, a special case in which the (1−α)n inliers in the optimal solution are promised to lie exactly on a k-dimensional linear subspace. However, robust subspace recovery is Small Set Expansion (SSE)-hard, and known algorithmic results for robust subspace recovery require strong assumptions on the input, e.g., any d outliers must be linearly independent.In this paper, we show how to extend dimension reduction techniques and bi-criteria approximations based on sampling and coresets to the problem of subspace approximation with outliers. To get around the SSE-hardness of robust subspace recovery, we assume that the squared distance error of the optimal k-dimensional subspace summed over the optimal (1−α)n inliers is at least δ times its squared-error summed over all n points, for some 0<δ≤1−α. Under this assumption, we give an efficient algorithm to find a weak coreset or a subset of poly(k/ϵ)log⁡(1/δ)log⁡log⁡(1/δ) points whose span contains a k-dimensional subspace that gives a multiplicative (1+ϵ)-approximation to the optimal solution. Our technique is based on the squared-length sampling algorithm suggested for low-rank approximation problems in the seminal work of Frieze, Kannan, and Vempala [12]. The running time of our algorithm is linear in n and d. Interestingly, our results hold even when the fraction of outliers α is large, as long as the obvious condition 0<δ≤1−α is satisfied. We show similar results for subspace approximation with ℓp error or more general M-estimator loss functions, and also give an additive approximation for the affine subspace approximation problem.

Best Low-rank Approximations and Kolmogorov $n$-widths

We relate the problem of best low-rank approximation in the spectral norm for a matrix $A$ to Kolmogorov $n$-widths and corresponding optimal spaces. We characterize all the optimal spaces for the image of the Euclidean unit ball under $A$ and we show that any orthonormal basis in an $n$-dimensional optimal space generates a best rank-$n$ approximation to $A$. We also present a simple and explicit construction to obtain a sequence of optimal $n$-dimensional spaces once an initial optimal space is known. This results in a variety of solutions to the best low-rank approximation problem and provides alternatives to the truncated singular value decomposition. This variety can be exploited to obtain best low-rank approximations with problem-oriented properties.