Low-rank Matrix Approximation Problem Research Articles

Pass-efficient randomized LU algorithms for computing low-rank matrix approximation

Abstract Low-rank matrix approximation is extremely useful in the analysis of data that arises in scientific computing, engineering applications, and data science. However, as data sizes grow, traditional low-rank matrix approximation methods, such as singular value decomposition (SVD) and column pivoting QR decomposition (CPQR), are either prohibitively expensive or cannot provide sufficiently accurate results. A solution is to use randomized low-rank matrix approximation methods such as randomized SVD, and randomized LU decomposition on extremely large data sets. In this paper, we focus on the randomized LU decomposition method. Then we propose a novel randomized LU algorithm, called SubspaceLU, for the fixed low-rank approximation problem. SubspaceLU is based on the sketch of the co-range of input matrices and allows for an arbitrary number of passes of the input matrix, v ≥ 2 {v\geq 2} . Numerical experiments on CPU show that our proposed SubspaceLU is generally faster than the existing randomized LU decomposition, while remaining accurate. Experiments on GPU shows that our proposed SubspaceLU can gain more speedup than the existing randomized LU decomposition. We also propose a version of SubspaceLU, called SubspaceLU_FP, for the fixed precision low-rank matrix approximation problem. SubspaceLU_FP is a post-processing step based on an efficient blocked adaptive rank determination Algorithm 5 proposed in this paper. We present numerical experiments that show that SubspaceLU_FP can achieve close results to SVD but faster in speed. We finally propose a single-pass algorithm based on LU factorization. Tests show that the accuracy of our single-pass algorithm is comparable with the existing single-pass algorithms.

Fast truncated SVD of sparse and dense matrices on graphics processors

We investigate the solution of low-rank matrix approximation problems using the truncated singular value decomposition (SVD). For this purpose, we develop and optimize graphics processing unit (GPU) implementations for the randomized SVD and a blocked variant of the Lanczos approach. Our work takes advantage of the fact that the two methods are composed of very similar linear algebra building blocks, which can be assembled using numerical kernels from existing high-performance linear algebra libraries. Furthermore, the experiments with several sparse matrices arising in representative real-world applications and synthetic dense test matrices reveal a performance advantage of the block Lanczos algorithm when targeting the same approximation accuracy.

Robust Matrix Completion Method Based on TNNR and Total Row Difference for Recovering Optical Image

Matrix completion aims to recover a matrix from an incomplete matrix with many unknown elements and has wide applications in optical image recovery and machine learning, in which the popular method is to formulate it as a general low-rank matrix approximation problem. However, the traditional optimization model for matrix completion is less robust. This paper proposes a robust matrix completion method in which the truncated nuclear norm regularization (TNNR) is used as the approximation of the rank function and the sum of absolute values of the row difference, which is called as the total row difference, is used to constrain the oscillations of the missing matrix. By minimizing the value of total row difference in the objective, the proposed model controls the oscillation and reduces the impact of missing part in the process of matrix completion continuously. Furthermore, we propose a two-step iterative algorithm framework and design an ADMM algorithm for subproblem model that includes minimizing the total row difference. Experiments show that the proposed algorithm has more stable performance and better recovery effect and obviously reduces the sensitivity of the traditional TNNR models to the truncated rank parameter.

Sketching for a low-rank nonnegative matrix approximation: Numerical study

Abstract We propose new approximate alternating projection methods, based on randomized sketching, for the low-rank nonnegative matrix approximation problem: find a low-rank approximation of a nonnegative matrix that is nonnegative, but whose factors can be arbitrary. We calculate the computational complexities of the proposed methods and evaluate their performance in numerical experiments. The comparison with the known deterministic alternating projection methods shows that the randomized approaches are faster and exhibit similar convergence properties.

Randomized Quaternion Singular Value Decomposition for Low-Rank Matrix Approximation

This paper presents a randomized quaternion singular value decomposition (QSVD) algorithm for low-rank matrix approximation problems, which are widely used in color face recognition, video compression, and signal processing problems. With quaternion normal distribution-based random sampling, the randomized QSVD algorithm projects high-dimensional data to a low-dimensional subspace and then identifies an approximate range subspace of the quaternion matrix. The key statistical properties of quaternion Wishart distribution are proposed and used to perform the approximation error analysis of the algorithm. Theoretical results show that the randomized QSVD algorithm can trace dominant singular value decomposition triplets of a quaternion matrix with acceptable accuracy. Numerical experiments also indicate the rationality of proposed theories. Applied to color face recognition problems, the randomized QSVD algorithm obtains higher recognition accuracies and behaves more efficient than the known Lanczos-based partial QSVD and a quaternion version of the fast frequent directions algorithm.

Effective implementation to reduce execution time of a low-rank matrix approximation problem

This paper proposes a new method to compute generalized low-rank matrix approximation (GLRMA). The GLRMA is a general case of the well-known low-rank approximation problem proposed by Eckart–Young in 1936. This new method, so-called the fast-GLRMA method, is based on tensor product and Tikhonov’s regularization to approximate the pseudoinverse and bilateral random projections to estimate, in turn, the low-rank approximation. The fast-GLRMA method significantly reduces the execution time to compute the optimal solution, while preserving the accuracy of the classical method of solving the GLRMA. Computational experiments to measure execution time and speedup confirmed the efficiency of the proposed method.

Seismic denoising via truncated nuclear norm minimization

Because there are many similar geologic structures underground, seismic profiles have an abundance of self-repeating patterns. Thus, we can divide a seismic profile into groups of blocks with a similar seismic structure. The matrix formed by stacking together similar blocks in each group should be of low rank. Hence, we can transfer the seismic denoising problem to a series of low-rank matrix approximation (LRMA) problems. The LRMA-based model commonly adopts the nuclear norm as a convex substitute of the rank of a matrix. However, the nuclear norm minimization (NNM) shrinks the different rank components equally and may cause some biases in practice. The recently introduced truncated nuclear norm (TNN) has been proven to more accurately approximate the rank of a matrix, which is given by the sum of the set of the smallest singular values. Based on this, we have adopted a novel denoising method using truncated nuclear norm minimization (TNNM). The objective function of this method consists of two terms, the F-norm data fidelity and a TNN regularization. We develop an efficient two-step iterative algorithm to solve this objective function. Then, we apply the proposed TNNM algorithm to groups of blocks with similar seismic structure, and we aggregate all resulting denoised blocks to get the denoised seismic data. We update the denoised results during each iteration to gradually attenuate the heavy noise. Numerical experiments demonstrate that, compared with f-x deconvolution and the curvelet- and NNM-based methods, TNNM not only attenuates noise more effectively even when the signal-to-noise ratio is as low as −10 dB and seismic data have complex structures, but it also accurately preserves the seismic structures without inducing Gibbs artifacts.

Approximation Schemes for Low-rank Binary Matrix Approximation Problems

We provide a randomized linear time approximation scheme for a generic problem about clustering of binary vectors subject to additional constraints. The new constrained clustering problem generalizes a number of problems and by solving it, we obtain the first linear time-approximation schemes for a number of well-studied fundamental problems concerning clustering of binary vectors and low-rank approximation of binary matrices. Among the problems solvable by our approach are L ow GF(2)-R ank A pproximation , L ow B oolean -R ank A pproximation , and various versions of B inary C lustering . For example, for L ow GF(2)-R ank A pproximation problem, where for an m × n binary matrix A and integer r > 0, we seek for a binary matrix B of GF(2) rank at most r such that the ℓ 0 -norm of matrix A−B is minimum, our algorithm, for any ϵ > 0 in time f ( r ,ϵ)⋅ n ⋅ m , where f is some computable function, outputs a (1+ϵ)-approximate solution with probability at least (1−1\ e ). This is the first linear time approximation scheme for these problems. We also give (deterministic) PTASes for these problems running in time n f ( r )1\ϵ 2 log 1\ϵ , where f is some function depending on the problem. Our algorithm for the constrained clustering problem is based on a novel sampling lemma, which is interesting on its own.

Low-rank matrix approximation in the infinity norm

The low-rank matrix approximation problem with respect to the entry-wise ℓ∞-norm is the following: given a matrix M and a factorization rank r, find a matrix X whose rank is at most r and that minimizes maxi,j⁡|Mij−Xij|. In this paper, we prove that the decision variant of this problem for r=1 is NP-complete using a reduction from the problem ‘not all equal 3SAT’. We also analyze several cases when the problem can be solved in polynomial time, and propose a simple practical heuristic algorithm which we apply on the problem of the recovery of a quantized low-rank matrix, that is, to the problem of recovering a low-rank matrix whose entries have been rounded up to some accuracy.

Single-image de-raining using low-rank matrix approximation

Most existing image de-raining methods are based on the assumption that rain has certain shape or chromatic priors, which may be inaccurate for some rain cases and may lead to unsatisfactory de-rained results. In this paper, the proposed image de-raining method is built on the priors of clear images that many self-similar patches exist in natural images. And similar patches satisfy the low-rank property when grouped together. This property has led to many powerful image/video denoising schemes with impressive performance but has never been used for single-image de-raining. We formulate the problem of removing rain as a low-rank matrix approximation problem by adopting this self-similarity image prior with a designed patch matching strategy applicable for rain images. The resulting nuclear norm minimization problem can be efficiently solved by many recently developed methods. Experiments on both real rain images and synthetic rain images show that the proposed method is competitive against other state-of-the-art methods.

A Sequential Block-Structured Dictionary Learning Algorithm for Block Sparse Representations

Dictionary learning algorithms have been successfully applied to a number of signal and image processing problems. In some applications, however, the observed signals may further have a multisubspace structure that enables block-sparse signal representations. For this case, a new algorithm for learning block-structured dictionaries for block-sparse signal representations is proposed in this paper. It is obtained by solving a sequence of penalized low rank matrix approximation problems where the $\ell _{1,2}$ -norm is introduced as a penalty promoting block sparsity and then using a block coordinate descent approach to estimate the unknowns. The proposed algorithm has the advantage of involving simple closed form solutions for both, the sparse coding and dictionary update stages. In particular, the sparse coding stage involves a simple shrinkage operation related to soft thresholding and it is related to the uniformly most powerful invariant testing procedure. Experimental results showing the improved efficacy and significant gain in computational time, offered by the proposed algorithm over the usual K singular value decomposition block extension are presented.

Doubling Phase Shifters for Efficient Hybrid Precoder Design in Millimeter-Wave Communication Systems

Hybrid precoding is a cost-effective approach to support directional transmissions for millimeter-wave (mmWave) communications, but its precoder design is highly complicated. In this paper, we propose a new hybrid precoder implementation, namely the double phase shifter (DPS) implementation, which enables highly tractable hybrid precoder design. Efficient algorithms are then developed for two popular hybrid precoder structures, i.e., the fully- and partially-connected structures. For the fully-connected one, the RF-only pre-coding and hybrid precoding problems are formulated as a least absolute shrinkage and selection operator problem and a low-rank matrix approximation problem, respectively. In this way, computationally efficient algorithms are provided to approach the performance of the fully digital one with a small number of radio frequency (RF) chains. On the other hand, the hybrid precoder design in the partially-connected structure is identified as an eigenvalue problem. To enhance the performance of this cost-effective structure, dynamic mapping from RF chains to antennas is further proposed, for which a greedy algorithm and a modified K-means algorithm are developed. Simulation results demonstrate the performance gains of the proposed hybrid precoding algorithms over existing ones. It shows that, with the proposed DPS implementation, the fully-connected structure enjoys both satisfactory performance and low design complexity while the partially-connected one serves as an economic solution with low hardware complexity.

A Hybrid Truncated Norm Regularization Method for Matrix Completion.

Matrix completion has been widely used in image processing, in which the popular approach is to formulate this issue as a general low-rank matrix approximation problem. This paper proposes a novel regularization method referred to as truncated Frobenius norm (TFN), and presents a hybrid truncated norm (HTN) model combining the truncated nuclear norm and truncated Frobenius norm for solving matrix completion problems. To address this model, a simple and effective two-step iteration algorithm is designed. Further, an adaptive way to change the penalty parameter is introduced to reduce the computational cost. Also, the convergence of the proposed method is discussed and proved mathematically. The proposed approach could not only effectively improve the recovery performance but also greatly promote the stability of the model. Meanwhile, the use of this new method could eliminate large variations that exist when estimating complex models, and achieve competitive successes in matrix completion. Experimental results on the synthetic data, real-world images as well as recommendation systems, particularly the use of the statistical analysis strategy, verify the effectiveness and superiority of the proposed method, i.e. the proposed method is more stable and effective than other state-of-the-art approaches.

On the Complexity of Robust PCA and ℓ1-Norm Low-Rank Matrix Approximation

The low-rank matrix approximation problem with respect to the component-wise ℓ1-norm (ℓ1-LRA), which is closely related to robust principal component analysis (PCA), has become a very popular tool in data mining and machine learning. Robust PCA aims to recover a low-rank matrix that was perturbed with sparse noise, with applications for example in foreground-background video separation. Although ℓ1-LRA is strongly believed to be NP-hard, there is, to our knowledge, no formal proof of this fact. In this paper, we prove that ℓ1-LRA is NP-hard, already in the rank-one case, using a reduction from MAX CUT. Our derivations draw interesting connections between ℓ1-LRA and several other well-known problems, i.e., robust PCA, ℓ0-LRA, binary matrix factorization, a particular densest bipartite subgraph problem, the computation of the cut norm of {−1, + 1} matrices, and the discrete basis problem, all of which we prove to be NP-hard.

Motion Capture Data Completion via Truncated Nuclear Norm Regularization

The objective of motion capture (mocap) data completion is to recover missing measurement of the body markers from mocap. It becomes increasingly challenging as the missing ratio and duration of mocap data grow. Traditional approaches usually recast this problem as a low-rank matrix approximation problem based on the nuclear norm. However, the nuclear norm defined as the sum of all the singular values of a matrix is not a good approximation to the rank of mocap data. This paper proposes a novel approach to solve mocap data completion problem by adopting a new matrix norm, called truncated nuclear norm. An efficient iterative algorithm is designed to solve this problem based on the augmented Lagrange multiplier. The convergence of the proposed method is proved mathematically under mild conditions. To demonstrate the effectiveness of the proposed method, various comparative experiments are performed on synthetic data and mocap data. Compared to other methods, the proposed method is more efficient and accurate.

Truncated Nuclear Norm Minimization Based Group Sparse Representation for Image Restoration

Group sparse representation has shown great potential in image restoration, which can be considered as a low-rank matrix approximation problem. The nuclear norm minimization method, as a convex rel...

Sum-of-exponentials modeling and common dynamics estimation using Tensorlab

Fitting a signal to a sum-of-exponentials model is a basic problem in signal processing. It can be posed and solved as a Hankel structured low-rank matrix approximation problem. Subsequently, local optimization, subspace, and convex relaxation methods can be used for the numerical solution. In this paper, we show another approach, based on the recently developed concept of structured data fusion. Structured data fusion problems are solved in the Tensorlab toolbox by local optimization methods. The approach allows fitting of signals with missing samples and adding constraints on the model, such as fixed exponents and common dynamics in multi-channel estimation problems. These problems are non-trivial to solve by other existing methods. Tensorlab is publicly available and the results presented are reproducible.

Semidefinite Representations of Gauge Functions for Structured Low-Rank Matrix Decomposition

This paper presents generalizations of semidefinite programming formulations of 1-norm optimization problems over infinite dictionaries of vectors of complex exponentials, which were recently proposed for superresolution, gridless compressed sensing, and other applications in signal processing. Results related to the generalized Kalman--Yakubovich--Popov lemma in linear system theory provide simple, constructive proofs of the semidefinite representations of the penalty functions used in these applications. The connection leads to several extensions to gauge functions and atomic norms for sets of vectors parameterized via the nullspace of matrix pencils. The techniques are illustrated with examples of low-rank matrix approximation problems arising in spectral estimation and array processing.

Salient object detection with low-rank approximation and ℓ2,1-norm minimization

Salient object detection is an important issue in computer vision and image procession in that it can facilitate humans to locate conspicuous visual regions in complex scenes rapidly and improve the performance of object detection and video tracking. In recent years, low-rank matrix approximation has been proved to be favorable in image saliency detection and gained a great deal of attention. An underlying assumption of low-rank recovery is that an image is a combination of background regions being low-rank and salient objects being sparse, which corresponds to tough non-smooth optimization problems. In this paper, by incorporating ℓ2,1-norm minimization, we obtain the corresponding smooth optimization problems and propose two effective algorithms with proved convergence. To guarantee the robustness of the proposed methods, the input image is divided into patches and each patch is approximately represented by its mean value. Besides, multi-scale visual features of each patch of the given image are extracted to capture common low-level features such as color, edge, shape and texture. The salient objects of a given image are indicated with sparse coefficients solved by the low-rank matrix approximation problem. Saliency maps are further produced with integration of the high-level prior knowledge. Finally, extensive experiments in four real-world datasets demonstrate that the proposed methods come with competitive performance over the eight compared state-of-the-arts.

Rank constrained nuclear norm minimization with application to image denoising

In the low rank matrix approximation problem, the well known nuclear norm minimization (NNM) problem plays a crucial role and attracts significant interests in recent years. In NNM the regularization parameter λ plays a decisive part, λ controls both the rank of the solution and the extent of the thresholding. However, it is hard for a single λ to balance the two issuses, and meanwhile the solving method calls singular value decomposition (SVD), of which the computational complexity is impracticable when the scale of the problem becomes large. This paper presents a rank constrained nuclear norm minimization (RNNM) method, in which the rank and the extent of thresholding are controlled separately by an added parameter k. More importantly, by proving its equivalence with an unconstrained bi-convex optimization problem RNNM can be solved in SVD free manner. In this paper, a SOR (Successive Over Relaxation) algorithm is designed for the equivalent bi-convex problem and its convergence is proved. We show that RNNM has a unique global optimal solution although being non-convex. We explicitly analyse the structure of the solution for the bi-convex problem and show some interesting properties. Finally, we verify the effectiveness of RNNM in image denoising. Experimental results show that the proposed solving method works faster than SVD based method. Thanks to the well balance of rank and thresholding, RNNM achieves superior results than the state-of-the-art methods in image denoising such as BM3D, SAIST in terms of both quantity measure and visual quality.