Recovery Of Low-rank Matrices Research Articles

Dynamic Matrix Recovery

Matrix recovery from sparse observations is an extensively studied topic emerging in various applications, such as recommendation system and signal processing, which includes the matrix completion and compressed sensing models as special cases. In this article, we propose a general framework for dynamic matrix recovery of low-rank matrices that evolve smoothly over time. We start from the setting that the observations are independent across time, then extend to the setting that both the design matrix and noise possess certain temporal correlation via modified concentration inequalities. By pooling neighboring observations, we obtain sharp estimation error bounds of both settings, showing the influence of the underlying smoothness, the dependence and effective samples. We propose a dynamic fast iterative shrinkage-thresholding algorithm that is computationally efficient, and characterize the interplay between algorithmic and statistical convergence. Simulated and real data examples are provided to support such findings. Supplementary materials for this article are available online.

Robust Recovery of Low-Rank Matrices and Low-Tubal-Rank Tensors from Noisy Sketches

Robust Recovery of Low-Rank Matrices and Low-Tubal-Rank Tensors from Noisy Sketches

The effect of perturbation and noise folding on the recovery performance of low-rank matrix via the nuclear norm minimization

• The completely perturbed low rank matrix recovery model takes in consideration the perturbations in the measurement map and the measurement matrix. • The perturbation on the measurement map result in a multiplicative noise. • The completely perturbed model is equivalent to the partially perturbed model. • The impact of the multiplicative noise on the Signal-to-Noise Ratio (SNR) is different than the additive noise. Previous works in low-rank matrix recovery literature focused on the study of the partially perturbed low-rank matrix recovery model y = A ( X ) + ω , where y ∈ R M is the observed vector, A : R m × n → R M is the measurement operator, X ∈ R m × n is the matrix to be recovered, and ω ∈ R M is the additive noise. However in practice A may only represent a perturbed version of the measurement operator since it models a physical phenomena and there is no guarantee that the desired conditions such as the restricted isometry property (RIP) hold. Similar to the measurement matrix X ∈ R m × n , in a variety of application scenarios, it may also be corrupted by noise. In this paper we introduce the completely perturbed low-rank matrix model by incorporating to the partially perturbed model a non zero perturbation E : R m × n → R M into the measurement operator A which results in a multiplicative noise, and a noise Z ∈ R m × n into the measurement matrix X . First by extending the results of Zhou et al. (2016) we explore the RIP constants of our new proposed model. Then based on the RIP we establish a sufficient condition for robust and stable recovery of low-rank matrices and give an upper bound estimation of the recovery error in the completely perturbed context. The analysis has shown that our model after whitening is equivalent to the partially perturbed model the only distinction is noise variance increase which has a sever impact on the Signal-to-Noise Ratio (SNR).

Iterative hard thresholding for low CP-rank tensor models

Recovery of low-rank matrices from a small number of linear measurements is now well-known to be possible under various model assumptions on the measurements. Such results demonstrate robustness and are backed with provable theoretical guarantees. However, extensions to tensor recovery have only recently began to be studied and developed, despite an abundance of practical tensor applications. Recently, a tensor variant of the Iterative Hard Thresholding method was proposed and theoretical results were obtained that exact guarantee recovery of tensors with low Tucker rank. In this paper, we utilize and prove a similar tensor version of the Restricted Isometry Property (RIP) to extend these results for tensors with low CANDECOMP/PARAFAC (CP) rank. In doing so, we leverage recent results on efficient approximations of CP decompositions that remove the need for challenging assumptions in prior works. We complement our theoretical findings with empirical results that showcase the potential of the approach.

Rank $2r$ Iterative Least Squares: Efficient Recovery of Ill-Conditioned Low Rank Matrices from Few Entries

We present a new, simple, and computationally efficient iterative method for low rank matrix completion. Our method is inspired by the class of factorization-type iterative algorithms, but substantially differs from them in the way the problem is cast. Precisely, given a target rank $r$, instead of optimizing on the manifold of rank $r$ matrices, we allow our interim estimated matrix to have a specific overparametrized rank $2r$ structure. Our algorithm, denoted R2RILS, for rank $2r$ iterative least squares, thus has low memory requirements, and at each iteration it solves a computationally cheap sparse least squares problem. We motivate our algorithm by its theoretical analysis for the simplified case of a rank 1 matrix. Empirically, R2RILS is able to recover ill-conditioned low rank matrices from very few observations---near the information limit---and it is stable to additive noise.

Low-Rank Matrix Estimation from Rank-One Projections by Unlifted Convex Optimization

We study an estimator with a convex formulation for recovery of low-rank matrices from rank-one projections. Using initial estimates of the factors of the target $d_1\times d_2$ matrix of rank-$r$,...

Robust recovery of low-rank matrices with non-orthogonal sparse decomposition from incomplete measurements

We consider the problem of recovering an unknown effectively (s1, s2)-sparse low-rank-R matrix X with possibly non-orthogonal rank-1 decomposition from incomplete and inaccurate linear measurements of the form y=A(X)+η, where η is an ineliminable noise11With “ineliminable” we mean that the focus of the paper is not on recovery from exact y=A(X) measurements, rather on the stable recovery under severe measurement noise.. We first derive an optimization formulation for matrix recovery under the considered model and propose a novel algorithm, called Alternating Tikhonov regularization and Lasso (A-T-LAS2,1), to solve it. The algorithm is based on a multi-penalty regularization, which is able to leverage both structures (low-rankness and sparsity) simultaneously. The algorithm is a fast first order method, and straightforward to implement. We prove global convergence for any linear measurement model to stationary points and local convergence to global minimizers. By adapting the concept of restricted isometry property from compressed sensing to our novel model class, we prove error bounds between global minimizers and ground truth, up to noise level, from a number of subgaussian measurements scaling as R(s1+s2), up to log-factors in the dimension, and relative-to-diameter distortion. Simulation results demonstrate both the accuracy and efficacy of the algorithm, as well as its superiority to the state-of-the-art algorithms in strong noise regimes and for matrices whose singular vectors do not possess exact (joint-) sparse support.

Block-sparse recovery of semidefinite systems and generalized null space conditions

This article considers the recovery of low-rank matrices via a convex nuclear-norm minimization problem and presents two null space properties (NSP) which characterize uniform recovery for the case of block-diagonal matrices and block-diagonal positive semidefinite matrices. These null-space conditions turn out to be special cases of a new general setup, which allows to derive the mentioned NSPs and well-known NSPs from the literature. We discuss the relative strength of these conditions and also present a deterministic class of matrices that satisfies the block-diagonal semidefinite NSP.

Recovery of simultaneous low rank and two-way sparse coefficient matrices, a nonconvex approach

We study the problem of recovery of matrices that are simultaneously low rank and row and/or column sparse. Such matrices appear in recent applications in cognitive neuroscience, imaging, computer vision, macroeconomics, and genetics. We propose a GDT (Gradient Descent with hard Thresholding) algorithm to efficiently recover matrices with such structure, by minimizing a bi-convex function over a nonconvex set of constraints. We show linear convergence of the iterates obtained by GDT to a region within statistical error of an optimal solution. As an application of our method, we consider multi-task learning problems and show that the statistical error rate obtained by GDT is near optimal compared to minimax rate. Experiments demonstrate competitive performance and much faster running speed compared to existing methods, on both simulations and real data sets.

Iterative hard thresholding for low-rank recovery from rank-one projections

A novel algorithm for the recovery of low-rank matrices acquired via compressive linear measurements is proposed and analyzed. The algorithm, a variation on the iterative hard thresholding algorithm for low-rank recovery, is designed to succeed in situations where the standard rank-restricted isometry property fails, e.g. in case of subexponential unstructured measurements or of subgaussian rank-one measurements. The stability and robustness of the algorithm are established based on distinctive matrix-analytic ingredients and its performance is substantiated numerically.

Recovering low-rank matrices from binary measurements

This article studies the approximate recovery of low-rank matrices acquired through binary measurements. Two types of recovery algorithms are considered, one based on hard singular value thresholding and the other one based on semidefinite programming. In case no thresholds are introduced before binary quantization, it is first shown that the direction of the low-rank matrices can be well approximated. Then, in case nonadaptive thresholds are incorporated, it is shown that both the direction and the magnitude can be well approximated. Finally, by allowing the thresholds to be chosen adaptively, we exhibit a recovery procedure for which low-rank matrices are fully approximated with error decaying exponentially with the number of binary measurements. In all cases, the procedures are robust to prequantization error. The underlying arguments are essentially deterministic: they rely only on an unusual restricted isometry property of the measurement process, which is established once and for all for Gaussian measurement processes.

Near-Optimal Compressed Sensing of a Class of Sparse Low-Rank Matrices Via Sparse Power Factorization

Compressed sensing of simultaneously sparse and low-rank matrices enables recovery of sparse signals from a few linear measurements of their bilinear form. One important question is how many measurements are needed for a stable reconstruction in the presence of measurement noise. Unlike conventional compressed sensing for sparse vectors, where convex relaxation via the $\ell _{1}$ -norm achieves near-optimal performance, for compressed sensing of sparse low-rank matrices, it has been shown recently that convex programmings using the nuclear norm and the mixed norm are highly suboptimal even in the noise-free scenario. We propose an alternating minimization algorithm called sparse power factorization (SPF) for compressed sensing of sparse rank-one matrices. For a class of signals whose sparse representation coefficients are fast-decaying, SPF achieves stable recovery of the rank-one matrix formed by their outer product and requires number of measurements within a logarithmic factor of the information-theoretic fundamental limit. For the recovery of general sparse low-rank matrices, we propose subspace-concatenated SPF (SCSPF), which has analogous near-optimal performance guarantees to SPF in the rank-one case. Numerical results show that SPF and SCSPF empirically outperform convex programmings using the best known combinations of mixed norm and nuclear norm.

Optimal Shrinkage of Singular Values

We consider the recovery of low-rank matrices from noisy data by shrinkage of singular values, in which a single, univariate nonlinearity is applied to each of the empirical singular values. We adopt an asymptotic framework, in which the matrix size is much larger than the rank of the signal matrix to be recovered, and the signal-to-noise ratio of the low-rank piece stays constant. For a variety of loss functions, including Mean Square Error (MSE) - (square Frobenius norm), the nuclear norm loss and the operator norm loss, we show that in this framework there is a well-defined asymptotic loss that we evaluate precisely in each case. In fact, each of the loss functions we study admits a unique admissible shrinkage nonlinearity dominating all other nonlinearities. We provide a general method for evaluating these optimal nonlinearities, and demonstrate our framework by working out simple, explicit formulas for the optimal nonlinearities in the Frobenius, nuclear and operator norm cases. For example, for a square low-rank n-by-n matrix observed in white noise with level σ, the optimal nonlinearity for MSE loss simply shrinks each data singular value y to √(y <sup xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">2</sup> -4nσ <sup xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">2</sup> ) (or to 0 if y <; 2√nσ). This optimal value y to nonlinearity guarantees an asymptotic MSE of 2nrσ <sup xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">2</sup> , which compares favorably with optimally tuned hard thresholding and optimally tuned soft thresholding, providing guarantees of 3nrσ <sup xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">2</sup> and 6nrσ <sup xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">2</sup> , respectively. Our general method also allows one to evaluate optimal shrinkers numerically to arbitrary precision. As an example, we compute optimal shrinkers for the Schatten-p norm loss, for any p > 0.

Recovery of low-rank matrices based on the rank null space properties

This paper first discusses the relationship between the rank null space property (NSP) and the nuclear norm minimization. Several versions of the rank NSP, i.e. the stable rank NSP, robust rank NSP and Frobenius robust rank NSP are proposed, and their equivalent forms are derived. At the same time, it is shown that the stable rank NSP is weaker than the rank restricted isometry property (RIP) to recover the low-rank matrices via the nuclear norm minimization. Finally, the rank NSP is extended to the case of Schatten-[Formula: see text] NSP for [Formula: see text], and the solutions to the Schatten-[Formula: see text] quasi-norm minimization are characterized by the different types of Schatten-[Formula: see text] NSP.

Denoising by low-rank and sparse representations

Due to the ill-posed nature of image denoising problem, good image priors are of great importance for an effective restoration. Nonlocal self-similarity and sparsity are two popular and widely used image priors which have led to several state-of-the-art methods in natural image denoising. In this paper, we take advantage of these priors and propose a new denoising algorithm based on sparse and low-rank representation of image patches under a nonlocal framework. This framework consists of two complementary steps. In the first step, noise removal from groups of matched image patches is formulated as recovery of low-rank matrices from noisy data. This problem is then efficiently solved under asymptotic matrix reconstruction model based on recent results from random matrix theory which leads to a parameter-free optimal estimator. Nonlocal learned sparse representation is adopted in the second step to suppress artifacts introduced in the previous estimate. Experimental results, demonstrate the superior denoising performance of the proposed algorithm as compared with the state-of-the-art methods.

Performance guarantees for Schatten-p quasi-norm minimization in recovery of low-rank matrices

We address some theoretical guarantees for Schatten-p quasi-norm minimization (p∈(0,1]) in recovering low-rank matrices from compressed linear measurements. Firstly, using null space properties of the measurement operator, we provide a sufficient condition for exact recovery of low-rank matrices. This condition guarantees unique recovery of matrices of ranks equal or larger than what is guaranteed by nuclear norm minimization. Secondly, this sufficient condition leads to a theorem proving that all restricted isometry property (RIP) based sufficient conditions for ℓp quasi-norm minimization generalize to Schatten-p quasi-norm minimization. Based on this theorem, we provide a few RIP-based recovery conditions.

ROP: Matrix recovery via rank-one projections

Estimation of low-rank matrices is of significant interest in a range of contemporary applications. In this paper, we introduce a rank-one projection model for low-rank matrix recovery and propose a constrained nuclear norm minimization method for stable recovery of low-rank matrices in the noisy case. The procedure is adaptive to the rank and robust against small perturbations. Both upper and lower bounds for the estimation accuracy under the Frobenius norm loss are obtained. The proposed estimator is shown to be rate-optimal under certain conditions. The estimator is easy to implement via convex programming and performs well numerically. The techniques and main results developed in the paper also have implications to other related statistical problems. An application to estimation of spiked covariance matrices from one-dimensional random projections is considered. The results demonstrate that it is still possible to accurately estimate the covariance matrix of a high-dimensional distribution based only on one-dimensional projections.

Stable recovery of low rank matrices from nuclear norm minimization

Low rank matrix recovery is a new topic drawing the attention of many researchers which addresses the problem of recovering an unknown low rank matrix from few linear measurements. The matrix Dantzig selector and the matrix Lasso are two important algorithms based on nuclear norm minimization. In this paper, we first prove some decay properties of restricted isometry constants, then we discuss the recovery errors of these two algorithms and give a new bound of restricted isometry constant to guarantee stable recovery, which improves the results of [11].

Iterative Concave Rank Approximation for Recovering Low-Rank Matrices

In this paper, we propose a new algorithm for recovery of low-rank matrices from compressed linear measurements. The underlying idea of this algorithm is to closely approximate the rank function with a smooth function of singular values, and then minimize the resulting approximation subject to the linear constraints. The accuracy of the approximation is controlled via a scaling parameter $\delta$, where a smaller $\delta$ corresponds to a more accurate fitting. The consequent optimization problem for any finite $\delta$ is nonconvex. Therefore, in order to decrease the risk of ending up in local minima, a series of optimizations is performed, starting with optimizing a rough approximation (a large $\delta$) and followed by successively optimizing finer approximations of the rank with smaller $\delta$'s. To solve the optimization problem for any $\delta > 0$, it is converted to a new program in which the cost is a function of two auxiliary positive semidefinete variables. The paper shows that this new program is concave and applies a majorize-minimize technique to solve it which, in turn, leads to a few convex optimization iterations. This optimization scheme is also equivalent to a reweighted Nuclear Norm Minimization (NNM), where weighting update depends on the used approximating function. For any $\delta > 0$, we derive a necessary and sufficient condition for the exact recovery which are weaker than those corresponding to NNM. On the numerical side, the proposed algorithm is compared to NNM and a reweighted NNM in solving affine rank minimization and matrix completion problems showing its considerable and consistent superiority in terms of success rate, especially, when the number of measurements decreases toward the lower-bound for the unique representation.

The Optimal Hard Threshold for Singular Values is &lt;inline-formula&gt; &lt;tex-math notation="TeX"&gt;\(4/\sqrt {3}\) &lt;/tex-math&gt;&lt;/inline-formula&gt;

We consider recovery of low-rank matrices from noisy data by hard thresholding of singular values, in which empirical singular values below a threshold λ are set to 0. We study the asymptotic mean squared error (AMSE) in a framework, where the matrix size is large compared with the rank of the matrix to be recovered, and the signal-to-noise ratio of the low-rank piece stays constant. The AMSE-optimal choice of hard threshold, in the case of n-by-n matrix in white noise of level σ, is simply (4/√3)√nσ ≈ 2.309√nσ when σ is known, or simply 2.858 · y <sub xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">med</sub> when σ is unknown, where y <sub xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">med</sub> is the median empirical singular value. For nonsquare, m by n matrices with m ≠ n the thresholding coefficients 4/√3 and 2.858 are replaced with different provided constants that depend on m/n. Asymptotically, this thresholding rule adapts to unknown rank and unknown noise level in an optimal manner: it is always better than hard thresholding at any other value, and is always better than ideal truncated singular value decomposition (TSVD), which truncates at the true rank of the low-rank matrix we are trying to recover. Hard thresholding at the recommended value to recover an n-by-n matrix of rank r guarantees an AMSE at most 3 nrσ <sup xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">2</sup> . In comparison, the guarantees provided by TSVD, optimally tuned singular value soft thresholding and the best guarantee achievable by any shrinkage of the data singular values are 5 nrσ <sup xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">2</sup> , 6 nrσ <sup xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">2</sup> , and 2 nrσ <sup xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">2</sup> , respectively. The recommended value for hard threshold also offers, among hard thresholds, the best possible AMSE guarantees for recovering matrices with bounded nuclear norm. Empirical evidence suggests that performance improvement over TSVD and other popular shrinkage rules can be substantial, for different noise distributions, even in relatively small n.