Affine Rank Minimization Research Articles

Stochastic Variance Reduced Gradient for Affine Rank Minimization Problem

Stochastic Variance Reduced Gradient for Affine Rank Minimization Problem

Fast, Efficient, and Viable Compressed Sensing, Low-Rank, and Robust Principle Component Analysis Algorithms for Radar Signal Processing

Modern radar signal processing techniques make strong use of compressed sensing, affine rank minimization, and robust principle component analysis. The corresponding reconstruction algorithms should fulfill the following desired properties: complex valued, viable in the sense of not requiring parameters that are unknown in practice, fast convergence, low computational complexity, and high reconstruction performance. Although a plethora of reconstruction algorithms are available in the literature, these generally do not meet all of the aforementioned desired properties together. In this paper, a set of algorithms fulfilling these conditions is presented. The desired requirements are met by a combination of turbo-message-passing algorithms and smoothed ℓ0-refinements. Their performance is evaluated by use of extensive numerical simulations and compared with popular conventional algorithms.

Positive Semidefinite Matrix Factorization: A Connection With Phase Retrieval and Affine Rank Minimization

Positive semidefinite matrix factorization (PSDMF) expresses each entry of a nonnegative matrix as the inner product of two positive semidefinite (psd) matrices. When all these psd matrices are constrained to be diagonal, this model is equivalent to nonnegative matrix factorization. Applications include combinatorial optimization, quantum-based statistical models, and recommender systems, among others. However, despite the increasing interest in PSDMF, only a few PSDMF algorithms were proposed in the literature. In this work, we provide a collection of tools for PSDMF, by showing that PSDMF algorithms can be designed based on phase retrieval (PR) and affine rank minimization (ARM) algorithms. This procedure allows a shortcut in designing new PSDMF algorithms, as it allows to leverage some of the useful numerical properties of existing PR and ARM methods to the PSDMF framework. Motivated by this idea, we introduce a new family of PSDMF algorithms based on iterative hard thresholding (IHT). This family subsumes previously-proposed projected gradient PSDMF methods. We show that there is high variability among PSDMF optimization problems that makes it beneficial to try a number of methods based on different principles to tackle difficult problems. In certain cases, our proposed methods are the only algorithms able to find a solution. In certain other cases, they converge faster. Our results support our claim that the PSDMF framework can inherit desired numerical properties from PR and ARM algorithms, leading to more efficient PSDMF algorithms, and motivate further study of the links between these models.

Analysis of Singular Value Thresholding Algorithm for Matrix Completion

This paper provides analysis for convergence of the singular value thresholding algorithm for solving matrix completion and affine rank minimization problems arising from compressive sensing, signal processing, machine learning, and related topics. A necessary and sufficient condition for the convergence of the algorithm with respect to the Bregman distance is given in terms of the step size sequence $$\{\delta _k\}_{k\in \mathbb {N}}$$ as $$\sum _{k=1}^{\infty }\delta _k=\infty $$. Concrete convergence rates in terms of Bregman distances and Frobenius norms of matrices are presented. Our novel analysis is carried out by giving an identity for the Bregman distance as the excess gradient descent objective function values and an error decomposition after viewing the algorithm as a mirror descent algorithm with a non-differentiable mirror map.

Learned Turbo Message Passing for Affine Rank Minimization and Compressed Robust Principal Component Analysis

This paper is focused on the efficient algorithm design for affine rank minimization (ARM) and compressed robust principal component analysis (CRPCA). Given the proliferation of the literature on the ARM and CRPCA problems, the existing algorithms mostly take a model-based approach in the algorithm design, and so are sensitive to the generation model of the linear measurement operator or to the generation model of the low-rank matrix to be recovered. It is known that, all these algorithms do not work well, e.g., when the low-rank matrix is ill-conditioned. As inspired by the success of learned iterative soft-thresholding (LIST) and learned approximate message passing (LAMP), we develop learning-based message passing algorithms, namely, the learned turbo message passing (LTMP) algorithm for affine rank minimization to cope with the ARM problem and the LTMP algorithm for the CRPCA problem. The LTMP algorithms learn their parameters from data, and hence are robust to the generation models of the linear operator and the low-rank matrix. We derive analytical expressions for the partial derivatives involved in training the LTMP network. Given the large size of the low-rank matrix in a typical ARM/CRPCA problem, these analytical expressions are of essential importance for the development of computationally feasible network training. Numerical results demonstrate that LTMP significantly outperforms the state-of-the-art counterparts for various generation models of the linear operator and the low-rank matrix, especially when the low-rank matrix is ill-conditioned.

Correntropy Based Matrix Completion.

This paper studies the matrix completion problems when the entries are contaminated by non-Gaussian noise or outliers. The proposed approach employs a nonconvex loss function induced by the maximum correntropy criterion. With the help of this loss function, we develop a rank constrained, as well as a nuclear norm regularized model, which is resistant to non-Gaussian noise and outliers. However, its non-convexity also leads to certain difficulties. To tackle this problem, we use the simple iterative soft and hard thresholding strategies. We show that when extending to the general affine rank minimization problems, under proper conditions, certain recoverability results can be obtained for the proposed algorithms. Numerical experiments indicate the improved performance of our proposed approach.

TARM: A Turbo-Type Algorithm for Affine Rank Minimization

The affine rank minimization (ARM) problem arises in many real-world applications. The goal is to recover a low-rank matrix from a small amount of noisy affine measurements. The original problem is NP-hard, and so directly solving the problem is computationally prohibitive. Approximate low-complexity solutions for ARM have recently attracted much research interest. In this paper, we design an iterative algorithm for ARM based on message passing principles. The proposed algorithm is termed turbo-type ARM (TARM), as inspired by the recently developed turbo compressed sensing algorithm for sparse signal recovery. We show that, when the linear operator for measurement is right-orthogonally invariant (ROIL), a scalar function called state evolution can be established to accurately predict the behaviour of the TARM algorithm. We also show that TARM converges much faster than the counterpart algorithms for low-rank matrix recovery. We further extend the TARM algorithm for matrix completion, where the measurement operator corresponds to a random selection matrix. We show that, although the state evolution is not accurate for matrix completion, the TARM algorithm with carefully tuned parameters still significantly outperforms its counterparts.

$$S_{1/2}$$ S 1 / 2 regularization methods and fixed point algorithms for affine rank minimization problems

The affine rank minimization problem is to minimize the rank of a matrix under linear constraints. It has many applications in various areas such as statistics, control, system identification and machine learning. Unlike the literatures which use the nuclear norm or the general Schatten $$q~ (0

Finding Low-rank Solutions of Sparse Linear Matrix Inequalities using Convex Optimization

This paper is concerned with the problem of finding a low-rank solution of an arbitrary sparse linear matrix inequality (LMI). To this end, we map the sparsity of the LMI problem into a graph. We develop a mathematical framework to relate the rank of the minimum-rank solution of the LMI problem to the sparsity of its underlying graph. Furthermore, we propose three graph-theoretic convex programs to obtain a low-rank solution. Two of these convex optimization problems are based on a tree decomposition of the sparsity graph. The third one does not rely on any computationally expensive graph analysis and is always polynomial-time solvable, at the cost of offering a milder theoretical guarantee on the rank of the obtained solution compared to the other two methods. The results of this work can be readily applied to three separate problems of minimum-rank matrix completion, conic relaxation for polynomial optimization, and affine rank minimization. The results are finally illustrated on two applications of opt...

Approximate Message Passing for Structured Affine Rank Minimization Problem

The topic of the rank minimization problem with affine constraints has been well studied in recent years. However, in many applications the data can exhibit other structures beyond simply being low rank. For example, images and videos present complex spatio-temporal structures, which are largely ignored by current affine rank minimization (ARM) methods. In this paper, we propose a novel approximate message passing (AMP)-based approach that is capable of capturing additional structures in the matrix entries, and can be implemented in a wide range of applications with little or no modification. Using probabilistic low-rank factorization, we derive our generalized AMP-based algorithm as an approximation of the loopy belief propagation algorithm. In addition, we apply a rank selection strategy and an expectation-maximization estimation strategy that adaptively obtain the optimal value of the algorithmic parameters. Then, we discuss the specializations of our proposed algorithm to the applications of structured ARM problems, such as compressive hyperspectral imaging and compressive video surveillance. Simulation results with both synthetic and real data demonstrate that the proposed algorithm yields the state-of-the-art reconstruction performance while maintaining competitive computational complexity.

Transformed Schatten-1 iterative thresholding algorithms for low rank matrix completion

c 2017 International Press C OMMUN. M ATH. S CI. Vol. 15, No. 3, pp. 839–862 TRANSFORMED SCHATTEN-1 ITERATIVE THRESHOLDING ALGORITHMS FOR LOW RANK MATRIX COMPLETION ∗ SHUAI ZHANG † , PENGHANG YIN ‡ , AND JACK XIN § Abstract. We study a non-convex low-rank promoting penalty function, the transformed Schatten- 1 (TS1), and its applications in matrix completion. The TS1 penalty, as a matrix quasi-norm defined on its singular values, interpolates the rank and the nuclear norm through a nonnegative parameter a ∈ (0,+∞). We consider the unconstrained TS1 regularized low-rank matrix recovery problem and develop a fixed point representation for its global minimizer. The TS1 thresholding functions are in closed analytical form for all parameter values. The TS1 threshold values differ in subcritical (supercritical) parameter regime where the TS1 threshold functions are continuous (discontinuous). We propose TS1 iterative thresholding algorithms and compare them with some state-of-the-art algorithms on matrix completion test problems. For problems with known rank, a fully adaptive TS1 iterative thresholding algorithm consistently performs the best under different conditions, where ground truth matrices are generated by multivariate Gaussian, (0,1) uniform and Chi-square distributions. For problems with unknown rank, TS1 algorithms with an additional rank estimation procedure approach the level of IRucL-q which is an iterative reweighted algorithm, non-convex in nature and best in performance. Keywords. transformed Schatten-1 penalty; fixed point representation; closed form thresholding function; iterative thresholding algorithms; matrix completion AMS subject classifications. 90C26; 90C46 1. Introduction Low rank matrix completion problems arise in many applications such as collabora- tive filtering in recommender systems [4,17], minimum order system and low-dimensional Euclidean embedding in control theory [14,15], network localization [18], and others [26]. The mathematical problem is: min rank(X) X∈ m×n s.t. X ∈ L where L is a convex set. In this paper, we are interested in methods for solving the affine rank minimization problem (ARMP) min rank(X) X∈ m×n s.t. A (X) = b in p , where the linear transformation A : m×n → p and vector b ∈ p are given. The matrix completion problem min rank(X) X∈ m×n s.t. X i,j = M i,j , (i,j) ∈ Ω is a special case of (1.2), where X and M are both m × n matrices and Ω is a subset of index pairs {(i,j)}. ∗ Received: September 25, 2016; accepted (in revised form): October 28, 2016. Communicated by Wotao Yin. The work was partially supported by NSF grant DMS-1222507 and DMS-1522383. † Department of Mathematics, University of California, Irvine, CA, 92697, USA (szhang3@uci.edu). Phone: (949)-824-5309. Fax: (949)-824-7993. ‡ Department of Mathematics, University of California, Irvine, CA, 92697, USA (penghany@uci.edu). § Department of Mathematics, University of California, Irvine, CA, 92697, USA (jxin@math.uci.edu).

An Efficient Method for Convex Constrained Rank Minimization Problems Based on DC Programming

The constrained rank minimization problem has various applications in many fields including machine learning, control, and signal processing. In this paper, we consider the convex constrained rank minimization problem. By introducing a new variable and penalizing an equality constraint to objective function, we reformulate the convex objective function with a rank constraint as a difference of convex functions based on the closed-form solutions, which can be reformulated as DC programming. A stepwise linear approximative algorithm is provided for solving the reformulated model. The performance of our method is tested by applying it to affine rank minimization problems and max-cut problems. Numerical results demonstrate that the method is effective and of high recoverability and results on max-cut show that the method is feasible, which provides better lower bounds and lower rank solutions compared with improved approximation algorithm using semidefinite programming, and they are close to the results of the latest researches.

On linear convergence of projected gradient method for a class of affine rank minimization problems

The affine rank minimization problem is to find a low-rank matrix satisfying a set of linear equations, which includes the well-known matrix completion problem as a special case and draws much attention in recent years. In this paper, a new model for affine rank minimization problem is proposed. The new model not only enhances the robustness of affine rank minimization problem, but also leads to high nonconvexity. We show that if the classical projected gradient method is applied to solve our new model, the linear convergence rate can be established under some conditions. Some preliminary experiments have been conducted to show the efficiency and effectiveness of our method.

NuMax: A Convex Approach for Learning Near-Isometric Linear Embeddings

We propose a novel framework for the deterministic construction of linear, near-isometric embeddings of a finite set of data points. Given a set of training points X ⊂ \BBR <sup xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">N</sup> , we consider the secant set S(X) that consists of all pairwise difference vectors of X, normalized to lie on the unit sphere. We formulate an affine rank minimization problem to construct a matrix Ψ that preserves the norms of all the vectors in S(X) up to a distortion parameter δ. While affine rank minimization is NP-hard, we show that this problem can be relaxed to a convex formulation that can be solved using a tractable semidefinite program (SDP). In order to enable scalability of our proposed SDP to very large-scale problems, we adopt a two-stage approach. First, in order to reduce compute time, we develop a novel algorithm based on the Alternating Direction Method of Multipliers (ADMM) that we call Nuclear norm minimization with Max-norm constraints (NuMax) to solve the SDP. Second, we develop a greedy, approximate version of NuMax based on the column generation method commonly used to solve large-scale linear programs. We demonstrate that our framework is useful for a number of signal processing applications via a range of experiments on large-scale synthetic and real datasets.

On finding a generalized lowest rank solution to a linear semi-definite feasibility problem

In this note, we generalize the affine rank minimization problem and the vector cardinality minimization problem and show that the resulting generalized problem can be solved by solving a sequence of continuous concave minimization problems. In the case of the vector cardinality minimization problem, we show that it can be solved exactly by solving the continuous concave minimization problem.

Iterative Concave Rank Approximation for Recovering Low-Rank Matrices

In this paper, we propose a new algorithm for recovery of low-rank matrices from compressed linear measurements. The underlying idea of this algorithm is to closely approximate the rank function with a smooth function of singular values, and then minimize the resulting approximation subject to the linear constraints. The accuracy of the approximation is controlled via a scaling parameter $\delta$, where a smaller $\delta$ corresponds to a more accurate fitting. The consequent optimization problem for any finite $\delta$ is nonconvex. Therefore, in order to decrease the risk of ending up in local minima, a series of optimizations is performed, starting with optimizing a rough approximation (a large $\delta$) and followed by successively optimizing finer approximations of the rank with smaller $\delta$'s. To solve the optimization problem for any $\delta > 0$, it is converted to a new program in which the cost is a function of two auxiliary positive semidefinete variables. The paper shows that this new program is concave and applies a majorize-minimize technique to solve it which, in turn, leads to a few convex optimization iterations. This optimization scheme is also equivalent to a reweighted Nuclear Norm Minimization (NNM), where weighting update depends on the used approximating function. For any $\delta > 0$, we derive a necessary and sufficient condition for the exact recovery which are weaker than those corresponding to NNM. On the numerical side, the proposed algorithm is compared to NNM and a reweighted NNM in solving affine rank minimization and matrix completion problems showing its considerable and consistent superiority in terms of success rate, especially, when the number of measurements decreases toward the lower-bound for the unique representation.

Sparse Representation of a Polytope and Recovery of Sparse Signals and Low-Rank Matrices

This paper considers compressed sensing and affine rank minimization in both noiseless and noisy cases and establishes sharp restricted isometry conditions for sparse signal and low-rank matrix recovery. The analysis relies on a key technical tool, which represents points in a polytope by convex combinations of sparse vectors. The technique is elementary while yielding sharp results. It is shown that for any given constant <formula formulatype="inline" xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"><tex Notation="TeX">$t\geq{4/3}$</tex></formula> , in compressed sensing, <formula formulatype="inline" xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"><tex Notation="TeX">$\delta_{tk}^{A}&lt;\sqrt{(t-1)/t}$</tex></formula> guarantees the exact recovery of all <formula formulatype="inline" xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"><tex Notation="TeX">$k$</tex></formula> sparse signals in the noiseless case through the constrained <formula formulatype="inline" xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"><tex Notation="TeX">$\ell_{1}$</tex></formula> minimization, and similarly, in affine rank minimization, <formula formulatype="inline" xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"><tex Notation="TeX">$\delta_{tr}^{\cal M}&lt;\sqrt{(t-1)/t}$</tex></formula> ensures the exact reconstruction of all matrices with rank at most <formula formulatype="inline" xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"><tex Notation="TeX">$r$</tex></formula> in the noiseless case via the constrained nuclear norm minimization. In addition, for any <formula formulatype="inline" xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"> <tex Notation="TeX">$\epsilon&gt;0$</tex></formula> , <formula formulatype="inline" xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"> <tex Notation="TeX">$\delta_{tk}^{A}&lt;\sqrt{{t-1}\over{t}}+\epsilon$</tex></formula> is not sufficient to guarantee the exact recovery of all <formula formulatype="inline" xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"><tex Notation="TeX">$k$</tex></formula> -sparse signals for large <formula formulatype="inline" xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"><tex Notation="TeX">$k$</tex></formula> . Similar results also hold for matrix recovery. In addition, the conditions <formula formulatype="inline" xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"><tex Notation="TeX">$\delta_{tk}^{A}&lt;\sqrt{(t-1)/t}$</tex></formula> and <formula formulatype="inline" xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"><tex Notation="TeX">$\delta_{tr}^{\cal M}&lt;\sqrt{(t-1)/t}$</tex></formula> are also shown to be sufficient, respectively, for stable recovery of approximately sparse signals and low-rank matrices in the noisy case.

Compressed Sensing and Affine Rank Minimization Under Restricted Isometry

This paper establishes new restricted isometry conditions for compressed sensing and affine rank minimization. It is shown for compressed sensing that $\delta_{k}^A+\theta_{k,k}^A < 1$ guarantees the exact recovery of all $k$ sparse signals in the noiseless case through the constrained $\ell_1$ minimization. Furthermore, the upper bound 1 is sharp in the sense that for any $\epsilon > 0$, the condition $\delta_k^A + \theta_{k, k}^A < 1+\epsilon$ is not sufficient to guarantee such exact recovery using any recovery method. Similarly, for affine rank minimization, if $\delta_{r}^\mathcal{M}+\theta_{r,r}^\mathcal{M}< 1$ then all matrices with rank at most $r$ can be reconstructed exactly in the noiseless case via the constrained nuclear norm minimization; and for any $\epsilon > 0$, $\delta_r^\mathcal{M} +\theta_{r,r}^\mathcal{M} < 1+\epsilon$ does not ensure such exact recovery using any method. Moreover, in the noisy case the conditions $\delta_{k}^A+\theta_{k,k}^A < 1$ and $\delta_{r}^\mathcal{M}+\theta_{r,r}^\mathcal{M}< 1$ are also sufficient for the stable recovery of sparse signals and low-rank matrices respectively. Applications and extensions are also discussed.

An introduction to a class of matrix cone programming

In this paper, we define a class of linear conic programming (which we call matrix cone programming or MCP) involving the epigraphs of five commonly used matrix norms and the well studied symmetric cone. MCP has recently been found to have many important applications, for example, in nuclear norm relaxations of affine rank minimization problems. In order to make the defined MCP tractable and meaningful, we must first understand the structure of these epigraphs. So far, only the epigraph of the Frobenius matrix norm, which can be regarded as a second order cone, has been well studied. Here, we take an initial step to study several important properties, including its closed form solution, calm Bouligand-differentiability and strong semismoothness, of the metric projection operator over the epigraph of the $$l_1,\,l_\infty $$, spectral or operator, and nuclear matrix norm, respectively. These properties make it possible to apply augmented Lagrangian methods, which have recently received a great deal of interests due to their high efficiency in solving large scale semidefinite programming, to this class of MCP problems. The work done in this paper is far from comprehensive. Rather it is intended as a starting point to call for more insightful research on MCP so that it can serve as a basic tool to solve more challenging convex matrix optimization problems in years to come.

Sampling and Reconstructing Signals From a Union of Linear Subspaces

In this paper, we study the problem of sampling and reconstructing signals which are assumed to lie on or close to one of several subspaces of a Hilbert space. Importantly, we here consider a very general setting in which we allow infinitely many subspaces in infinite dimensional Hilbert spaces. This general approach allows us to unify many results derived recently in areas such as compressed sensing, affine rank minimization, analog compressed sensing and structured matrix decompositions.