Underdetermined System Of Linear Equations Research Articles

Restricted Normal Cones and Sparsity Optimization with Affine Constraints

The problem of finding a vector with the fewest nonzero elements that satisfies an underdetermined system of linear equations is an NP-complete problem that is typically solved numerically via convex heuristics or nicely behaved nonconvex relaxations. In this paper we consider the elementary method of alternating projections (MAP) for solving the sparsity optimization problem without employing convex heuristics. In a parallel paper we recently introduced the restricted normal cone which generalizes the classical Mordukhovich normal cone and reconciles some fundamental gaps in the theory of sufficient conditions for local linear convergence of the MAP algorithm. We use the restricted normal cone together with the notion of superregularity, which is inherently satisfied for the affine sparse optimization problem, to obtain local linear convergence results with estimates for the radius of convergence of the MAP algorithm applied to sparsity optimization with an affine constraint.

Large sparse signal recovery by conjugate gradient algorithm based on smoothing technique

Finding sparse solutions to under-determined linear systems of equations have intensively involved in fields of machine learning, signal processing, compressive sensing, linear inverse problems and statistical inference. Generally, the task can be realized by solving ℓ1-norm regularized minimization problems. However, the resulting problem is challenging due to the non-smoothness of the regularization term. Inspired by Nesterov’s smoothing technique, this paper proposes, analyzes and tests a modified Polak–Ribière–Polyak conjugate gradient method to solve large-scale ℓ1-norm least squares problem for sparse signal recovery. The per-iteration cost of the proposed algorithm is dominated by three matrix–vector multiplications and the global convergence is guaranteed by results in optimization literature. Moreover, the algorithm is also accelerated by continuation loops as usual. The limited experiments show that this continuation technique benefits to its performance. Numerical experiments which decode a sparse signal from its limited measurements illustrate that the proposed algorithm performs better than NESTA—a recently developed code with Nesterov’s smoothing technique and gradient algorithm.

Convergence Speed of a Dynamical System for Sparse Recovery

This paper studies the convergence rate of a continuous-time dynamical system for l <sub xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">1</sub> -minimization, known as the Locally Competitive Algorithm (LCA). Solving l <sub xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">1</sub> -minimization problems efficiently and rapidly is of great interest to the signal processing community, as these programs have been shown to recover sparse solutions to underdetermined systems of linear equations and come with strong performance guarantees. The LCA under study differs from the typical l <sub xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">1</sub> -solver in that it operates in continuous time: instead of being specified by discrete iterations, it evolves according to a system of nonlinear ordinary differential equations. The LCA is constructed from simple components, giving it the potential to be implemented as a large-scale analog circuit. The goal of this paper is to give guarantees on the convergence time of the LCA system. To do so, we analyze how the LCA evolves as it is recovering a sparse signal from underdetermined measurements. We show that under appropriate conditions on the measurement matrix and the problem parameters, the path the LCA follows can be described as a sequence of linear differential equations, each with a small number of active variables. This allows us to relate the convergence time of the system to the restricted isometry constant of the matrix. Interesting parallels to sparse-recovery digital solvers emerge from this study. Our analysis covers both the noisy and noiseless settings and is supported by simulation results.

Total variation minimization approach in in-line x-ray phase-contrast tomography

The reconstruction problem in in-line X-ray Phase-Contrast Tomography is usually approached by solving two independent linearized sub-problems: phase retrieval and tomographic reconstruction. Both problems are often ill-posed and require the use of regularization techniques that lead to artifacts in the reconstructed image. We present a novel reconstruction approach that solves two coupled linear problems algebraically. Our approach is based on the assumption that the frequency space of the tomogram can be divided into bands that are accurately recovered and bands that are undefined by the observations. This results in an underdetermined linear system of equations. We investigate how this system can be solved using three different algebraic reconstruction algorithms based on Total Variation minimization. These algorithms are compared using both simulated and experimental data. Our results demonstrate that in many cases the proposed algebraic algorithms yield a significantly improved accuracy over the conventional L2-regularized closed-form solution. This work demonstrates that algebraic algorithms may become an important tool in applications where the acquisition time and the delivered radiation dose must be minimized.

Phase retrieval in in-line x-ray phase contrast imaging based on total variation minimization

State-of-the-art techniques for phase retrieval in propagation based X-ray phase-contrast imaging are aiming to solve an underdetermined linear system of equations. They commonly employ Tikhonov regularization - an L2-norm regularized deconvolution scheme - despite some of its limitations. We present a novel approach to phase retrieval based on Total Variation (TV) minimization. We incorporated TV minimization for deconvolution in phase retrieval using a variety of the most common linear phase-contrast models. The results of our TV minimization was compared with Tikhonov regularized deconvolution on simulated as well as experimental data. The presented method was shown to deliver improved accuracy in reconstructions based on a single distance as well as multiple distance phase-contrast images corrupted by noise and hampered by errors due to nonlinear imaging effects.

Block sparse recovery via mixed l 2/l 1 minimization

We consider efficient methods for the recovery of block sparse signals from underdetermined system of linear equations. We show that if the measurement matrix satisfies the block RIP with δ2s < 0.4931, then every block s-sparse signal can be recovered through the proposed mixed l2/l1-minimization approach in the noiseless case and is stably recovered in the presence of noise and mismodeling error. This improves the result of Eldar and Mishali (in IEEE Trans. Inform. Theory 55: 5302–5316, 2009). We also give another sufficient condition on block RIP for such recovery method: δs < 0.307.

A Multilevel Iterated-Shrinkage Approach to $l_{1}$ Penalized Least-Squares Minimization

The area of sparse approximation of signals is drawing tremendous attention in recent years. Typically, sparse solutions of underdetermined linear systems of equations are required. Such solutions are often achieved by minimizing an <i xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">l</i> <sub xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">1</sub> penalized least squares functional. Various iterative-shrinkage algorithms have recently been developed and are quite effective for handling these problems, often surpassing traditional optimization techniques. In this paper, we suggest a new iterative multilevel approach that reduces the computational cost of existing solvers for these inverse problems. Our method takes advantage of the typically sparse representation of the signal, and at each iteration it adaptively creates and processes a hierarchy of lower-dimensional problems employing well-known iterated shrinkage methods. Analytical observations suggest, and numerical results confirm, that this new approach may significantly enhance the performance of existing iterative shrinkage algorithms in cases where the matrix is given explicitly.

Computing Sparse Representations of Multidimensional Signals Using Kronecker Bases

Recently there has been great interest in sparse representations of signals under the assumption that signals (data sets) can be well approximated by a linear combination of few elements of a known basis (dictionary). Many algorithms have been developed to find such representations for one-dimensional signals (vectors), which requires finding the sparsest solution of an underdetermined linear system of algebraic equations. In this letter, we generalize the theory of sparse representations of vectors to multiway arrays (tensors)--signals with a multidimensional structure--by using the Tucker model. Thus, the problem is reduced to solving a large-scale underdetermined linear system of equations possessing a Kronecker structure, for which we have developed a greedy algorithm, Kronecker-OMP, as a generalization of the classical orthogonal matching pursuit (OMP) algorithm for vectors. We also introduce the concept of multiway block-sparse representation of N-way arrays and develop a new greedy algorithm that exploits not only the Kronecker structure but also block sparsity. This allows us to derive a very fast and memory-efficient algorithm called N-BOMP (N-way block OMP). We theoretically demonstrate that under the block-sparsity assumption, our N-BOMP algorithm not only has a considerably lower complexity but is also more precise than the classic OMP algorithm. Moreover, our algorithms can be used for very large-scale problems, which are intractable using standard approaches. We provide several simulations illustrating our results and comparing our algorithms to classical algorithms such as OMP and BP (basis pursuit) algorithms. We also apply the N-BOMP algorithm as a fast solution for the compressed sensing (CS) problem with large-scale data sets, in particular, for 2D compressive imaging (CI) and 3D hyperspectral CI, and we show examples with real-world multidimensional signals.

Microlocal Analysis of the Geometric Separation Problem

AbstractImage data are often composed of two or more geometrically distinct constituents; in galaxy catalogs, for instance, one sees a mixture of pointlike structures (galaxy superclusters) and curvelike structures (filaments). It would be ideal to process a single image and extract two geometrically “pure” images, each one containing features from only one of the two geometric constituents. This seems to be a seriously underdetermined problem but recent empirical work achieved highly persuasive separations.We present a theoretical analysis showing that accurate geometric separation of point and curve singularities can be achieved by minimizing the ℓ1norm of the representing coefficients in two geometrically complementary frames: wavelets and curvelets. Driving our analysis is a specific property of the ideal (but unachievable) representation where each content type is expanded in the frame best adapted to it. This ideal representation has the property that important coefficients areclustered geometricallyin phase space, and that at fine scales, there is very little coherence between a cluster of elements in one frame expansion and individual elements in the complementary frame. We formally introduce notions ofcluster coherenceandclustered sparsityand use this machinery to show that the underdetermined systems of linear equations can be stably solved by ℓ1minimization; microlocal phase space helps organize the calculations that cluster coherence requires. © 2012 Wiley Periodicals, Inc.

Asymptotic Performance of Sparse Signal Detection Using Convex Programming Method

The detection of sparse signals against background noise is considered. Detecting signals of such kind is difficult since only a small portion of the signal carries information. Prior knowledge is usually assumed to ease detection. In this paper, we consider the general unknown and arbitrary sparse signal detection problem when no prior knowledge is available. Under a Neyman-Pearson hypothesis-testing framework, a new detection scheme is proposed by combining a generalized likelihood ratio test (GLRT)-like test statistic and convex programming methods which directly exploit sparsity in an underdetermined system of linear equations. We characterize large sample behavior of the proposed method by analyzing its asymptotic performance. Specifically, we give the condition for the Chernoff-consistent detection which shows that the proposed method is very sensitive to the ℓ2 norm energy of the sparse signals. Both the false alarm rate and the miss rate tend to zero at vanishing signal-to-noise ratio (SNR), as long as the signal energy grows at least logarithmically with the problem dimension. Next we give a large deviation analysis to characterize the error exponent for the Neyman-Pearson detection. We derive the oracle error exponent assuming signal knowledge. Then we explicitly derive the error exponent of the proposed scheme and compare it with the oracle exponent. We complement our study with numerical experiments, showing that the proposed method performs in the vicinity of the likelihood ratio test (LRT) method in the finite sample scenario and the error probability degrades exponentially with the number of observations.

Sparse Solution of Underdetermined Systems of Linear Equations by Stagewise Orthogonal Matching Pursuit

Finding the sparsest solution to underdetermined systems of linear equations y = Φx is NP-hard in general. We show here that for systems with “typical”/“random” Φ, a good approximation to the sparsest solution is obtained by applying a fixed number of standard operations from linear algebra. Our proposal, Stagewise Orthogonal Matching Pursuit (StOMP), successively transforms the signal into a negligible residual. Starting with initial residual r0 = y, at the s -th stage it forms the “matched filter” ΦTrs-1, identifies all coordinates with amplitudes exceeding a specially chosen threshold, solves a least-squares problem using the selected coordinates, and subtracts the least-squares fit, producing a new residual. After a fixed number of stages (e.g., 10), it stops. In contrast to Orthogonal Matching Pursuit (OMP), many coefficients can enter the model at each stage in StOMP while only one enters per stage in OMP; and StOMP takes a fixed number of stages (e.g., 10), while OMP can take many (e.g., n). We give both theoretical and empirical support for the large-system effectiveness of StOMP. We give numerical examples showing that StOMP rapidly and reliably finds sparse solutions in compressed sensing, decoding of error-correcting codes, and overcomplete representation.

Improving resolution of miniature spectrometers by exploiting sparse nature of signals

In this paper, we present a signal processing approach to improve the resolution of a spectrometer with a fixed number of low-cost, non-ideal filters. We aim to show that the resolution can be improved beyond the limit set by the number of filters by exploiting the sparse nature of a signal spectrum. We consider an underdetermined system of linear equations as a model for signal spectrum estimation. We design a non-negative L1 norm minimization algorithm for solving the system of equations. We demonstrate that the resolution can be improved multiple times by using the proposed algorithm.

Reweighted $\ell_1$-Minimization for Sparse Solutions to Underdetermined Linear Systems

Numerical experiments have indicated that the reweighted $\ell_1$-minimization performs exceptionally well in locating sparse solutions of underdetermined linear systems of equations. We show that reweighted $\ell_1$-methods are intrinsically associated with the minimization of the so-called merit functions for sparsity, which are essentially concave approximations to the cardinality function. Based on this observation, we further show that a family of reweighted $\ell_1$-algorithms can be systematically derived from the perspective of concave optimization through the linearization technique. In order to conduct a unified convergence analysis for this family of algorithms, we introduce the concept of the range space property (RSP) of a matrix and prove that if its adjoint has this property, the reweighted $\ell_1$-algorithm can find a sparse solution to the underdetermined linear system provided that the merit function for sparsity is properly chosen. In particular, some convergence conditions for the Can...

Linear Program Relaxation of Sparse Nonnegative Recovery in Compressive Sensing Microarrays

Compressive sensing microarrays (CSM) are DNA-based sensors that operate using group testing and compressive sensing principles. Mathematically, one can cast the CSM as sparse nonnegative recovery (SNR) which is to find the sparsest solutions subjected to an underdetermined system of linear equations and nonnegative restriction. In this paper, we discuss the l 1 relaxation of the SNR. By defining nonnegative restricted isometry/orthogonality constants, we give a nonnegative restricted property condition which guarantees that the SNR and the l 1 relaxation share the common unique solution. Besides, we show that any solution to the SNR must be one of the extreme points of the underlying feasible set.

LDPC Codes for Compressed Sensing

We present a mathematical connection between channel coding and compressed sensing. In particular, we link, on the one hand, channel coding linear programming decoding (CC-LPD), which is a well-known relaxation of maximum-likelihood channel decoding for binary linear codes, and, on the other hand, compressed sensing linear programming decoding (CS-LPD), also known as basis pursuit, which is a widely used linear programming relaxation for the problem of finding the sparsest solution of an underdetermined system of linear equations. More specifically, we establish a tight connection between CS-LPD based on a zero-one measurement matrix over the reals and CC-LPD of the binary linear channel code that is obtained by viewing this measurement matrix as a binary parity-check matrix. This connection allows the translation of performance guarantees from one setup to the other. The main message of this paper is that parity-check matrices of “good” channel codes can be used as provably “good” measurement matrices under basis pursuit. In particular, we provide the first deterministic construction of compressed sensing measurement matrices with an order-optimal number of rows using high-girth low-density parity-check codes constructed by Gallager.

The Noise-Sensitivity Phase Transition in Compressed Sensing

Consider the noisy underdetermined system of linear equations: y = Ax0 + z, with A an n × N measurement matrix, n <; N, and z ~ N(0, σ2I) a Gaussian white noise. Both y and A are known, both x0 and z are unknown, and we seek an approximation to x0. When x0 has few nonzeros, useful approximations are often obtained by l1-penalized l2 minimization, in which the reconstruction x1,λ solves min{||y - Ax||22/2 + λ||x||1}. Consider the reconstruction mean-squared error MSE = E|| x1,λ - x0||22/N, and define the ratio MSE/σ2 as the noise sensitivity. Consider matrices A with i.i.d. Gaussian entries and a large-system limit in which n, N → ∞ with n/N → δ and k/n → ρ. We develop exact expressions for the asymptotic MSE of x1,λ , and evaluate its worst-case noise sensitivity over all types of k-sparse signals. The phase space 0 ≤ 8, ρ ≤ 1 is partitioned by the curve ρ = ρMSE(δ) into two regions. Formal noise sensitivity is bounded throughout the region ρ = ρMSE(δ) and is unbounded throughout the region ρ = ρMSE(δ). The phase boundary ρ = ρMSE(δ) is identical to the previously known phase transition curve for equivalence of l1 - l0 minimization in the k-sparse noiseless case. Hence, a single phase boundary describes the fundamental phase transitions both for the noise less and noisy cases. Extensive computational experiments validate these predictions, including the existence of game-theoretical structures underlying it (saddlepoints in the payoff, least-favorable signals and maximin penalization). Underlying our formalism is an approximate message passing soft thresholding algorithm (AMP) introduced earlier by the authors. Other papers by the authors detail expressions for the formal MSE of AMP and its close connection to l1-penalized reconstruction. The focus of the present paper is on computing the minimax formal MSE within the class of sparse signals x0.

Real versus complex null space properties for sparse vector recovery

We identify and solve an overlooked problem about the characterization of underdetermined systems of linear equations for which sparse solutions have minimal ℓ 1 -norm. This characterization is known as the null space property. When the system has real coefficients, sparse solutions can be considered either as real or complex vectors, leading to two seemingly distinct null space properties. We prove that the two properties actually coincide by establishing a link with a problem about convex polygons in the real plane. Incidentally, we also show the equivalence between stable null space properties which account for the stable reconstruction by ℓ 1 -minimization of vectors that are not exactly sparse.

Phase transitions for greedy sparse approximation algorithms

A major enterprise in compressed sensing and sparse approximation is the design and analysis of computationally tractable algorithms for recovering sparse, exact or approximate, solutions of underdetermined linear systems of equations. Many such algorithms have now been proven to have optimal-order uniform recovery guarantees using the ubiquitous Restricted Isometry Property (RIP) (Candès and Tao (2005) [11]). However, without specifying a matrix, or class of matrices, it is unclear when the RIP-based sufficient conditions on the algorithm are satisfied. Bounds on RIP constants can be inserted into the algorithms RIP-based conditions, translating the conditions into requirements on the signal's sparsity level, length, and number of measurements. We illustrate this approach for Gaussian matrices on three of the state-of-the-art greedy algorithms: CoSaMP (Needell and Tropp (2009) [29]), Subspace Pursuit (SP) (Dai and Milenkovic (2009) [13]) and Iterative Hard Thresholding (IHT) (Blumensath and Davies (2009) [8]). Designed to allow a direct comparison of existing theory, our framework implies that, according to the best available analysis on these three algorithms, IHT requires the fewest number of compressed sensing measurements, has the best proven stability bounds, and has the lowest per iteration computational cost.

Minimizing Nonconvex Functions for Sparse Vector Reconstruction

In this paper, we develop a novel methodology for minimizing a class of nonconvex (concave on the non-negative orthant) functions for solving an underdetermined linear system of equations As = x when the solution vector s is known a priori to be sparse. The proposed technique is based on locally replacing the original objective function by a quadratic convex function which is easily minimized. The resulting algorithm is iterative and is absolutely converging to a fixed point of the original objective function. For a certain selection of convex objective functions, the class of algorithms called iterative reweighted least squares (IRLS) is shown to be a special case of the proposed methodology. Thus, the proposed algorithms are a generalization and unification of the previous methods. In addition, we also propose a new class of algorithms with better convergence properties compared to the regular IRLS algorithms and, hence, can be considered as enhancements to these algorithms. Since the original objective functions are nonconvex, the proposed algorithm is susceptible to convergence to a local minimum. To alleviate this difficulty, we propose a random perturbation technique that enhances the performance of the proposed algorithm. The numerical results show that the proposed algorithms outperform some of the well-known algorithms that are usually utilized for solving the same problem.

$\ell_{2}/\ell_{1}$-Optimization in Block-Sparse Compressed Sensing and Its Strong Thresholds

It has been known for a while that l1-norm relaxation can in certain cases solve an under-determined system of linear equations. Recently, E. Candes (Robust uncertainty principles: Exact signal reconstruction from highly incomplete frequency information, IEEE Trans. Information Theory, vol. 52, no. 12, pp. 489-509, Dec. 2006) and D. Donoho (High-dimensional centrally symmetric polytopes with neighborlines proportional to dimension, Disc. Comput. Geometry, vol. 35, no. 4, pp. 617-652, 2006) proved (in a large dimensional and statistical context) that if the number of equations (measurements in the compressed sensing terminology) in the system is proportional to the length of the unknown vector then there is a sparsity (number of nonzero elements of the unknown vector) also proportional to the length of the unknown vector such that l1-norm relaxation succeeds in solving the system. In this paper, in a large dimensional and statistical context, we determine sharp lower bounds on the values of allowable sparsity for any given number (proportional to the length of the unknown vector) of equations for the case of the so-called block-sparse unknown vectors considered in On the reconstruction of block-sparse signals with an optimal number of measurements, (M. Stojnic et al., IEEE Trans, Signal Processing, submitted for publication.