Underdetermined Linear Measurements Research Articles

A theory of optimal convex regularization for low-dimensional recovery

Abstract We consider the problem of recovering elements of a low-dimensional model from under-determined linear measurements. To perform recovery, we consider the minimization of a convex regularizer subject to a data fit constraint. Given a model, we ask ourselves what is the ‘best’ convex regularizer to perform its recovery. To answer this question, we define an optimal regularizer as a function that maximizes a compliance measure with respect to the model. We introduce and study several notions of compliance. We give analytical expressions for compliance measures based on the best-known recovery guarantees with the restricted isometry property. These expressions permit to show the optimality of the $\ell ^{1}$-norm for sparse recovery and of the nuclear norm for low-rank matrix recovery for these compliance measures. We also investigate the construction of an optimal convex regularizer using the examples of sparsity in levels and of sparse plus low-rank models.

More is less: inducing sparsity via overparameterization

Abstract In deep learning, it is common to overparameterize neural networks, that is, to use more parameters than training samples. Quite surprisingly training the neural network via (stochastic) gradient descent leads to models that generalize very well, while classical statistics would suggest overfitting. In order to gain understanding of this implicit bias phenomenon, we study the special case of sparse recovery (compressed sensing) which is of interest on its own. More precisely, in order to reconstruct a vector from underdetermined linear measurements, we introduce a corresponding overparameterized square loss functional, where the vector to be reconstructed is deeply factorized into several vectors. We show that, if there exists an exact solution, vanilla gradient flow for the overparameterized loss functional converges to a good approximation of the solution of minimal $\ell _1$-norm. The latter is well-known to promote sparse solutions. As a by-product, our results significantly improve the sample complexity for compressed sensing via gradient flow/descent on overparameterized models derived in previous works. The theory accurately predicts the recovery rate in numerical experiments. Our proof relies on analyzing a certain Bregman divergence of the flow. This bypasses the obstacles caused by non-convexity and should be of independent interest.

Estimation of block sparsity in compressive sensing

Explicitly using the block structure of the unknown signal can achieve better reconstruction performance in compressive sensing. An unknown signal with block structure can be accurately recovered from under-determined linear measurements provided that it is sufficiently block sparse. However, in practice, the block sparsity level is typically unknown. In this paper, we propose a soft measure of block sparsity [Formula: see text] with [Formula: see text], and present a procedure to estimate it by using multivariate centered isotropic symmetric [Formula: see text]-stable random projections. The limiting distribution of the estimator is given. Simulations are conducted to illustrate our theoretical results.

Asymptotic Performance Prediction for ADMM-Based Compressed Sensing

In this paper, we propose a method to predict the asymptotic performance of the alternating direction method of multipliers (ADMM) for compressed sensing, where we reconstruct an unknown structured signal from its underdetermined linear measurements. The derivation of the proposed method is based on the recently developed convex Gaussian min-max theorem (CGMT), which can be applied to various convex optimization problems to obtain its asymptotic error performance. Our main idea is to analyze the convex subproblem in the update of ADMM iteratively and characterize the asymptotic distribution of the tentative estimate obtained at each iteration. However, since the original CGMT cannot be used directly for the analysis of the iterative updates, we intuitively assume an extended version of CGMT in the derivation of the proposed method. Under the assumption, the result shows that the update equations in ADMM can be decoupled into a scalar-valued stochastic process in the asymptotic regime with the large system limit. From the asymptotic result, we can predict the evolution of the error (e.g., mean-square-error (MSE) and symbol error rate (SER)) in ADMM for large-scale compressed sensing problems. Simulation results show that the empirical performance of ADMM and its prediction are close to each other in sparse vector reconstruction and binary vector reconstruction.

Using Black-Box Compression Algorithms for Phase Retrieval

Compressive phase retrieval refers to the problem of recovering a structured $n$ -dimensional complex-valued vector from its phase-less under-determined linear measurements. The non-linearity of the measurement process makes designing theoretically-analyzable efficient phase retrieval algorithms challenging. As a result, to a great extent, existing recovery algorithms only take advantage of simple structures such as sparsity and its convex generalizations. The goal of this article is to move beyond simple models through employing compression codes. Such codes are typically developed to take advantage of complex signal models to represent the signals as efficiently as possible. In this work, it is shown how an existing compression code can be treated as a black box and integrated into an efficient solution for phase retrieval. First, COmpressive PhasE Retrieval (COPER) optimization, a computationally-intensive compression-based phase retrieval method, is proposed. COPER provides a theoretical framework for studying compression-based phase retrieval. The number of measurements required by COPER is connected to $\kappa $ , the $\alpha $ -dimension (closely related to the rate-distortion dimension) of a given family of compression codes. To finds the solution of COPER, an efficient iterative algorithm called gradient descent for COPER (GD-COPER) is proposed. It is proven that under some mild conditions on the initialization and the compression code, if the number of measurements is larger than $ C \kappa ^{2} \log ^{2}~n$ , where $C$ is a constant, GD-COPER obtains an accurate estimate of the input vector in polynomial time. In the simulation results, JPEG2000 is integrated in GD-COPER to confirm the state-of-the-art performance of the resulting algorithm on real-world images.

Solving Inverse Problems via Auto-Encoders

Compressed sensing (CS) is about recovering a structured signal from its under-determined linear measurements. Starting from sparsity, recovery methods have steadily moved towards more complex structures. Emerging machine learning tools such as generative functions that are based on neural networks are able to learn general complex structures from training data. This makes them potentially powerful tools for designing CS algorithms. Consider a desired class of signals $\cal Q$, ${\cal Q}\subset{R}^n$, and a corresponding generative function $g:{\cal U}^k\rightarrow {R}^n$, ${\cal U}\subset {R}$, such that $\sup_{{\bf x}\in {\cal Q}}\min_{{\bf u}\in{\cal U}^k}{1\over \sqrt{n}}\|g({\bf u})-{\bf x}\|\leq \delta$. A recovery method based on $g$ seeks $g({\bf u})$ with minimum measurement error. In this paper, the performance of such a recovery method is studied, under both noisy and noiseless measurements. In the noiseless case, roughly speaking, it is proven that, as $k$ and $n$ grow without bound and $\delta$ converges to zero, if the number of measurements ($m$) is larger than the input dimension of the generative model ($k$), then asymptotically, almost lossless recovery is possible. Furthermore, the performance of an efficient iterative algorithm based on projected gradient descent is studied. In this case, an auto-encoder is used to define and enforce the source structure at the projection step. The auto-encoder is defined by encoder and decoder (generative) functions $f:{R}^n\to{\cal U}^k$ and $g:{\cal U}^k\to{R}^n$, respectively. We theoretically prove that, roughly, given $m>40k\log{1\over \delta}$ measurements, such an algorithm converges to the vicinity of the desired result, even in the presence of additive white Gaussian noise. Numerical results exploring the effectiveness of the proposed method are presented.

The basins of attraction of the global minimizers of the non-convex sparse spike estimation problem

The sparse spike estimation problem consists in estimating a number of off-the-grid impulsive sources from under-determined linear measurements. Information theoretic results ensure that the minimization of a non-convex functional is able to recover the spikes for adequately chosen measurements (deterministic or random). To solve this problem, methods inspired from the case of finite dimensional sparse estimation where a convex program is used have been proposed. Also greedy heuristics have shown nice practical results. However, little is known on the ideal non-convex minimization method. In this article, we study the shape of the global minimum of this non-convex functional: we give an explicit basin of attraction of the global minimum that shows that the non-convex problem becomes easier as the number of measurements grows. This has important consequences for methods involving descent algorithms (such as the greedy heuristic) and it gives insights for potential improvements of such descent methods.

Toward Theoretically Founded Learning-Based Compressed Sensing

Noiseless compressed sensing refers to the problem of recovering a (high-dimensional) signal from its under-determined linear measurements. For compressed sensing to be feasible, the signal needs to be structured. While the main focus of the field has been on simple structures such as sparsity, there has been a growing interest in moving beyond sparsity and having a comprehensive compressed sensing framework that covers general structures. Two recent approaches that aim at developing such a framework from different perspectives are i) Quantized maximum a posteriori (Q-MAP), a Bayesian method that assumes full knowledge of the source distribution, and ii) Lagrangian minimum entropy pursuit (L-MEP), a universal recovery method that requires no prior knowledge about the distribution of the source. In this paper, by establishing theoretical connections between L-MEP and Q-MAP, it is shown how the two methods are complementary to each other and lead to a theoretically-founded learning-based recovery method that applies to sources with general structures. Unlike a Bayesian or a universal method, a learning-based method is able to extract the source structure from training data. The effect of error in estimating the source structure on the performance of the learning-based compressed sensing recovery method is characterized.

An Orthogonal Matching Pursuit Algorithm Based on Singular Value Decomposition

A recovery algorithm is one of the most important components in compressive sensing. It is responsible for the recovery of sparse coefficients in some bases of the original signal from a set of non-adaptive and underdetermined linear measurements, and it is a key link between the front-end signal sensing system and back-end processing. In this study, an improved orthogonal matching pursuit algorithm based on singular value decomposition is proposed to overcome the limitations of existing algorithms, which effectively eliminates the correlation between the measured values. The results of simulation experiments show that the proposed algorithm significantly improves the average signal-to-noise ratio, and it performs more robustly than the classical orthogonal matching pursuit algorithm.

Discreteness-Aware Approximate Message Passing for Discrete-Valued Vector Reconstruction

This paper considers the reconstruction of a discrete-valued random vector from possibly underdetermined linear measurements using sum-of-absolute-value (SOAV) optimization. The proposed algorithm, referred to as discreteness-aware approximate message passing (DAMP), is based on the idea of approximate message passing (AMP), which has been originally proposed for compressed sensing. The DAMP algorithm has low computational complexity and its performance in the large system limit can be predicted analytically via state evolution framework, where we provide a condition for the exact reconstruction with DAMP in the noise-free case. From the analysis, we also propose a method to determine the parameters of the SOAV optimization. Moreover, based on the state evolution, we provide Bayes optimal DAMP, which has the minimum mean-square-error at each iteration of the algorithm. Simulation results show that the DAMP algorithms can reconstruct the discrete-valued vector from underdetermined linear measurements and the empirical performance agrees with our theoretical results in large-scale systems. When the problem size is not large enough, the SOAV optimization with the proposed parameters can achieve better performance than the DAMP algorithms for high signal-to-noise ratio.

Optimality of 1-norm regularization among weighted 1-norms for sparse recovery: a case study on how to find optimal regularizations

The 1-norm was proven to be a good convex regularizer for the recovery of sparse vectors from under-determined linear measurements. It has been shown that with an appropriate measurement operator, a number of measurements of the order of the sparsity of the signal (up to log factors) is sufficient for stable and robust recovery. More recently, it has been shown that such recovery results can be generalized to more general low-dimensional model sets and (convex) regularizers. These results lead to the following question: to recover a given low-dimensional model set from linear measurements, what is the “best” convex regularizer? To approach this problem, we propose a general framework to define several notions of “best regularizer” with respect to a low-dimensional model. We show in the minimal case of sparse recovery in dimension 3 that the 1-norm is optimal for these notions. However, generalization of such results to the n-dimensional case seems out of reach. To tackle this problem, we propose looser notions of best regularizer and show that the 1-norm is optimal among weighted 1-norms for sparse recovery within this framework.

Sparse Recovery From Multiple Measurement Vectors Using Exponentiated Gradient Updates

In this letter, we address the problem of reconstructing the common nonzero support of multiple joint sparse vectors from their noisy and underdetermined linear measurements. The support recovery problem is formulated as the selection of nonnegative hyperparameters of a correlation-aware, joint sparsity inducing Gaussian prior. The hyperparameters are recovered as a nonnegative sparse solution of covariance-matching constraints formulated in the observation space by solving a sequence of proximal regularized convex optimization problems. For proximal regularization based on Von Neumann Bregman matrix divergence, an exponentiated gradient (EG) update is proposed, which when applied iteratively, converges to hyperparameters with the correct sparse support. Compared to existing multiple measurement vector support recovery algorithms, the proposed multiplicative EG update has a significantly lower computational and storage complexity and takes fewer iterations to converge. We empirically demonstrate that the support-recovery algorithm based on the proposed EG update can solve million variable support recovery problems in tens of seconds. Additionally, by leveraging its correlation-awareness property, the proposed algorithm can recover supports of size as high as $O(m^2)$ from only $m$ linear measurements per joint sparse vector.

On the Gap Between Restricted Isometry Properties and Sparse Recovery Conditions

We consider the problem of recovering sparse vectors from underdetermined linear measurements via $\ell _{p}$ -constrained basis pursuit. Previous analyses of this problem based on generalized restricted isometry properties have suggested that two phenomena occur if $p\neq 2$ . First, one may need substantially more than $s \log (en/s)$ measurements (optimal for $p=2$ ) for uniform recovery of all $s$ -sparse vectors. Second, the matrix that achieves recovery with the optimal number of measurements may not be Gaussian (as for $p=2$ ). We present a new, direct analysis, which shows that in fact neither of these phenomena occur. Via a suitable version of the null space property, we show that a standard Gaussian matrix provides $\ell _{q}/\ell _{1}$ -recovery guarantees for $\ell _{p}$ -constrained basis pursuit in the optimal measurement regime. Our result extends to several heavier-tailed measurement matrices. As an application, we show that one can obtain a consistent reconstruction from uniform scalar quantized measurements in the optimal measurement regime.

Reconstruction of Complex Discrete-Valued Vector via Convex Optimization With Sparse Regularizers

In this paper, we propose a method for the reconstruction of a complex discrete-valued vector from its linear measurements. In particular, we mainly focus on the underdetermined cases, where the number of measurements is less than that of the unknown complex discrete variables, and propose a reconstruction approach of solving an optimization problem called sum of complex sparse regularizers (SCSR) optimization. The sum of sparse regularizers in the objective function can directly utilize the discrete nature of the unknown vector in the complex domain. We also propose an algorithm for the SCSR optimization problem on the basis of alternating direction method of multipliers. For the proposed convex regularizers, we analytically prove that the sequence obtained by the proposed algorithm converges to the optimal solution of the problem. To obtain better reconstruction performance, we further propose an iterative approach named iterative weighted SCSR (IW-SCSR), where we update the parameters in the objective function in each iteration by using the tentative estimate in the previous iteration. Simulation results show that IW-SCSR can reconstruct the complex discrete-valued vector from its underdetermined linear measurements and achieve good performance in the applications of overloaded multiple-input multiple-output signal detection and channel equalization.

A Noniterative Online Bayesian Algorithm for the Recovery of Temporally Correlated Sparse Vectors

In this paper, we address the problem of online (sequential) recovery of temporally correlated sparse vectors sharing a common support, from noisy underdetermined linear measurements. The temporal correlation of the sparse vectors is modeled using a first-order autoregressive process. The online algorithm is formulated using the sparse Bayesian learning framework and is implemented using a sequential expectation-maximization procedure. Our algorithm is noniterative in nature, and requires less computational and memory resources compared to offline processing. We analyze the convergence of the algorithm in the case when the sparse vectors are uncorrelated, using tools from stochastic approximation theory. We show that the sequence of the covariance estimates converge either to the global minimum of the offline equivalent cost function or to the all zero vector, regardless of the sparsity level of the signal. Through numerical results, we demonstrate the efficacy of the proposed online algorithm and compare it with its offline counterpart as well as with existing online sparse vector recovery algorithms. We also illustrate the performance of the algorithm in the context of sparse orthogonal frequency division multiplexing channel estimation.

Communication-Efficient Decentralized Sparse Bayesian Learning of Joint Sparse Signals

We consider the problem of decentralized estimation of multiple joint sparse vectors by a network of nodes from locally acquired noisy and underdetermined linear measurements, when the cost of communication between the nodes is at a premium. We propose an iterative, decentralized Bayesian algorithm called fusion-based distributed sparse Bayesian learning (FB-DSBL) in which the nodes collaborate by exchanging highly compressed messages to learn a common joint sparsity inducing signal prior. The learnt signal prior is subsequently used by each node to compute the maximum a posteriori probability estimate of its respective sparse vector. Since the internode communication cost is expensive, the size of the messages exchanged between nodes is reduced substantially by exchanging only those local signal prior parameters which are associated with the nonzero support detected via multiple composite log-likelihood ratio tests. The average message size is empirically shown to be proportional to the information rate of the unknown vectors. The proposed sparse Bayesian learning (SBL)-based distributed algorithm allows nodes to exploit the underlying joint sparsity of the signals. In turn, this enables the nodes to recover sparse vectors with significantly lower number of measurements compared to the standalone SBL algorithm. The proposed algorithm is interpreted as a degenerate case of a distributed consensus-based stochastic approximation algorithm for finding a fixed point of a function, and its generalized version with Robbins–Monro-type iterations is also developed. Using Monte Carlo simulations, we demonstrate that the proposed FB-DSBL has superior mean squared error and support recovery performance compared to the existing decentralized algorithms with similar or higher communication complexity.

Decentralized Joint-Sparse Signal Recovery: A Sparse Bayesian Learning Approach

This work proposes a decentralized, iterative, Bayesian algorithm called CB-DSBL for in-network estimation of multiple jointly sparse vectors by a network of nodes, using noisy and underdetermined linear measurements. The proposed algorithm exploits the network wide joint sparsity of the un- known sparse vectors to recover them from significantly fewer number of local measurements compared to standalone sparse signal recovery schemes. To reduce the amount of inter-node communication and the associated overheads, the nodes exchange messages with only a small subset of their single hop neighbors. Under this communication scheme, we separately analyze the convergence of the underlying Alternating Directions Method of Multipliers (ADMM) iterations used in our proposed algorithm and establish its linear convergence rate. The findings from the convergence analysis of decentralized ADMM are used to accelerate the convergence of the proposed CB-DSBL algorithm. Using Monte Carlo simulations, we demonstrate the superior signal reconstruction as well as support recovery performance of our proposed algorithm compared to existing decentralized algorithms: DRL-1, DCOMP and DCSP.

Recovery analysis for weighted ℓ1-minimization using the null space property

We study the recovery of sparse signals from underdetermined linear measurements when a potentially erroneous support estimate is available. Our results are twofold. First, we derive necessary and sufficient conditions for signal recovery from compressively sampled measurements using weighted ℓ1-norm minimization. These conditions, which depend on the choice of weights as well as the size and accuracy of the support estimate, are on the null space of the measurement matrix. They can guarantee recovery even when standard ℓ1 minimization fails. Second, we derive bounds on the number of Gaussian measurements for these conditions to be satisfied, i.e., for weighted ℓ1 minimization to successfully recover all sparse signals whose support has been estimated sufficiently accurately. Our bounds show that weighted ℓ1 minimization requires significantly fewer measurements than standard ℓ1 minimization when the support estimate is relatively accurate.

Sparse Recovery With Graph Constraints

Sparse recovery can recover sparse signals from a set of underdetermined linear measurements. Motivated by the need to monitor the key characteristics of large-scale networks from a limited number of measurements, this paper addresses the problem of recovering sparse signals in the presence of network topological constraints. Unlike conventional sparse recovery where a measurement can contain any subset of the unknown variables, we use a graph to characterize the topological constraints and allow an additive measurement over nodes (unknown variables) only if they induce a connected subgraph. We provide explicit measurement constructions for several special graphs, and the number of measurements by our construction is less than that needed by existing random constructions. Moreover, our construction for a line network is provably optimal in the sense that it requires the minimum number of measurements. A measurement construction algorithm for general graphs is also proposed and evaluated. For any given graph <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"> <tex-math notation="LaTeX">$G$ </tex-math></inline-formula> with <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"> <tex-math notation="LaTeX">$n$ </tex-math></inline-formula> nodes, we derive bounds of the minimum number of measurements needed to recover any <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"> <tex-math notation="LaTeX">$k$ </tex-math></inline-formula> -sparse vector over <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"> <tex-math notation="LaTeX">$G$ </tex-math></inline-formula> ( <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"> <tex-math notation="LaTeX">$M^{G}_{k,n}$ </tex-math></inline-formula> ). Using the Erdős-Rényi random graph as an example, we characterize the dependence of <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"> <tex-math notation="LaTeX">$M^{G}_{k,n}$ </tex-math></inline-formula> on the graph structure. This paper suggests that <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"> <tex-math notation="LaTeX">$M^{G}_{k,n}$ </tex-math></inline-formula> may serve as a graph connectivity metric.

Testable uniqueness conditions for empirical assessment of undersampling levels in total variation-regularized X-ray CT

We study recoverability in fan-beam computed tomography (CT) with sparsity and total variation priors: how many underdetermined linear measurements suffice for recovering images of given sparsity? Results from compressed sensing (CS) establish such conditions for example for random measurements, but not for CT. Recoverability is typically tested by checking whether a computed solution recovers the original. This approach cannot guarantee solution uniqueness and the recoverability decision therefore depends on the optimization algorithm. We propose new computational methods to test recoverability by verifying solution uniqueness conditions. Using both reconstruction and uniqueness testing, we empirically study the number of CT measurements sufficient for recovery on new classes of sparse test images. We demonstrate an average-case relation between sparsity and sufficient sampling and observe a sharp phase transition as known from CS, but never established for CT. In addition to assessing recoverability more reliably, we show that uniqueness tests are often the faster option.