Optimal sparse recovery for multi-sensor measurements

The problem of multiple sensors simultaneously acquiring measurements of a single object can be found in many applications. In this paper, we present the optimal recovery guarantees for the recovery of compressible signals from multi-sensor measurements using compressed sensing. In the first half of the paper, we present both uniform and nonuniform recovery guarantees for the conventional sparse signal model in a so-called distinct sensing scenario. In the second half, using the so-called sparse and distributed signal model, we present nonuniform recovery guarantees which effectively broaden the class of sensing scenarios for which optimal recovery is possible, including to the so-called identical sampling scenario. To verify our recovery guarantees we provide several numerical results including phase transition curves and numerically-computed bounds.

Parallel acquisition systems arise in various applications in order to moderate problems caused by insufficient measurements in single-sensor systems. These systems allow simultaneous data acquisition in multiple sensors, thus alleviating such problems by providing more overall measurements. In this work we consider the combination of compressed sensing with parallel acquisition. We establish the theoretical improvements of such systems by providing recovery guarantees for which, subject to appropriate conditions, the number of measurements required per sensor decreases linearly with the total number of sensors. Throughout, we consider two different sampling scenarios -- distinct (corresponding to independent sampling in each sensor) and identical (corresponding to dependent sampling between sensors) -- and a general mathematical framework that allows for a wide range of sensing matrices (e.g., subgaussian random matrices, subsampled isometries, random convolutions and random Toeplitz matrices). We also consider not just the standard sparse signal model, but also the so-called sparse in levels signal model. This model includes both sparse and distributed signals and clustered sparse signals. As our results show, optimal recovery guarantees for both distinct and identical sampling are possible under much broader conditions on the so-called sensor profile matrices (which characterize environmental conditions between a source and the sensors) for the sparse in levels model than for the sparse model. To verify our recovery guarantees we provide numerical results showing phase transitions for a number of different multi-sensor environments.

The importance of sparse signal structures has been recognized in a plethora of applications ranging from medical imaging to group disease testing to radar technology. It has been shown in practice that various signals of interest may be (approximately) sparsely modeled, and that sparse modeling is often beneficial, or even indispensable to signal recovery. Alongside an increase in applications, a rich theory of sparse and compressible signal recovery has recently been developed under the names compressed sensing (CS) and sparse approximation (SA). This revolutionary research has demonstrated that many signals can be recovered from severely undersampled measurements by taking advantage of their inherent low-dimensional structure. More recently, an offshoot of CS and SA has been a focus of research on other low-dimensional signal structures such as matrices of low rank. Low-rank matrix recovery (LRMR) is demonstrating a rapidly growing array of important applications such as quantum state tomography, triangulation from incomplete distance measurements, recommender systems (e.g., the Netflix problem), and system identification and control. In this dissertation, we examine CS, SA, and LRMR from a theoretical perspective. We consider a variety of different measurement and signal models, both random and deterministic, and mainly ask two questions. How many measurements are necessary? How large is the recovery error? We give theoretical lower bounds for both of these questions, including oracle and minimax lower bounds for the error. However, the main emphasis of the thesis is to demonstrate the efficacy of convex optimization---in particular l1 and nuclear-norm minimization based programs---in CS, SA, and LRMR. We derive upper bounds for the number of measurements required and the error derived by convex optimization, which in many cases match the lower bounds up to constant or logarithmic factors. The majority of these results do not require the restricted isometry property (RIP), a ubiquitous condition in the literature.

Joint near-isometry and optimal sparse recovery: Nonuniform recovery from multi-sensor measurements

To evaluate the feasibility of using compressed sensing (CS) to accelerate 3D-T1ρ mapping of cartilage and to reduce total scan times without degrading the estimation of T1ρ relaxation times. Fully sampled 3D-T1ρ datasets were retrospectively undersampled by factors 2-10. CS reconstruction using 12 different sparsifying transforms were compared, including finite differences, temporal and spatial wavelets, learned transforms using principal component analysis (PCA) and K-means singular value decomposition (K-SVD), explicit exponential models, low rank and low rank plus sparse models. Spatial filtering prior to T1ρ parameter estimation was also tested. Synthetic phantom (n = 6) and in vivo human knee cartilage datasets (n = 7) were included. Most CS methods performed satisfactorily for an acceleration factor (AF) of 2, with relative T1ρ error lower than 4.5%. Some sparsifying transforms, such as spatiotemporal finite difference (STFD), exponential dictionaries (EXP) and low rank combined with spatial finite difference (L+S SFD) significantly improved this performance, reaching average relative T1ρ error below 6.5% on T1ρ relaxation times with AF up to 10, when spatial filtering was used before T1ρ fitting, at the expense of smoothing the T1ρ maps. The STFD achieved 5.1% error at AF = 10 with spatial filtering prior to T1ρ fitting. Accelerating 3D-T1ρ mapping of cartilage with CS is feasible up to AF of 10 when using STFD, EXP or L+S SFD regularizers. These three best CS methods performed satisfactorily on synthetic phantom and in vivo knee cartilage for AFs up to 10, with T1ρ error of 6.5%.

<p>The presence of missing entries pose a hindrance to data analysis and interpretation. The missing entries may occur due to a variety of reasons such as sensor malfunction, limited acquisition time or unavailability of information. In this thesis, we present algorithms to analyze and complete data which contain several missing entries. We consider the recovery of a group of signals, given a few under-sampled and noisy measurements of each signal. This involves solving ill-posed inverse problems, since the number of available measurements are considerably fewer than the dimensionality of the signal that we aim to recover. In this work, we consider different data models to enable joint recovery of the signals from their measurements, as opposed to the independent recovery of each signal. This prior knowledge makes the inverse problems well-posed. While compressive sensing techniques have been proposed for low-rank or sparse models, such techniques have not been studied to the same extent for other models such as data appearing in clusters or lying on a low-dimensional manifold. In this work, we consider several data models arising in different applications, and present some theoretical guarantees for the joint reconstruction of the signals from few measurements. Our proposed techniques make use of fusion penalties, which are regularizers that promote solutions with similarity between certain pairs of signals.</p><p>The first model that we consider is that of points lying on a low-dimensional manifold, embedded in high dimensional ambient space. This model is apt for describing a collection of signals, each of which is a function of only a few parameters; the manifold dimension is equal to the number of parameters. We propose a technique to recover a series of such signals, given a few measurements for each signal. We demonstrate this in the context of dynamic Magnetic Resonance Imaging (MRI) reconstruction, where only a few Fourier measurements are available for each time frame. A novel acquisition scheme enables us to detect the neighbours of each frame on the manifold. We then recover each frame by enforcing similarity with its neighbours. The proposed scheme is used to enable fast free-breathing cardiac and speech MRI scans.</p><p>Next, we consider the recovery of curves/surfaces from few sampled points. We model the curves as the zero-level set of a trigonometric polynomial, whose bandwidth controls the complexity of the curve. We present theoretical results for the minimum number of samples required to uniquely identify the curve. We show that the null-space vectors of high dimensional feature maps of these points can be used to recover the curve. The method is demonstrated on the recovery of the structure of DNA filaments from a few clicked points. This idea is then extended to recover data lying on a high-dimensional surface from few measurements. The formulated algorithm has similarities to our algorithm for recovering points on a manifold. Hence, we apply the above ideas to the cardiac MRI reconstruction problem, and are able to show better image quality with reduced computational complexity.</p><p>Finally, we consider the case where the data is organized into clusters. The goal is to recover the true clustering of the data, even when a few features of each data point is unknown. We propose a fusion-penalty based optimization problem to cluster data reliably in the presence of missing entries, and present theoretical guarantees for successful recovery of the correct clusters. We next propose a computationally efficient algorithm to solve a relaxation of this problem. We demonstrate that our algorithm reliably recovers the true clusters in the presence of large fractions of missing entries on simulated and real datasets.</p><p>This work thus results in several theoretical insights and solutions to different practical problems which involve reconstructing and analyzing data with missing entries. The fusion penalties that are used in each of the above models are obtained directly as a result of model assumptions. The proposed algorithms show very promising results on several real datasets, and we believe that they are general enough to be easily extended to several other practical applications.</p>

In compressed sensing (CS) framework, a signal is sampled below Nyquist rate, and the acquired samples are generally random in nature. Thus, for efficient estimation of the actual signal, the sensing matrix must preserve the relative distances among the underlying sparse vectors. Provided this condition is fulfilled, we show that CS samples will also preserve the envelope of the actual signal. Exploiting this envelope preserving property of CS samples, we propose a new fast method which is able to extract prototype signals from compressive samples for efficient sparse representation and recovery of signals. These prototype signals are orthogonal intrinsic mode functions (IMFs) extracted from CS samples using empirical mode decomposition (EMD), which is one of the popular methods to capture the envelope of a signal. The extracted IMFs are used to seed the dictionary without even comprehending the original signal or the sensing matrix. Moreover, one can update the dictionary on-line as new CS samples are available. In particularly, to recover first L signals (∊ Rn) at the decoder, one can seed the dictionary in just O(nL log n) operations, that is far less as compared to existing approaches. The efficiency of the proposed approach is demonstrated experimentally for recovery of speech signals.

This survey provides a brief introduction to compressed sensing as well as several major algorithms to solve it and its various applications to communications systems. We firstly review linear simultaneous equations as ill-posed inverse problems, since the idea of compressed sensing could be best understood in the context of the linear equations. Then, we consider the problem of compressed sensing as an underdetermined linear system with a prior information that the true solution is sparse, and explain the sparse signal recovery based on l1 optimization, which plays the central role in compressed sensing, with some intuitive explanations on the optimization problem. Moreover, we introduce some important properties of the sensing matrix in order to establish the guarantee of the exact recovery of sparse signals from the underdetermined system. After summarizing several major algorithms to obtain a sparse solution focusing on the l1 optimization and the greedy approaches, we introduce applications of compressed sensing to communications systems, such as wireless channel estimation, wireless sensor network, network tomography, cognitive radio, array signal processing, multiple access scheme, and networked control.

Sparse recovery with coherent tight frames via analysis Dantzig selector and analysis LASSO

Data compression is crucial for resource-constrained signal acquisition and wireless transmission applications with limited data bandwidth. In such applications, wireless data transmission dominates the energy consumption, and the limited telemetry bandwidth could be overwhelmed by the large amount of data generated from multiple sensors. Conventional data compression techniques are computationally intensive, consume large silicon area and offset the energy benefits from reduced data transmission. Recently, compressed sensing (CS) has shown potential in achieving compression performance comparable to previous methods but it has simpler hardware. Especially, one-bit CS theory proves that the signs of compressed measurements contain sufficient information about signal reconstruction, gives that the signals are sparse or compressible in specific dictionaries, thus demonstrating its potential in energy-constrained signal recording and wireless transmission applications. However, the sparsity assumption is too restrictive in many actual scenarios, especially when it is difficult to seek sparse representation for signals. In this paper, a novel one-bit CS method is proposed to reconstruct the signals that are difficult to represent with traditional sparse models. It is capable of recovering signal with comparable compression ratio but avoiding the dictionary selection procedure.The proposed method consists of two parts. 1) The block sparse model is adopted to enforce the structured sparsity of the signals. It not only overcomes the drawbacks of conventional sparse models but also enhances the signal representation accuracy. 2) The probabilistic model of one-bit CS procedure is constructed. Because of the existence of logistic function in probabilistic model of one-bit CS, the Bayesian inference cannot be used to proceed, and the variational Bayesian inference algorithm is developed to reconstruct the original signals from one-bit measurements.Various experiments on different quantities of compressed measurements and iterations are carried out to evaluate the recovery performance of the proposed approach. The photoplethysmography (PPG) signals recorded from subject wrist (dorsal locations) by using PPG sensors built in a wristband are selected as the validation data because they are difficult to represent with traditional sparse dictionaries. The experimental results reveal that the proposed approach outperforms the state-of-the-art one-bit CS method in terms of both reconstruction accuracy and convergence rate.Compared with prior method on one-bit CS, the proposed method shows competitive or superior performance in three aspects. Firstly, by adopting the block sparse model, the proposed method improves the capability to compress signals that are difficult to represent with traditional sparse models, thus making it more practical for long term and real applications. Secondly, by embedding the statistical properties of the one-bit measurements into the recovery algorithm, the proposed method outperforms other one-bit CS methods in terms of both reconstruction performance and convergence speed. Finally, energy and computational efficiency of the proposed method make it an ideal candidate for resource-constrained, large scale, multiple channel signal acquisition and transmission applications.

The theory and techniques of compressed sensing (CS) have shown their potential as a breakthrough in accelerating k-space data acquisition for parallel magnetic resonance imaging (pMRI). However, the performance of CS reconstruction models in pMRI has not been fully maximized, and CS recovery guarantees for pMRI are largely absent. To improve reconstruction accuracy from parsimonious amounts of k-space data while maintaining flexibility, a new CS SENSitivity Encoding (SENSE) pMRI reconstruction framework promoting joint sparsity (JS) across channels (JS CS SENSE) is proposed in this paper. The recovery guarantee derived for the proposed JS CS SENSE model is demonstrated to be better than that of the conventional CS SENSE model and similar to that of the coil-by-coil CS model. The flexibility of the new model is better than the coil-by-coil CS model and the same as that of CS SENSE. For fast image reconstruction and fair comparisons, all the introduced CS-based constrained optimization problems are solved with split Bregman, variable splitting, and combined-variable splitting techniques. For the JS CS SENSE model in particular, these techniques lead to an efficient algorithm. Numerical experiments show that the reconstruction accuracy is significantly improved by JS CS SENSE compared with the conventional CS SENSE. In addition, an accurate residual-JS regularized sensitivity estimation model is also proposed and extended to calibration-less (CaL) JS CS SENSE. Numerical results show that CaL JS CS SENSE outperforms other state-of-the-art CS-based calibration-less methods in particular for reconstructing non-piecewise constant images.

The past decade has witnessed the emergence of compressed sensing as a way of acquiring sparsely representable signals in a compressed form. These developments have greatly motivated research in sparse signal recovery, which lies at the heart of compressed sensing, and which has recently found its use in altogether new applications. In the first part of this thesis we study the theoretical aspects of jointsparse recovery by means of sum-of-norms minimization, and the ReMBo-`1 algorithm, which combines boosting techniques with `1-minimization. For the sum-of-norms approach we derive necessary and sufficient conditions for recovery, by extending existing results to the joint-sparse setting. We focus in particular on minimization of the sum of `1, and `2 norms, and give concrete examples where recovery succeeds with one formulation but not with the other. We base our analysis of ReMBo-`1 on its geometrical interpretation, which leads to a study of orthant intersections with randomly oriented subspaces. This work establishes a clear picture of the mechanics behind the method, and explains the different aspects of its performance. The second part and main contribution of this thesis is the development of a framework for solving a wide class of convex optimization problems for sparse recovery. We provide a detailed account of the application of the framework on several problems, but also consider its limitations. The framework has been implemented in the spgl1 algorithm, which is already well established as an effective solver. Numerical results show that our algorithm is state-of-the-art, and compares favorably even with solvers for the easier—but less natural— Lagrangian formulations. The last part of this thesis discusses two supporting software packages: sparco, which provides a suite of test problems for sparse recovery, and spot, a Matlab toolbox for the creation and manipulation of linear operators. spot greatly facilitates rapid prototyping in sparse recovery and compressed sensing, where linear operators form the elementary building blocks. Following the practice of reproducible research, all code used for the experiments and generation of figures is available online at http://www.cs.ubc.ca/labs/scl/thesis/09vandenBerg/

Compressive sensing (CS) has gained a lot of attention in recent years due to its benefits in saving measurement time and storage cost in many applications including biomedical imaging, wireless communications, image reconstruction, remote sensing, and so on. The CS framework enables signal recovery from a small number of linear measurements with an acceptable fidelity taking advantage of signal sparsity in some potentially unknown domain. The core idea of different variants of CS methods is incorporating prior knowledge about the input signal (e.g., prior distribution or sparsity of signals) into the recovery algorithm to restrict the search space and enhance the signal recovery performance. However, the accuracy of signal reconstruction can be significantly compromised if the designed and implemented measurement matrices do not fully match. Often times, the measurement matrix mismatch is treated as an additional noise term in the recovery algorithm ignoring the fact that this mismatch is a learnable quantity which includes random but constant or slow-varying terms during the lifetime of the measurement system. In this paper, we consider this problem for a simple case of Gaussian prior with a sparsity-driven diagonal covariance matrix and find strict bounds on the deviation of the reconstructed signal from the optimal case of fully known measurement matrix based on the properties of the mismatch matrix. The obtained bounds are general, and hence can be used to assess the performance of learning algorithms designed for learning measurement matrix uncertainty and eliminating its effect from the signal recovery. We provide numerical results to illustrate this concept in real-world applications.

We investigate the recovery of signals exhibiting a sparse representation in a general (i.e., possibly redundant or incomplete) dictionary that are corrupted by additive noise admitting a sparse representation in another general dictionary. This setup covers a wide range of applications, such as image inpainting, super-resolution, signal separation, and the recovery of signals that are corrupted by, e.g., clipping, impulse noise, or narrowband interference. We present deterministic recovery guarantees based on a recently developed uncertainty relation and provide corresponding recovery algorithms. The recovery guarantees we find depend on the signal and noise sparsity levels, on the coherence parameters of the involved dictionaries, and on the amount of prior knowledge on the support sets of signal and noise.

Compressive sensing is an emerging research field that has applications in signal processing, error correction, medical imaging, seismology, and many more other areas. It promises to efficiently recover a sparse signal vector via a much smaller number of linear measurements than its dimension. Naturally, how to design these linear measurements, how to construct the original high-dimensional signals efficiently and accurately, and how to analyze the sparse signal recovery algorithms are important issues in the developments of compressive sensing. This thesis is devoted to addressing these fundamental issues in the field of compressive sensing. In compressive sensing, random measurement matrices are generally used and e1 minimization algorithms often use linear programming or other optimization methods to recover the sparse signal vectors. But explicitly constructible measurement matrices providing performance guarantees were elusive and e1 minimization algorithms are often very demanding in computational complexity for applications involving very large problem dimensions. In chapter 2, we propose and discuss a compressive sensing scheme with deterministic performance guarantees using deterministic explicitly constructible expander graph-based measurement matrices and show that the sparse signal recovery can be achieved with linear complexity. This is the first of such a kind of compressive sensing scheme with linear decoding complexity, deterministic performance guarantees of linear sparsity recovery, and deterministic explicitly constructible measurement matrices. The popular and powerful e1 minimization algorithms generally give better sparsity recovery performances than known greedy decoding algorithms. In chapter 3, starting from a necessary and sufficient null-space condition for achieving a certain signal recovery accuracy, using high-dimensional geometry, we give a unified null-space Grassmann angle-based analytical framework for compressive sensing. This new framework gives sharp quantitative trade-offs between the signal sparsity and the recovery accuracy of the e 1 optimization for approximately sparse signal. Our results concern the fundamental balancedness properties of linear subspaces and so may be of independent mathematical interest. The conventional approach to compressed sensing assumes no prior information on the unknown signal other than the fact that it is sufficiently sparse over a particular basis. In many applications, however, additional prior information is available. In chapter 4, we will consider a particular model for the sparse signal that assigns a probability of being zero or nonzero to each entry of the unknown vector. The standard compressed sensing model is therefore a special case where these probabilities are all equal. Following the introduction of the null-space Grassmann angle-based analytical framework in this thesis, we are able to characterize the optimal recoverable sparsity thresholds using weighted e1 minimization algorithms with the prior information. The roles of e1 minimization algorithm in recovering sparse signals from incomplete measurements are now well understood, and sharp recoverable sparsity thresholds for e1 minimization have been obtained. The iterative reweighted e1 minimization algorithms or related algorithms have been empirically observed to boost the recoverable sparsity thresholds for certain types of signals, but no rigorous theoretical results have been established to prove this fact. In chapter 5, we try to provide a theoretical foundation for analyzing the iterative reweighted e 1 algorithms. In particular, we show that for a nontrivial class of signals, the iterative reweighted e1 minimization can indeed deliver recoverable sparsity thresholds larger thanthe e1 minimization. Again, our results are based on the null-space Grassmann angle-based analytical framework. Evolving from compressive sensing problems, where we are interested in recovering sparse vector signals from compressed linear measurements, we will turn our attention to recovering matrices of low rank from compressed linear measurements in chapter 6, which is a challenging problem that arises in many applications in machine learning, control theory, and discrete geometry. This class of optimization problems is NP-HARD, and for most practical problems there are no efficient algorithms that yield exact solutions. A popular heuristic replaces the rank function with the nuclear norm of the decision variable and has been shown to provide the optimal low rank solution in a variety of scenarios. We analytically assess the practical performance of this heuristic for finding the minimum rank matrix subject to linear constraints. We start from the characterization of a necessary and sufficient condition that determines when this heuristic finds the minimum rank solution. We then obtain probabilistic bounds on the matrix dimensions and rank and the number of constraints, such that our conditions for success are satisfied for almost all linear constraint sets as the matrix dimensions tend to infinity. Empirical evidence shows that these probabilistic bounds provide accurate predictions of the heuristic's performance in non-asymptotic scenarios.