Iterative Hard Thresholding Algorithm Research Articles

A New and Improved Quantitative Recovery Analysis for Iterative Hard Thresholding Algorithms in Compressed Sensing

We present a new recovery analysis for a standard compressed sensing algorithm, Iterative Hard Thresholding (IHT) (Blumensath and Davies, 2008), which considers the fixed points of the algorithm. In the context of arbitrary measurement matrices, we derive a sufficient condition for the convergence of IHT to a fixed point and a necessary condition for the existence of fixed points. These conditions allow us to perform a sparse signal recovery analysis in the deterministic noiseless case by implying that the original sparse signal is the unique fixed point and limit point of IHT, and in the case of Gaussian measurement matrices and noise by generating a bound on the approximation error of the IHT limit as a multiple of the noise level. By generalizing the notion of fixed points, we extend our analysis to the variable stepsize Normalised IHT (Blumensath and Davies, 2010). For both stepsize schemes, we obtain lower bounds on asymptotic phase transitions in a proportional-dimensional framework, quantifying the sparsity/undersampling tradeoff for which recovery is guaranteed. Exploiting the reasonable average-case assumption that the underlying signal and measurement matrix are independent, comparison with previous results within this framework shows a substantial quantitative improvement.

Minimum n-rank approximation via iterative hard thresholding

The problem of recovering a low n-rank tensor is an extension of sparse recovery problem from the low dimensional space (matrix space) to the high dimensional space (tensor space) and has many applications in computer vision and graphics such as image inpainting and video inpainting. In this paper, we consider a new tensor recovery model, named as minimum n-rank approximation (MnRA), and propose an appropriate iterative hard thresholding algorithm with giving the upper bound of the n-rank in advance. The convergence analysis of the proposed algorithm is also presented. Particularly, we show that for the noiseless case, the linear convergence with rate 12 can be obtained for the proposed algorithm under proper conditions. Additionally, combining an effective heuristic for determining n-rank, we can also apply the proposed algorithm to solve MnRA when n-rank is unknown in advance. Some preliminary numerical results on randomly generated and real low n-rank tensor completion problems are reported, which show the efficiency of the proposed algorithms.

Conjugate Gradient Iterative Hard Thresholding: Observed Noise Stability for Compressed Sensing

Conjugate gradient iterative hard thresholding (CGIHT) for compressed sensing combines the low per iteration computational cost of simple line search iterative hard thresholding algorithms with the improved convergence rates of more sophisticated sparse approximation algorithms. This paper shows that the average case performance of CGIHT is robust to additive noise well beyond its theoretical worst case guarantees and, in this setting, is typically the fastest iterative hard thresholding algorithm for sparse approximation. Moreover, CGIHT is observed to benefit more than other iterative hard thresholding algorithms when jointly considering multiple sparse vectors whose sparsity patterns coincide.

Priori-Information Aided Iterative Hard Threshold: A Low-Complexity High-Accuracy Compressive Sensing Based Channel Estimation for TDS-OFDM

This paper develops a low-complexity channel estimation (CE) scheme based on compressive sensing (CS) for time-domain synchronous (TDS) orthogonal frequency-division multiplexing (OFDM) to overcome the performance loss under doubly selective fading channels. Specifically, an overlap-add method of the time-domain training sequence is first proposed to obtain the coarse estimates of the channel length, path delays and path gains of the wireless channel, by exploiting the channel's temporal correlation to improve the robustness of the coarse CE under the severe fading channel with long delay spread. We then propose the priori-information aided (PA) iterative hard threshold (IHT) algorithm, which utilizes the priori information of the acquired coarse estimate for the wireless channel and therefore is capable of obtaining an accurate channel estimate of the doubly selective fading channel. Compared with the classical IHT algorithm whose convergence requires the $l_2$ norm of the measurement matrix being less than 1, the proposed PA-IHT algorithm exploits the priori information acquired to remove such a limitation as well as to reduce the number of required iterations. Compared with the existing CS based CE method for TDS-OFDM, the proposed PA-IHT algorithm significantly reduces the computational complexity of CE as well as enhances the CE accuracy. Simulation results demonstrate that, without sacrificing spectral efficiency and changing the current TDS-OFDM signal structure, the proposed scheme performs better than the existing CE schemes for TDS-OFDM in various scenarios, especially under severely doubly selective fading channels.

A Three-Phase Top-k Query Based Distributed Data Collection Scheme in Wireless Sensor Networks

We propose a three-phase top-[Formula: see text] query based distributed data collection scheme which is designed for clustered or multisink wireless sensor networks. The proposed scheme consists of a distributed iterative hard thresholding algorithm and a three-phase top-[Formula: see text] query algorithm. In the distributed iterative hard thresholding algorithm, the cluster heads or sink nodes reconstruct the compressed data in a distributed and cooperative manner. Meanwhile, the top-[Formula: see text] query operation in the above algorithm is realized by pruning unnecessary elements among cluster heads or sink nodes in the three-phase top-[Formula: see text] query algorithm. Simulation results show that there is no obvious difference in the performance of data reconstruction between our proposed scheme and existing compressive sensing theory based data collection schemes. However, both the number of interactions and the amount of transmitted data among cluster heads or sink nodes can be effectively reduced in the proposed scheme. The performance of the proposed scheme is analyzed in detail in this paper to support the claims.

Image reconstruction from few views by ℓ0-norm optimization

In the medical computer tomography (CT) field, total variation (TV), which is the ℓ1-norm of the discrete gradient transform (DGT), is widely used as regularization based on the compressive sensing (CS) theory. To overcome the TV model's disadvantageous tendency of uniformly penalizing the image gradient and over smoothing the low-contrast structures, an iterative algorithm based on the ℓ0-norm optimization of the DGT is proposed. In order to rise to the challenges introduced by the ℓ0-norm DGT, the algorithm uses a pseudo-inverse transform of DGT and adapts an iterative hard thresholding (IHT) algorithm, whose convergence and effective efficiency have been theoretically proven. The simulation demonstrates our conclusions and indicates that the algorithm proposed in this paper can obviously improve the reconstruction quality.

Distributed Compressed Sensing for Static and Time-Varying Networks

We consider the problem of in-network compressed sensing from distributed measurements. Every agent has a set of measurements of a signal $x$, and the objective is for the agents to recover $x$ from their collective measurements using only communication with neighbors in the network. Our distributed approach to this problem is based on the centralized Iterative Hard Thresholding algorithm (IHT). We first present a distributed IHT algorithm for static networks that leverages standard tools from distributed computing to execute in-network computations with minimized bandwidth consumption. Next, we address distributed signal recovery in networks with time-varying topologies. The network dynamics necessarily introduce inaccuracies to our in-network computations. To accommodate these inaccuracies, we show how centralized IHT can be extended to include inexact computations while still providing the same recovery guarantees as the original IHT algorithm. We then leverage these new theoretical results to develop a distributed version of IHT for time-varying networks. Evaluations show that our distributed algorithms for both static and time-varying networks outperform previously proposed solutions in time and bandwidth by several orders of magnitude.

Lorentzian Iterative Hard Thresholding: Robust Compressed Sensing With Prior Information

Commonly employed reconstruction algorithms in compressed sensing (CS) use the $L_2$ norm as the metric for the residual error. However, it is well-known that least squares (LS) based estimators are highly sensitive to outliers present in the measurement vector leading to a poor performance when the noise no longer follows the Gaussian assumption but, instead, is better characterized by heavier-than-Gaussian tailed distributions. In this paper, we propose a robust iterative hard Thresholding (IHT) algorithm for reconstructing sparse signals in the presence of impulsive noise. To address this problem, we use a Lorentzian cost function instead of the $L_2$ cost function employed by the traditional IHT algorithm. We also modify the algorithm to incorporate prior signal information in the recovery process. Specifically, we study the case of CS with partially known support. The proposed algorithm is a fast method with computational load comparable to the LS based IHT, whilst having the advantage of robustness against heavy-tailed impulsive noise. Sufficient conditions for stability are studied and a reconstruction error bound is derived. We also derive sufficient conditions for stable sparse signal recovery with partially known support. Theoretical analysis shows that including prior support information relaxes the conditions for successful reconstruction. Simulation results demonstrate that the Lorentzian-based IHT algorithm significantly outperform commonly employed sparse reconstruction techniques in impulsive environments, while providing comparable performance in less demanding, light-tailed environments. Numerical results also demonstrate that the partially known support inclusion improves the performance of the proposed algorithm, thereby requiring fewer samples to yield an approximate reconstruction.

Compressed Sensing With Nonlinear Observations and Related Nonlinear Optimization Problems

Nonconvex constraints are valuable regularizers in many optimization problems. In particular, sparsity constraints have had a significant impact on sampling theory, where they are used in compressed sensing and allow structured signals to be sampled far below the rate traditionally prescribed. Nearly, all of the theory developed for compressed sensing signal recovery assumes that samples are taken using linear measurements. In this paper, we instead address the compressed sensing recovery problem in a setting where the observations are nonlinear. We show that, under conditions similar to those required in the linear setting, the iterative hard thresholding algorithm can be used to accurately recover sparse or structured signals from few nonlinear observations. Similar ideas can also be developed in a more general nonlinear optimization framework. In the second part of this paper, we therefore present related result that shows how this can be done under sparsity and union of subspaces constraints, whenever a generalization of the restricted isometry property traditionally imposed on the compressed sensing system holds.

Geophysical data sparse reconstruction based on L0-norm minimization

Missing data are a problem in geophysical surveys, and interpolation and reconstruction of missing data is part of the data processing and interpretation. Based on the sparseness of the geophysical data or the transform domain, we can improve the accuracy and stability of the reconstruction by transforming it to a sparse optimization problem. In this paper, we propose a mathematical model for the sparse reconstruction of data based on the L0-norm minimization. Furthermore, we discuss two types of the approximation algorithm for the L0-norm minimization according to the size and characteristics of the geophysical data: namely, the iteratively reweighted least-squares algorithm and the fast iterative hard thresholding algorithm. Theoretical and numerical analysis showed that applying the iteratively reweighted least-squares algorithm to the reconstruction of potential field data exploits its fast convergence rate, short calculation time, and high precision, whereas the fast iterative hard thresholding algorithm is more suitable for processing seismic data, moreover, its computational efficiency is better than that of the traditional iterative hard thresholding algorithm.

Robust 1-Bit Compressive Sensing via Binary Stable Embeddings of Sparse Vectors

The compressive sensing (CS) framework aims to ease the burden on analog-to-digital converters (ADCs) by reducing the sampling rate required to acquire and stably recover sparse signals. Practical ADCs not only sample but also quantize each measurement to a finite number of bits; moreover, there is an inverse relationship between the achievable sampling rate and the bit depth. In this paper, we investigate an alternative CS approach that shifts the emphasis from the sampling rate to the number of bits per measurement. In particular, we explore the extreme case of 1-bit CS measurements, which capture just their sign. Our results come in two flavors. First, we consider ideal reconstruction from noiseless 1-bit measurements and provide a lower bound on the best achievable reconstruction error. We also demonstrate that i.i.d. random Gaussian matrices provide measurement mappings that, with overwhelming probability, achieve nearly optimal error decay. Next, we consider reconstruction robustness to measurement errors and noise and introduce the binary e-stable embedding property, which characterizes the robustness of the measurement process to sign changes. We show that the same class of matrices that provide almost optimal noiseless performance also enable such a robust mapping. On the practical side, we introduce the binary iterative hard thresholding algorithm for signal reconstruction from 1-bit measurements that offers state-of-the-art performance.

A unifying analysis of projected gradient descent for [formula omitted]-constrained least squares

In this paper we study the performance of the Projected Gradient Descent (PGD) algorithm for ℓp-constrained least squares problems that arise in the framework of compressed sensing. Relying on the restricted isometry property, we provide convergence guarantees for this algorithm for the entire range of 0⩽p⩽1, that include and generalize the existing results for the iterative hard thresholding algorithm and provide a new accuracy guarantee for the iterative soft thresholding algorithm as special cases. Our results suggest that in this group of algorithms, as p increases from zero to one, conditions required to guarantee accuracy become stricter and robustness to noise deteriorates.

L1/2 regularization: a thresholding representation theory and a fast solver.

The special importance of L1/2 regularization has been recognized in recent studies on sparse modeling (particularly on compressed sensing). The L1/2 regularization, however, leads to a nonconvex, nonsmooth, and non-Lipschitz optimization problem that is difficult to solve fast and efficiently. In this paper, through developing a threshoding representation theory for L1/2 regularization, we propose an iterative half thresholding algorithm for fast solution of L1/2 regularization, corresponding to the well-known iterative soft thresholding algorithm for L1 regularization, and the iterative hard thresholding algorithm for L0 regularization. We prove the existence of the resolvent of gradient of ||x||1/2(1/2), calculate its analytic expression, and establish an alternative feature theorem on solutions of L1/2 regularization, based on which a thresholding representation of solutions of L1/2 regularization is derived and an optimal regularization parameter setting rule is formulated. The developed theory provides a successful practice of extension of the well- known Moreau's proximity forward-backward splitting theory to the L1/2 regularization case. We verify the convergence of the iterative half thresholding algorithm and provide a series of experiments to assess performance of the algorithm. The experiments show that the half algorithm is effective, efficient, and can be accepted as a fast solver for L1/2 regularization. With the new algorithm, we conduct a phase diagram study to further demonstrate the superiority of L1/2 regularization over L1 regularization.

Accelerated iterative hard thresholding

The iterative hard thresholding algorithm (IHT) is a powerful and versatile algorithm for compressed sensing and other sparse inverse problems. The standard IHT implementation faces several challenges when applied to practical problems. The step-size and sparsity parameters have to be chosen appropriately and, as IHT is based on a gradient descend strategy, convergence is only linear. Whilst the choice of the step-size can be done adaptively as suggested previously, this letter studies the use of acceleration methods to improve convergence speed. Based on recent suggestions in the literature, we show that a host of acceleration methods are also applicable to IHT. Importantly, we show that these modifications not only significantly increase the observed speed of the method, but also satisfy the same strong performance guarantees enjoyed by the original IHT method.

Iterative Hard Thresholding Algorithm Based on Backtracking

The backtracking-based iterative hard thresholding (BIHT) algorithm is proposed to solve the problem that the number of iterations is too large and the iteration time is too long when the iterative hard thresholding (IHT) algorithm is applied to the compressive sensing.The BIHT algorithm optimizes the sub-optimal choice of supports for each iteration and reduces the times of some supports iterated repeatedly by adding the idea of backtracking.The simulation demonstrates that backtracking-based algorithm ensures the reconstruction quality and decreases the time by two orders of magnitude when compared with IHT and Normalized iterative hard thresholding (NHT) algorithms for low noise level.Simulation on the 0-1 sparse signal demonstrates that the reconstruction probability of BIHT algorithm is higher than that of the IHT algorithm if the measurement times and sparsity of the signal are the same.

Hard Thresholding Pursuit: An Algorithm for Compressive Sensing

We introduce a new iterative algorithm to find sparse solutions of underdetermined linear systems. The algorithm, a simple combination of the Iterative Hard Thresholding algorithm and the Compressive Sampling Matching Pursuit algorithm, is called Hard Thresholding Pursuit. We study its general convergence and notice in particular that only a finite number of iterations are required. We then show that, under a certain condition on the restricted isometry constant of the matrix of the linear system, the Hard Thresholding Pursuit algorithm indeed finds all s-sparse solutions. This condition, which reads $\delta_{3 s} < 1/\sqrt{3}$, is heuristically better than the sufficient conditions currently available for other compressive sensing algorithms. It applies to fast versions of the algorithm, too, including the Iterative Hard Thresholding algorithm. Stability with respect to sparsity defect and robustness with respect to measurement error are also guaranteed under the condition $\delta_{3 s} < 1/\sqrt{3}$. We conclude with some numerical experiments to demonstrate the good empirical performance and the low complexity of the Hard Thresholding Pursuit algorithm.

Iterative hard-threshold algorithm with momentum

The iterative hard-threshold algorithm has recently been proposed to handle sparse regularisation problems arising in compressed sensing and other sparse signal processing problems. The algorithm is attractive owing to its simplicity; however, it converges slowly. An accelerated algorithm of this iterative method is proposed based on momentum and adaptive learning rate. Experiments with the sparse signal reconstruction show the effectiveness of the proposed algorithm.

Normalized Iterative Hard Thresholding: Guaranteed Stability and Performance

Sparse signal models are used in many signal processing applications. The task of estimating the sparsest coefficient vector in these models is a combinatorial problem and efficient, often suboptimal strategies have to be used. Fortunately, under certain conditions on the model, several algorithms could be shown to efficiently calculate near-optimal solutions. In this paper, we study one of these methods, the so-called Iterative Hard Thresholding algorithm. While this method has strong theoretical performance guarantees whenever certain theoretical properties hold, empirical studies show that the algorithm's performance degrades significantly, whenever the conditions fail. What is more, in this regime, the algorithm also often fails to converge. As we are here interested in the application of the method to real world problems, in which it is not known in general, whether the theoretical conditions are satisfied or not, we suggest a simple modification that guarantees the convergence of the method, even in this regime. With this modification, empirical evidence suggests that the algorithm is faster than many other state-of-the-art approaches while showing similar performance. What is more, the modified algorithm retains theoretical performance guarantees similar to the original algorithm.

Iterative hard thresholding for compressed sensing

Compressed sensing is a technique to sample compressible signals below the Nyquist rate, whilst still allowing near optimal reconstruction of the signal. In this paper we present a theoretical analysis of the iterative hard thresholding algorithm when applied to the compressed sensing recovery problem. We show that the algorithm has the following properties (made more precise in the main text of the paper) • It gives near-optimal error guarantees. • It is robust to observation noise. • It succeeds with a minimum number of observations. • It can be used with any sampling operator for which the operator and its adjoint can be computed. • The memory requirement is linear in the problem size. • Its computational complexity per iteration is of the same order as the application of the measurement operator or its adjoint. • It requires a fixed number of iterations depending only on the logarithm of a form of signal to noise ratio of the signal. • Its performance guarantees are uniform in that they only depend on properties of the sampling operator and signal sparsity.