Iterative Hard Thresholding Method Research Articles

Cardinality-Constrained Multi-objective Optimization: Novel Optimality Conditions and Algorithms

In this paper, we consider multi-objective optimization problems with a sparsity constraint on the vector of variables. For this class of problems, inspired by the homonymous necessary optimality condition for sparse single-objective optimization, we define the concept of L-stationarity and we analyze its relationships with other existing conditions and Pareto optimality concepts. We then propose two novel algorithmic approaches: the first one is an iterative hard thresholding method aiming to find a single L-stationary solution, while the second one is a two-stage algorithm designed to construct an approximation of the whole Pareto front. Both methods are characterized by theoretical properties of convergence to points satisfying necessary conditions for Pareto optimality. Moreover, we report numerical results establishing the practical effectiveness of the proposed methodologies.

Adaptive iterative hard thresholding for low-rank matrix recovery and rank-one measurements

In low-rank matrix recovery, many kinds of measurements fail to meet the standard restricted isometry property (RIP), such as rank-one measurements, that is, [A(X)]i=〈Ai,X〉 with rank(Ai)=1, i=1,...,m. Historical iterative hard thresholding sequence for low-rank matrix recovery and rank-one measurements was taken as Xn+1=Ps(Xn−μnPt(A⁎sign(A(Xn)−y))), which introduced the “tail” and “head” approximations Ps and Pt, respectively. In this paper, we remove the term Pt and provide a new iterative hard thresholding algorithm with adaptive step size (abbreviated as AIHT). The linear convergence analysis and stability results on AIHT are established under the ℓ1/ℓ2-RIP. Particularly, we discuss the rank-one Gaussian measurements under the tight upper and lower bounds on E‖A(X)‖1, and provide better convergence rate and sampling complexity. Besides, several empirical experiments are provided to show that AIHT performs better than the historical rank-one iterative hard thresholding method.

Riemannian thresholding methods for row-sparse and low-rank matrix recovery

In this paper, we present modifications of the iterative hard thresholding (IHT) method for recovery of jointly row-sparse and low-rank matrices. In particular, a Riemannian version of IHT is considered which significantly reduces computational cost of the gradient projection in the case of rank-one measurement operators, which have concrete applications in blind deconvolution. Experimental results are reported that show near-optimal recovery for Gaussian and rank-one measurements, and that adaptive stepsizes give crucial improvement. A Riemannian proximal gradient method is derived for the special case of unknown sparsity.

Stochastic privacy-preserving methods for nonconvex sparse learning

Sparse learning is essential in mining high-dimensional data. Iterative hard thresholding (IHT) methods are effective for optimizing nonconvex objectives for sparse learning. However, IHT methods are vulnerable to adversary attacks that infer sensitive data. Although pioneering works attempted to relieve such vulnerability, they confront the issue of high computational cost for large-scale problems. We propose two differentially private stochastic IHT: one based on the stochastic gradient descent method (DP-SGD-HT) and the other based on the stochastically controlled stochastic gradient method (DP-SCSG-HT). The DP-SGD-HT method perturbs stochastic gradients with small Gaussian noise rather than full gradients, which are computationally expensive. As a result, computational complexity is reduced from O(nlog(n)) to a lower O(blog(n)), where n is the sample size and b is the mini-batch size used to compute stochastic gradients. The DP-SCSG-HT method further perturbs the stochastic gradients controlled by large-batch snapshot gradients to reduce stochastic gradient variance. We prove that both algorithms guarantee differential privacy and have linear convergence rates with estimation bias. A utility analysis examines the relationship between convergence rate and the level of perturbation, yielding the best-known utility bound for nonconvex sparse optimization. Extensive experiments show that our algorithms outperform existing methods.

Federated Optimization of ℓ0-norm Regularized Sparse Learning

Regularized sparse learning with the ℓ0-norm is important in many areas, including statistical learning and signal processing. Iterative hard thresholding (IHT) methods are the state-of-the-art for nonconvex-constrained sparse learning due to their capability of recovering true support and scalability with large datasets. The current theoretical analysis of IHT assumes the use of centralized IID data. In realistic large-scale scenarios, however, data are distributed, seldom IID, and private to edge computing devices at the local level. Consequently, it is required to study the property of IHT in a federated environment, where local devices update the sparse model individually and communicate with a central server for aggregation infrequently without sharing local data. In this paper, we propose the first group of federated IHT methods: Federated Hard Thresholding (Fed-HT) and Federated Iterative Hard Thresholding (FedIter-HT) with theoretical guarantees. We prove that both algorithms have a linear convergence rate and guarantee for recovering the optimal sparse estimator, which is comparable to classic IHT methods, but with decentralized, non-IID, and unbalanced data. Empirical results demonstrate that the Fed-HT and FedIter-HT outperform their competitor—a distributed IHT, in terms of reducing objective values with fewer communication rounds and bandwidth requirements.

Iterative hard thresholding for low CP-rank tensor models

Recovery of low-rank matrices from a small number of linear measurements is now well-known to be possible under various model assumptions on the measurements. Such results demonstrate robustness and are backed with provable theoretical guarantees. However, extensions to tensor recovery have only recently began to be studied and developed, despite an abundance of practical tensor applications. Recently, a tensor variant of the Iterative Hard Thresholding method was proposed and theoretical results were obtained that exact guarantee recovery of tensors with low Tucker rank. In this paper, we utilize and prove a similar tensor version of the Restricted Isometry Property (RIP) to extend these results for tensors with low CANDECOMP/PARAFAC (CP) rank. In doing so, we leverage recent results on efficient approximations of CP decompositions that remove the need for challenging assumptions in prior works. We complement our theoretical findings with empirical results that showcase the potential of the approach.

A fast Hankel matrix nonconvex factorization reconstruction method with inertia momentum for non-uniformly sampled NMR spectroscopy

Multidimensional nuclear magnetic resonance (NMR) spectroscopy is one of the most powerful tools for qualitative or quantitative analysis of the composition and structure of various organic and inorganic substances. However, the time required to acquire NMR signals increases exponentially with dimensionality. Therefore, non-uniform sampling is commonly adopted to accelerate data acquisition and then the complete spectrum can be obtained by reconstruction method. At present, the state-of-the-art reconstruction methods are based on the idea of low-rank Hankel matrix completion and solved by different singular value thresholding methods. However, the computation of singular value decomposition (SVD) is very time-consuming, especially for high-resolution spectra. In this paper, we proposed a Hankel matrix nonconvex factorization optimization model to avoid SVD, thus greatly reducing the computational time. We developed a numerical algorithm based on the linearized Alternating direction method of multipliers to solve the proposed optimization problem, and Nesterov momentum is adopted to accelerate the convergence of the algorithm. However, for nonconvex model, the high quality reconstruction can only be obtained when the number of the spectral peaks is known. Thus we designed an adaptive strategy to alleviate this problem. Experiments are performed to demonstrate that the proposed algorithm can obtain higher quality reconstruction in less computational time than low rank Hankel matrix factorization and fast iterative hard thresholding methods.

Estimation and Mitigation of Time-Variant RFI Based on Iterative Dual Sparse Recovery in Ultra-Wide Band Through-Wall Radar

This paper focuses on the time-variant radio frequency interference (RFI) issue that ultra-wide band (UWB) through-wall radar (TWR) is faced with, and presents an iterative dual sparse recovery (IDSR) framework to combat it. The framework consists of two stages: 1) RFI estimation and detection and 2) IDSR of scattered echoes from objects and RFI signals. In the first stage, an overlapped short time Fourier transform is employed to construct and update the discrete frequency Doppler spectrum (DFDS). Then, a minimum statistic operation is conducted on the DFDS to estimate RFI signals, followed by detection via a 1-D cell-averaging constant false alarm rate detector to determinate whether RFI signals exist or not. In the second stage, a dual sparse model of the collected signals is set up, based on the fast-time frequency sparsity of RFI signals because of their narrow bands and the Doppler frequency sparsity of scattered echoes from objects because of their limited moving velocities. The alternating direction method of multipliers (ADMM) is introduced to iteratively and alternately recover RFI signals and scattered echoes from objects. Specifically, the iterative hard thresholding (IHT) method is used to complete the two sparse recovery operations. The involved two sparse dictionaries are simple and only made up of inverse discrete Fourier transform basis and independent of the received signals. The IDSR framework is able to relieve the influences of RFI signals and scattered echoes from objects on each other, and thus can reconstruct the two type signals to the maximum limit. Field experiments using a UWB TWR were carried out to verify the proposed method.

A New Proximal Iterative Hard Thresholding Method with Extrapolation for $$\ell _0$$ ℓ 0 Minimization

In this paper, we consider a non-convex problem which is the sum of $$\ell _0$$ -norm and a convex smooth function under a box constraint. We propose one proximal iterative hard thresholding type method with an extrapolation step for acceleration and establish its global convergence results. In detail, the sequence generated by the proposed method globally converges to a local minimizer of the objective function. Finally, we conduct numerical experiments to show the proposed method’s effectiveness on comparison with some other efficient methods.

Comparison of the projection onto convex sets and iterative hard thresholding methods for seismic data interpolation and denoising

Because of the environment limitations, irregularity appears in the observed seismic data. In addition, the observed seismic data contains random noise from the acquisition equipment and surrounding environment, which affects the performances of multi-channel techniques, such as surface related multiple elimination (SRME) and amplitude variation with offset (AVO) analysis. The projection onto convex sets (POCS) method, known as an efficient interpolation method, is suitable for high signal-to-noise ratio (SNR) situations; however, the existing random noise may affect its final performance. In our previously published paper, the POCS formula was deduced in the view of iterative hard thresholding (IHT) method using a projection operator. In this paper, more physical illustrations about its detailed deduction are provided to show the differences between IHT and POCS in noise-free and noisy situations with easy understanding for readers. Then, performances of the POCS and IHT methods are compared in both noise-free and noisy situations, in terms of seismograms, frequency wavenumber (FK) spectra and single traces. For noise-free data, both the POCS and IHT methods can achieve good interpolation results. For noisy data, the POCS method is unsuitable because of the observed noisy data insertion, while the IHT performance is satisfactory because it uses a thresholding operator to eliminate random noise. Numerical examples on noise-free datasets demonstrate the validities of the POCS and IHT methods for interpolation. Tests on noisy data contaminated with additive white Gaussian noise prove the ability and superiority of the IHT method with anti-noise property compared with the POCS method.

Vandermonde Factorization of Hankel Matrix for Complex Exponential Signal Recovery—Application in Fast NMR Spectroscopy

Many signals are modeled as a superposition of exponential functions in spectroscopy of chemistry, biology and medical imaging. This paper studies the problem of recovering exponential signals from a random subset of samples. We exploit the Vandermonde structure of the Hankel matrix formed by the exponential signal and formulate signal recovery as Hankel matrix completion with Vandermonde factorization (HVaF). A numerical algorithm is developed to solve the proposed model and its sequence convergence is analyzed theoretically. Experiments on synthetic data demonstrate that HVaF succeeds over a wider regime than the state-of-the-art nuclear-normminimization-based Hankel matrix completion method, while has a less restriction on frequency separation than the state-of-the-art atomic norm minimization and fast iterative hard thresholding methods. The effectiveness of HVaF is further validated on biological magnetic resonance spectroscopy data.

An iterative hard thresholding estimator for low rank matrix recovery with explicit limiting distribution

We consider the problem of low rank matrix recovery in a stochastically noisy high-dimensional setting. We propose a new estimator for the low rank matrix, based on the iterative hard thresholding method, that is computationally efficient and simple. We prove that our estimator is optimal in terms of the Frobenius risk and in terms of the entry-wise risk uniformly over any change of orthonormal basis, allowing us to provide the limiting distribution of the estimator. When the design is Gaussian, we prove that the entry-wise bias of the limiting distribution of the estimator is small, which is of interest for constructing tests and confidence sets for low-dimensional subsets of entries of the low rank matrix.

Lagrange Dual Method for Sparsity Constrained Optimization

In this paper, we investigate the $l_{0}$ quasi-norm constrained optimization problem in the Lagrange dual framework and show that the strong duality property holds. Motivated by the property, we propose a Lagrange dual method for the sparsity constrained optimization problem. The method adopts the bisection search technique to maximize the Lagrange dual function. For each Lagrange multiplier, we adopt the iterative hard thresholding method to minimize the Lagrange function. We show that the proposed method converges to an $L$ -stationary point of the primal problem. Computational experiments and comparisons on a number of test instances (including random compressed sensing instances and random and real sparse logistic regression instances) demonstrate the effectiveness of the proposed method in generating sparse solution accurately.

Low rank tensor recovery via iterative hard thresholding

We study extensions of compressive sensing and low rank matrix recovery (matrix completion) to the recovery of low rank tensors of higher order from a small number of linear measurements. While the theoretical understanding of low rank matrix recovery is already well-developed, only few contributions on the low rank tensor recovery problem are available so far. In this paper, we introduce versions of the iterative hard thresholding algorithm for several tensor decompositions, namely the higher order singular value decomposition (HOSVD), the tensor train format (TT), and the general hierarchical Tucker decomposition (HT). We provide a partial convergence result for these algorithms which is based on a variant of the restricted isometry property of the measurement operator adapted to the tensor decomposition at hand that induces a corresponding notion of tensor rank. We show that subgaussian measurement ensembles satisfy the tensor restricted isometry property with high probability under a certain almost optimal bound on the number of measurements which depends on the corresponding tensor format. These bounds are extended to partial Fourier maps combined with random sign flips of the tensor entries. Finally, we illustrate the performance of iterative hard thresholding methods for tensor recovery via numerical experiments where we consider recovery from Gaussian random measurements, tensor completion (recovery of missing entries), and Fourier measurements for third order tensors.

Homotopy Methods Based on $l_{0}$ -Norm for Compressed Sensing.

This paper proposes two homotopy methods for solving the compressed sensing (CS) problem, which combine the homotopy technique with the iterative hard thresholding (IHT) method. The homotopy methods overcome the difficulty of the IHT method on the choice of the regularization parameter value, by tracing solutions of the regularized problem along a homotopy path. We prove that any accumulation point of the sequences generated by the proposed homotopy methods is a feasible solution of the problem. We also show an upper bound on the sparsity level for each solution of the proposed methods. Moreover, to improve the solution quality, we modify the two methods into the corresponding heuristic algorithms. Computational experiments demonstrate effectiveness of the two heuristic algorithms, in accurately and efficiently generating sparse solutions of the CS problem, whether the observation is noisy or not.

The proximal alternating iterative hard thresholding method for [formula omitted] minimization, with complexity [formula omitted

Since digital images are usually sparse in the wavelet frame domain, some nonconvex minimization models based on wavelet frame have been proposed and sparse approximations have been widely used in image restoration in recent years. Among them, the proximal alternating iterative hard thresholding method is proposed in this paper to solve the nonconvex model based on wavelet frame. Through combining the proposed algorithm with the iterative hard thresholding algorithm which is well studied in compressed sensing theory, this paper proves that the complexity of the proposed method is O(1/k). On the other hand, a more general nonconvex–nonsmooth model is adopted and the pseudo proximal alternating linearized minimization method is developed to solve the above problem. With the Kurdyka–Łojasiewicz (KL) property, it is proved that the sequence generated by the proposed algorithm converges to some critical points of the corresponding model. Finally, the proposed method is applied to restore the blurred noisy gray images. As the numerical results reveal, the performance of the proposed method is comparable or better than some well-known convex image restoration methods.

Randomized Algorithms for Distributed Nonlinear Optimization Under Sparsity Constraints

Distributed optimization in multi-agent systems under sparsity constraints has recently received a lot of attention. In this paper, we consider the in-network minimization of a continuously differentiable nonlinear function which is a combination of local agent objective functions subject to sparsity constraints on the variables. A crucial issue of in-network optimization is the handling of the communications, which may be expensive. This calls for efficient algorithms, that are able to reduce the number of required communication links and transmitted messages. To this end, we focus on asynchronous and randomized distributed techniques. Based on consensus techniques and iterative hard thresholding methods, we propose three methods that attempt to minimize the given function, promoting sparsity of the solution: asynchronous hard thresholding (AHT), broadcast hard thresholding (BHT), and gossip hard thresholding (GHT). Although similar in many aspects, it is difficult to obtain a unified analysis for the proposed algorithms. Specifically, we theoretically prove the convergence and characterize the limit points of AHT in regular networks under some proper assumptions on the functions to be minimized. For BHT and GHT, instead, we characterize the fixed points of the maps that rule their dynamics in terms of stationary points of original problem. Finally, we illustrate the implementation of our techniques in compressed sensing and present several numerical results on performance and number of transmissions required for convergence.

Polynomial approximation via compressed sensing of high-dimensional functions on lower sets

This work proposes and analyzes a compressed sensing approach to polynomial approximation of complex-valued functions in high dimensions. Of particular interest is the setting where the target function is smooth, characterized by a rapidly decaying orthonormal expansion, whose most important terms are captured by a lower (or downward closed) set. By exploiting this fact, we present an innovative weighted $\ell_1$ minimization procedure with a precise choice of weights, and a new iterative hard thresholding method, for imposing the downward closed preference. Theoretical results reveal that our computational approaches possess a provably reduced sample complexity compared to existing compressed sensing techniques presented in the literature. In addition, the recovery of the corresponding best approximation using these methods is established through an improved bound for the restricted isometry property. Our analysis represents an extension of the approach for Hadamard matrices in [5] to the general case of continuous bounded orthonormal systems, quantifies the dependence of sample complexity on the successful recovery probability, and provides an estimate on the number of measurements with explicit constants. Numerical examples are provided to support the theoretical results and demonstrate the computational efficiency of the novel weighted $\ell_1$ minimization strategy.

Design of Wideband Fractional-Order Differentiator Using Interlaced Sampling Method

In this paper, the design of a wideband digital fractional-order differentiator (FOD) is investigated. First, conventional FOD designs are reviewed, and the reconstruction formula of the interlaced sampling method is used to design the proposed wideband FOD by index substitution and the Grunwald---Letnikov fractional derivative. Because a closed-form window design is obtained, the filter coefficients are easily computed. Then, the weighted least squares and convex optimization methods are applied to design non-sparse digital FODs that are optimal in the least squares or min---max sense. Next, the iterative hard thresholding and orthogonal matching pursuit methods are used to design sparse digital FODs to reduce the implementation complexity. Finally, several numerical examples are presented to show that the proposed FODs have smaller design errors in the high-frequency band than conventional digital FODs that do not use the auxiliary interlaced sampling signal.

An Accelerated Iterative Hard Thresholding Method for Matrix Completion

An Accelerated Iterative Hard Thresholding Method for Matrix Completion