Iterative Soft Thresholding Algorithm Research Articles

Thresholding gradient methods in Hilbert spaces: support identification and linear convergence

We study theℓ1regularized least squares optimization problem in a separable Hilbert space. We show that the iterative soft-thresholding algorithm (ISTA) converges linearly, without making any assumption on the linear operator into play or on the problem. The result is obtained combining two key concepts: the notion ofextended support, a finite set containing the support, and the notion ofconditioning over finite-dimensional sets. We prove that ISTA identifies the solution extended support after a finite number of iterations, and we derive linear convergence from the conditioning property, which is always satisfied forℓ1regularized least squares problems. Our analysis extends to the entire class of thresholding gradient algorithms, for which we provide a conceptually new proof of strong convergence, as well as convergence rates.

Video Anomaly Detection with Sparse Coding Inspired Deep Neural Networks.

This paper presents an anomaly detection method that is based on a sparse coding inspired Deep Neural Networks (DNN). Specifically, in light of the success of sparse coding based anomaly detection, we propose a Temporally-coherent Sparse Coding (TSC), where a temporally-coherent term is used to preserve the similarity between two similar frames. The optimization of sparse coefficients in TSC with the Sequential Iterative Soft-Thresholding Algorithm (SIATA) is equivalent to a special stacked Recurrent Neural Networks (sRNN) architecture. Further, to reduce the computational cost in alternatively updating the dictionary and sparse coefficients in TSC optimization and to alleviate hyperparameters selection in TSC, we stack one more layer on top of the TSC-inspired sRNN to reconstruct the inputs, and arrive at an sRNN-AE. We further improve sRNN-AE in the following aspects: i) rather than using a predefined similarity measurement between two frames, we propose to learn a data-dependent similarity measurement between neighboring frames in sRNN-AE to make it more suitable for anomaly detection; ii) to reduce computational costs in the inference stage, we reduce the depth of the sRNN in sRNN-AE and, consequently, our framework achieves real-time anomaly detection; iii) to improve computational efficiency, we conduct temporal pooling over the appearance features of several consecutive frames for summarizing information temporally, then we feed appearance features and temporally summarized features into a separate sRNN-AE for more robust anomaly detection. To facilitate anomaly detection evaluation, we also build a large-scale anomaly detection dataset which is even larger than the summation of all existing datasets for anomaly detection in terms of both the volume of data and the diversity of scenes. Extensive experiments on both a toy dataset under controlled settings and real datasets demonstrate that our method significantly outperforms existing methods, which validates the effectiveness of our sRNN-AE method for anomaly detection. Codes and data have been released at https://github.com/StevenLiuWen/sRNN_TSC_Anomaly_Detection.

Directional tensor product complex tight framelets for compressed sensing MRI reconstruction

Compressed sensing magnetic resonance imaging (CS-MRI) is an effective way of reducing the sampling data in the k-space and shortening the scanning time. Motivated by the high performance of directional tensor product complex tight framelets (TPCTFs) for the image denoising problem, the authors proposed a novel framework that integrated TPCTF for sparse representation and projected fast iterative soft-thresholding algorithm (pFISTA) for CS-MRI reconstruction. Furthermore, to take advantage of the cross-scale relations in the wavelet tree of frame coefficients, the bivariate shrinkage (BS) function with local variance estimation is proposed to shrink thresholding. Such TPCTFs can provide sparse directional representations very well for MR image. When compared with other the state-of-the-art CS-MRI algorithms in numerical experiments, the proposed TPCTF-BS method achieves a higher reconstruction quality with respect to image edge preservation and the artefact suppression.

Positive‐contrast susceptibility imaging based on first‐order primal‐dual optimization

To achieve faster reconstruction and better imaging quality of positive-contrast MRI based on the susceptibility mapping by incorporating a primal-dual (PD) formulation. The susceptibility-based positive contrast MR technique was applied to estimate arbitrary magnetic susceptibility distributions of the metallic devices using a kernel deconvolution algorithm with a regularized minimization. The regularized positive-contrast inversion problem and its PD formulation were derived. The visualization of the positive contrast and convergence behavior of the PD algorithm were compared with those of the nonlinear conjugate gradient algorithm, fast iterative soft-thresholding algorithm, and alternating direction method of multipliers. These methods were tested and validated on computer simulations and phantom experiments. The PD approach could provide a faster reconstruction time compared with other methods. Experimental results showed that the PD algorithm could achieve comparable or even better visualization and accuracy of the metallic interventional devices in positive-contrast imaging with different SNRs and orientations to the B0 field. A susceptibility-based positive-contrast imaging technique by PD algorithm was proposed. The PD approach has more superior performance than other algorithms in terms of reconstruction time and accuracy for imaging the metallic interventional devices.

A novel iterative soft thresholding algorithm for L1 regularization based SAR image enhancement

A novel iterative soft thresholding algorithm for L1 regularization based SAR image enhancement

MM ALGORITHMS FOR VARIANCE COMPONENT ESTIMATION AND SELECTION IN LOGISTIC LINEAR MIXED MODEL.

Logistic linear mixed models are widely used in experimental designs and genetic analyses of binary traits. Motivated by modern applications, we consider the case of many groups of random effects, where each group corresponds to a variance component. When the number of variance components is large, fitting a logistic linear mixed model is challenging. Thus, we develop two efficient and stable minorization-maximization (MM) algorithms for estimating variance components based on a Laplace approximation of the logistic model. One of these leads to a simple iterative soft-thresholding algorithm for variance component selection using the maximum penalized approximated likelihood. We demonstrate the variance component estimation and selection performance of our algorithms by means of simulation studies and an analysis of real data.

Efficient Dynamic Parallel MRI Reconstruction for the Low-Rank Plus Sparse Model.

The low-rank plus sparse (L+S) decomposition model enables the reconstruction of under-sampled dynamic parallel magnetic resonance imaging (MRI) data. Solving for the low-rank and the sparse components involves non-smooth composite convex optimization, and algorithms for this problem can be categorized into proximal gradient methods and variable splitting methods. This paper investigates new efficient algorithms for both schemes. While current proximal gradient techniques for the L+S model involve the classical iterative soft thresholding algorithm (ISTA), this paper considers two accelerated alternatives, one based on the fast iterative shrinkage-thresholding algorithm (FISTA), and the other with the recent proximal optimized gradient method (POGM). In the augmented Lagrangian (AL) framework, we propose an efficient variable splitting scheme based on the form of the data acquisition operator, leading to simpler computation than the conjugate gradient (CG) approach required by existing AL methods. Numerical results suggest faster convergence of the efficient implementations for both frameworks, with POGM providing the fastest convergence overall and the practical benefit of being free of algorithm tuning parameters.

Nonstationary acoustic-impedance inversion algorithm via a novel equivalent Q-value estimation scheme and sparse regularizations

Nonstationary seismograms can be mathematically expressed by the nonstationary convolution model (NSCM), which integrates the equivalent [Formula: see text]-value [Formula: see text], source wavelet, and acoustic impedance AI into a single formula. Performing nonstationary AI inversion based on NSCM requires subsurface [Formula: see text] field distribution and a source wavelet, which are all challenging to perform. The constant [Formula: see text] field and the extracted wavelet obtained from the early part of the seismograms are commonly used as alternatives; consequently, inaccurate AI inversion results will be easily yielded. To deal with this problem, we decompose nonstationary seismic inversion into three subproblems. First, [Formula: see text] curves are estimated at well locations using a novel scheme. Because the Gabor transform can induce approximate factorization of NSCM, the time-varying wavelet amplitude spectrum (TVWAS) can be obtained by dividing the time-frequency amplitude spectrum (TFAS) of the seismic signal by the TFAS of the reflection coefficient. Stable [Formula: see text] curves at well locations then can be estimated by the compensation-based [Formula: see text]-analysis method, whose input data are TVWAS. Second, having an estimation of [Formula: see text] and AI logging information at well locations, the source-wavelet inversion objective function with wavelet transform-domain sparse constraint is proposed based on the time-domain NSCM. This inverse problem is solved efficiently using projected fast iterative soft-thresholding algorithm. We have developed the AI inversion objective function with the low-frequency constraint and isotropic total variation transform-domain sparse constraint based on frequency-domain NSCM and the split Bregman method for minimizing this objective function. Benchmark model synthetic tests and field-data application indicate the effectiveness of our nonstationary AI inversion algorithm by demonstrating the improved accuracy and resolution of the AI inversion results.

Block tensor train decomposition for missing data estimation

We propose a method for imputation of missing values in large scale matrix data based on a low-rank tensor approximation technique called the block tensor train (BTT) decomposition. Given sparsely observed data points, the proposed method iteratively computes the singular value decomposition (SVD) of the underlying data matrix with missing values. The SVD of the matrices is performed based on a low-rank BTT decomposition, by which storage and time complexities can be reduced dramatically for large-scale data matrices admitting a low-rank tensor structure. An iterative soft-thresholding algorithm is implemented for missing data estimation based on an alternating least squares method for BTT decomposition. Experimental results on simulated data and real benchmark data demonstrate that the proposed method can estimate a large amount of missing values accurately compared to a matrix-based standard method. The R source code of the BTT-based imputation method is available at https://github.com/namgillee/BTTSoftImpute .

Super-resolution least-squares prestack Kirchhoff depth migration using the L0-norm

Least-squares migration (LSM) is applied to image subsurface structures and lithology by minimizing the objective function of the observed seismic and reverse-time migration residual data of various underground reflectivity models. LSM reduces the migration artifacts, enhances the spatial resolution of the migrated images, and yields a more accurate subsurface reflectivity distribution than that of standard migration. The introduction of regularization constraints effectively improves the stability of the least-squares offset. The commonly used regularization terms are based on the L2-norm, which smooths the migration results, e.g., by smearing the reflectivities, while providing stability. However, in exploration geophysics, reflection structures based on velocity and density are generally observed to be discontinuous in depth, illustrating sparse reflectance. To obtain a sparse migration profile, we propose the super-resolution least-squares Kirchhoff prestack depth migration by solving the L0-norm-constrained optimization problem. Additionally, we introduce a two-stage iterative soft and hard thresholding algorithm to retrieve the super-resolution reflectivity distribution. Further, the proposed algorithm is applied to complex synthetic data. Furthermore, the sensitivity of the proposed algorithm to noise and the dominant frequency of the source wavelet was evaluated. Finally, we conclude that the proposed method improves the spatial resolution and achieves impulse-like reflectivity distribution and can be applied to structural interpretations and complex subsurface imaging.

Compressively sampled MR image reconstruction using generalized thresholding iterative algorithm

Compressed sensing (CS) is an emerging area of interest in Magnetic Resonance Imaging (MRI). CS is used for the reconstruction of the images from a very limited number of samples in k-space. This significantly reduces the MRI data acquisition time. One important requirement for signal recovery in CS is the use of an appropriate non-linear reconstruction algorithm. It is a challenging task to choose a reconstruction algorithm that would accurately reconstruct the MR images from the under-sampled k-space data. Various algorithms have been used to solve the system of non-linear equations for better image quality and reconstruction speed in CS. In the recent past, iterative soft thresholding algorithm (ISTA) has been introduced in CS-MRI. This algorithm directly cancels the incoherent artifacts produced because of the undersampling in k-space. This paper introduces an improved iterative algorithm based on p-thresholding technique for CS-MRI image reconstruction. The use of p-thresholding function promotes sparsity in the image which is a key factor for CS based image reconstruction. The p-thresholding based iterative algorithm is a modification of ISTA, and minimizes non-convex functions. It has been shown that the proposed p-thresholding iterative algorithm can be used effectively to recover fully sampled image from the under-sampled data in MRI. The performance of the proposed method is verified using simulated and actual MRI data taken at St. Mary’s Hospital, London. The quality of the reconstructed images is measured in terms of peak signal-to-noise ratio (PSNR), artifact power (AP), and structural similarity index measure (SSIM). The proposed approach shows improved performance when compared to other iterative algorithms based on log thresholding, soft thresholding and hard thresholding techniques at different reduction factors.

Enhancement of dynamic myocardial perfusion PET images based on low-rank plus sparse decomposition

Background and objective: The absolute quantification of dynamic myocardial perfusion (MP) PET imaging is challenged by the limited spatial resolution of individual frame images due to division of the data into shorter frames. This study aims to develop a method for restoration and enhancement of dynamic PET images.Methods: We propose that the image restoration model should be based on multiple constraints rather than a single constraint, given the fact that the image characteristic is hardly described by a single constraint alone. At the same time, it may be possible, but not optimal, to regularize the image with multiple constraints simultaneously. Fortunately, MP PET images can be decomposed into a superposition of background vs. dynamic components via low-rank plus sparse (L + S) decomposition. Thus, we propose an L + S decomposition based MP PET image restoration model and express it as a convex optimization problem. An iterative soft thresholding algorithm was developed to solve the problem. Using realistic dynamic 82Rb MP PET scan data, we optimized and compared its performance with other restoration methods.Results: The proposed method resulted in substantial visual as well as quantitative accuracy improvements in terms of noise versus bias performance, as demonstrated in extensive 82Rb MP PET simulations. In particular, the myocardium defect in the MP PET images had improved visual as well as contrast versus noise tradeoff. The proposed algorithm was also applied on an 8-min clinical cardiac 82Rb MP PET study performed on the GE Discovery PET/CT, and demonstrated improved quantitative accuracy (CNR and SNR) compared to other algorithms.Conclusions: The proposed method is effective for restoration and enhancement of dynamic PET images.

A new linear convergence result for the iterative soft thresholding algorithm

The iterative soft thresholding algorithm (ISTA) is one of the most popular optimization algorithms for solving the regularized least squares problem, and its linear convergence has been investigated under the assumption of finite basis injectivity property or strict sparsity pattern. In this paper, we consider the regularized least squares problem in finite- or infinite-dimensional Hilbert space, introduce a weaker notion of orthogonal sparsity pattern (OSP) and establish the Q-linear convergence of ISTA under the assumption of OSP. Examples are provided to illustrate the cases where the linear convergence of ISTA can be established only by our result, but cannot be ensured by any existing result in the literature.

Linearized dynamic warping with ℓ1-norm constraint for multi-component registration

Registration of the travel time of PP waves and PS waves on the same temporal coordinate is critical for joint interpretation of multi-component seismic data. We propose an ℓ1-norm minimization approach to register the corresponding amplitude of reflection events in the PP waves and PS waves data. The dynamic warping function is linearized in the ℓ2-norm fitting term, and a sparsity constrained is used to estimate smooth warping functions. The latter is possible by defining the warping function in terms of a cosine basis. Sparsity is used to minimize the number of basis functions that are used to model the warping function. We have also applied a fast iterative soft thresholding algorithm for solving the non-linear registration optimization problem. We apply the linearized dynamic warping approach with ℓ1-norm regularization to a synthetic example and to a field data set from the Shengli Oilfield, China.

Sparsity-enforcing regularisation and ISTA revisited

About two decades ago, the concept of sparsity emerged in different disciplines such as statistics, imaging, signal processing and inverse problems, and proved to be useful for several applications. Sparsity-enforcing constraints or penalties were then shown to provide a viable alternative to the usual quadratic ones for the regularisation of ill-posed problems. To compute the corresponding regularised solutions, a simple, iterative and provably convergent algorithm was proposed and later on referred to as the iterative soft-thresholding algorithm. This paper provides a brief review of these early results as well as that of the subsequent literature, albeit from the authors’ limited perspective. It also presents the previously unpublished proof of an extension of the original framework.

Projected Iterative Soft-Thresholding Algorithm for Tight Frames in Compressed Sensing Magnetic Resonance Imaging.

Compressed sensing (CS) has exhibited great potential for accelerating magnetic resonance imaging (MRI). In CS-MRI, we want to reconstruct a high-quality image from very few samples in a short time. In this paper, we propose a fast algorithm, called projected iterative soft-thresholding algorithm (pISTA), and its acceleration pFISTA for CS-MRI image reconstruction. The proposed algorithms exploit sparsity of the magnetic resonance (MR) images under the redundant representation of tight frames. We prove that pISTA and pFISTA converge to a minimizer of a convex function with a balanced tight frame sparsity formulation. The pFISTA introduces only one adjustable parameter, the step size, and we provide an explicit rule to set this parameter. Numerical experiment results demonstrate that pFISTA leads to faster convergence speeds than the state-of-art counterpart does, while achieving comparable reconstruction errors. Moreover, reconstruction errors incurred by pFISTA appear insensitive to the step size.

Conjugate gradient acceleration of iteratively re-weighted least squares methods

Iteratively re-weighted least squares (IRLS) is a method for solving minimization problems involving non-quadratic cost functions, perhaps non-convex and non-smooth, which however can be described as the infimum over a family of quadratic functions. This transformation suggests an algorithmic scheme that solves a sequence of quadratic problems to be tackled efficiently by tools of numerical linear algebra. Its general scope and its usually simple implementation, transforming the initial non-convex and non-smooth minimization problem into a more familiar and easily solvable quadratic optimization problem, make it a versatile algorithm. However, despite its simplicity, versatility, and elegant analysis, the complexity of IRLS strongly depends on the way the solution of the successive quadratic optimizations is addressed. For the important special case of compressed sensing and sparse recovery problems in signal processing, we investigate theoretically and numerically how accurately one needs to solve the quadratic problems by means of the conjugate gradient (CG) method in each iteration in order to guarantee convergence. The use of the CG method may significantly speed-up the numerical solution of the quadratic subproblems, in particular, when fast matrix-vector multiplication (exploiting for instance the FFT) is available for the matrix involved. In addition, we study convergence rates. Our modified IRLS method outperforms state of the art first order methods such as Iterative Hard Thresholding (IHT) or Fast Iterative Soft-Thresholding Algorithm (FISTA) in many situations, especially in large dimensions. Moreover, IRLS is often able to recover sparse vectors from fewer measurements than required for IHT and FISTA.

Compressed Sensing MRI Using Discrete Nonseparable Shearlet Transform and FISTA

We propose a new compressed sensing MRI approach that uses the discrete nonseparable shearlet transform (DNST) as a sparsifying transform and the fast iterative soft thresholding algorithm (FISTA) for reconstruction. FISTA has a simple design and has shown good convergence behavior. The DNST transform has excellent localization properties within the space domain and excellent directional selectivity. We utilize the frequency representation of the DNST canonical dual filters to obtain a memory efficient modified FISTA based algorithm with a simple and efficient way of calculating the update, tuned to the non tight frame DNST transform. The proposed approach shows improved performance and similar execution time when compared with other state of the art reconstruction approaches.

Discrete and Continuous-Time Soft-Thresholding for Dynamic Signal Recovery

There exist many well-established techniques to recover sparse signals from compressed measurements with known performance guarantees in the static case. More recently, new methods have been proposed to tackle the recovery of time-varying signals, but few benefit from a theoretical analysis. In this paper, we give theoretical guarantees for the Iterative Soft-Thresholding Algorithm (ISTA) and its continuous-time analogue the Locally Competitive Algorithm (LCA) to perform this tracking in real time. ISTA is a well-known digital solver for static sparse recovery, whose iteration is a first-order discretization of the LCA differential equation. Our analysis is based on the Restricted Isometry Property (RIP) and shows that the outputs of both algorithms can track a time-varying signal while compressed measurements are streaming, even when no convergence criterion is imposed at each time step. The l2-distance between the target signal and the outputs of both discrete- and continuous-time solvers is shown to decay to a bound that is essentially optimal. Our analysis is supported by simulations on both synthetic and real data.

An algorithm for quadratic ℓ1-regularized optimization with a flexible active-set strategy

We present an active-set method for minimizing an objective that is the sum of a convex quadratic and regularization term. Unlike two-phase methods that combine a first-order active set identification step and a subspace phase consisting of a cycle of conjugate gradient (CG) iterations, the method presented here has the flexibility of computing a first-order proximal gradient step or a subspace CG step at each iteration. The decision of which type of step to perform is based on the relative magnitudes of some scaled components of the minimum norm subgradient of the objective function. The paper establishes global rates of convergence, as well as work complexity estimates for two variants of our approach, which we call the interleaved iterative soft-thresholding algorithm (ISTA)–CG method. Numerical results illustrating the behaviour of the method on a variety of test problems are presented.