Iterative Soft Thresholding Algorithm Research Articles

Imputation and low-rank estimation with Missing Not At Random data

Missing values challenge data analysis because many supervised and unsupervised learning methods cannot be applied directly to incomplete data. Matrix completion based on low-rank assumptions are very powerful solution for dealing with missing values. However, existing methods do not consider the case of informative missing values which are widely encountered in practice. This paper proposes matrix completion methods to recover Missing Not At Random (MNAR) data. Our first contribution is to suggest a model-based estimation strategy by modelling the missing mechanism distribution. An EM algorithm is then implemented, involving a Fast Iterative Soft-Thresholding Algorithm (FISTA). Our second contribution is to suggest a computationally efficient surrogate estimation by implicitly taking into account the joint distribution of the data and the missing mechanism: the data matrix is concatenated with the mask coding for the missing values; a low-rank structure for exponential family is assumed on this new matrix, in order to encode links between variables and missing mechanisms. The methodology that has the great advantage of handling different missing value mechanisms is robust to model specification errors. The performances of our methods are assessed on the real data collected from a trauma registry (TraumaBase $$^{\textregistered }$$ ) containing clinical information about over twenty thousand severely traumatized patients in France. The aim is then to predict if the doctors should administrate tranexomic acid to patients with traumatic brain injury, that would limit excessive bleeding.

Towards understanding residual and dilated dense neural networks via convolutional sparse coding.

Convolutional neural network (CNN) and its variants have led to many state-of-the-art results in various fields. However, a clear theoretical understanding of such networks is still lacking. Recently, a multilayer convolutional sparse coding (ML-CSC) model has been proposed and proved to equal such simply stacked networks (plain networks). Here, we consider the initialization, the dictionary design and the number of iterations to be factors in each layer that greatly affect the performance of the ML-CSC model. Inspired by these considerations, we propose two novel multilayer models: the residual convolutional sparse coding (Res-CSC) model and the mixed-scale dense convolutional sparse coding (MSD-CSC) model. They are closely related to the residual neural network (ResNet) and the mixed-scale (dilated) dense neural network (MSDNet), respectively. Mathematically, we derive the skip connection in the ResNet as a special case of a new forward propagation rule for the ML-CSC model. We also find a theoretical interpretation of dilated convolution and dense connection in the MSDNet by analyzing the MSD-CSC model, which gives a clear mathematical understanding of each. We implement the iterative soft thresholding algorithm and its fast version to solve the Res-CSC and MSD-CSC models. The unfolding operation can be employed for further improvement. Finally, extensive numerical experiments and comparison with competing methods demonstrate their effectiveness.

Debiasing of seismic reflectivity inversion using basis pursuit de-noising algorithm

Seismic reflectivity inversion using basis pursuit de-noising algorithm (BPDN) produces sparse solutions by applying l1-norm regularization. Due to the presence of the regularization term, there is always a deviation between the solution obtained by BPDN and the true one. A debiasing step of “adding back the residual” was introduced into seismic reflectivity inversion using BPDN to reduce the deviation, so that the accuracy of seismic reflectivity inversion using BPDN is improved. The debiasing step is carried out using a moderately large trade-off factor by iteratively performing the fast iterative soft thresholding algorithm (FISTA) to solve the objective function of BPDN with the input updated by adding back the residual to the initial input. The solution obtained by using FISTA with incorporation of the debiasing step of “adding back the residual” is a better approximation of the true solution than the result obtained by using FISTA a single time. In addition, the debiasing step of “adding back the residual” is not limited to the number of nonzero components in the solution and can be used in a low signal-to-noise ratio circumstance. A multichannel inversion method was adopted to recover the acoustic impedance from the inverted reflection coefficient, and it gained good results both in numerical and field examples.

Thresholding gradient methods in Hilbert spaces: support identification and linear convergence

We study theℓ1regularized least squares optimization problem in a separable Hilbert space. We show that the iterative soft-thresholding algorithm (ISTA) converges linearly, without making any assumption on the linear operator into play or on the problem. The result is obtained combining two key concepts: the notion ofextended support, a finite set containing the support, and the notion ofconditioning over finite-dimensional sets. We prove that ISTA identifies the solution extended support after a finite number of iterations, and we derive linear convergence from the conditioning property, which is always satisfied forℓ1regularized least squares problems. Our analysis extends to the entire class of thresholding gradient algorithms, for which we provide a conceptually new proof of strong convergence, as well as convergence rates.

Video Anomaly Detection with Sparse Coding Inspired Deep Neural Networks.

This paper presents an anomaly detection method that is based on a sparse coding inspired Deep Neural Networks (DNN). Specifically, in light of the success of sparse coding based anomaly detection, we propose a Temporally-coherent Sparse Coding (TSC), where a temporally-coherent term is used to preserve the similarity between two similar frames. The optimization of sparse coefficients in TSC with the Sequential Iterative Soft-Thresholding Algorithm (SIATA) is equivalent to a special stacked Recurrent Neural Networks (sRNN) architecture. Further, to reduce the computational cost in alternatively updating the dictionary and sparse coefficients in TSC optimization and to alleviate hyperparameters selection in TSC, we stack one more layer on top of the TSC-inspired sRNN to reconstruct the inputs, and arrive at an sRNN-AE. We further improve sRNN-AE in the following aspects: i) rather than using a predefined similarity measurement between two frames, we propose to learn a data-dependent similarity measurement between neighboring frames in sRNN-AE to make it more suitable for anomaly detection; ii) to reduce computational costs in the inference stage, we reduce the depth of the sRNN in sRNN-AE and, consequently, our framework achieves real-time anomaly detection; iii) to improve computational efficiency, we conduct temporal pooling over the appearance features of several consecutive frames for summarizing information temporally, then we feed appearance features and temporally summarized features into a separate sRNN-AE for more robust anomaly detection. To facilitate anomaly detection evaluation, we also build a large-scale anomaly detection dataset which is even larger than the summation of all existing datasets for anomaly detection in terms of both the volume of data and the diversity of scenes. Extensive experiments on both a toy dataset under controlled settings and real datasets demonstrate that our method significantly outperforms existing methods, which validates the effectiveness of our sRNN-AE method for anomaly detection. Codes and data have been released at https://github.com/StevenLiuWen/sRNN_TSC_Anomaly_Detection.

Directional tensor product complex tight framelets for compressed sensing MRI reconstruction

Compressed sensing magnetic resonance imaging (CS-MRI) is an effective way of reducing the sampling data in the k-space and shortening the scanning time. Motivated by the high performance of directional tensor product complex tight framelets (TPCTFs) for the image denoising problem, the authors proposed a novel framework that integrated TPCTF for sparse representation and projected fast iterative soft-thresholding algorithm (pFISTA) for CS-MRI reconstruction. Furthermore, to take advantage of the cross-scale relations in the wavelet tree of frame coefficients, the bivariate shrinkage (BS) function with local variance estimation is proposed to shrink thresholding. Such TPCTFs can provide sparse directional representations very well for MR image. When compared with other the state-of-the-art CS-MRI algorithms in numerical experiments, the proposed TPCTF-BS method achieves a higher reconstruction quality with respect to image edge preservation and the artefact suppression.

Positive‐contrast susceptibility imaging based on first‐order primal‐dual optimization

To achieve faster reconstruction and better imaging quality of positive-contrast MRI based on the susceptibility mapping by incorporating a primal-dual (PD) formulation. The susceptibility-based positive contrast MR technique was applied to estimate arbitrary magnetic susceptibility distributions of the metallic devices using a kernel deconvolution algorithm with a regularized minimization. The regularized positive-contrast inversion problem and its PD formulation were derived. The visualization of the positive contrast and convergence behavior of the PD algorithm were compared with those of the nonlinear conjugate gradient algorithm, fast iterative soft-thresholding algorithm, and alternating direction method of multipliers. These methods were tested and validated on computer simulations and phantom experiments. The PD approach could provide a faster reconstruction time compared with other methods. Experimental results showed that the PD algorithm could achieve comparable or even better visualization and accuracy of the metallic interventional devices in positive-contrast imaging with different SNRs and orientations to the B0 field. A susceptibility-based positive-contrast imaging technique by PD algorithm was proposed. The PD approach has more superior performance than other algorithms in terms of reconstruction time and accuracy for imaging the metallic interventional devices.

A novel iterative soft thresholding algorithm for L1 regularization based SAR image enhancement

A novel iterative soft thresholding algorithm for L1 regularization based SAR image enhancement

MM ALGORITHMS FOR VARIANCE COMPONENT ESTIMATION AND SELECTION IN LOGISTIC LINEAR MIXED MODEL.

Logistic linear mixed models are widely used in experimental designs and genetic analyses of binary traits. Motivated by modern applications, we consider the case of many groups of random effects, where each group corresponds to a variance component. When the number of variance components is large, fitting a logistic linear mixed model is challenging. Thus, we develop two efficient and stable minorization-maximization (MM) algorithms for estimating variance components based on a Laplace approximation of the logistic model. One of these leads to a simple iterative soft-thresholding algorithm for variance component selection using the maximum penalized approximated likelihood. We demonstrate the variance component estimation and selection performance of our algorithms by means of simulation studies and an analysis of real data.

Efficient Dynamic Parallel MRI Reconstruction for the Low-Rank Plus Sparse Model.

The low-rank plus sparse (L+S) decomposition model enables the reconstruction of under-sampled dynamic parallel magnetic resonance imaging (MRI) data. Solving for the low-rank and the sparse components involves non-smooth composite convex optimization, and algorithms for this problem can be categorized into proximal gradient methods and variable splitting methods. This paper investigates new efficient algorithms for both schemes. While current proximal gradient techniques for the L+S model involve the classical iterative soft thresholding algorithm (ISTA), this paper considers two accelerated alternatives, one based on the fast iterative shrinkage-thresholding algorithm (FISTA), and the other with the recent proximal optimized gradient method (POGM). In the augmented Lagrangian (AL) framework, we propose an efficient variable splitting scheme based on the form of the data acquisition operator, leading to simpler computation than the conjugate gradient (CG) approach required by existing AL methods. Numerical results suggest faster convergence of the efficient implementations for both frameworks, with POGM providing the fastest convergence overall and the practical benefit of being free of algorithm tuning parameters.

Nonstationary acoustic-impedance inversion algorithm via a novel equivalent Q-value estimation scheme and sparse regularizations

Nonstationary seismograms can be mathematically expressed by the nonstationary convolution model (NSCM), which integrates the equivalent [Formula: see text]-value [Formula: see text], source wavelet, and acoustic impedance AI into a single formula. Performing nonstationary AI inversion based on NSCM requires subsurface [Formula: see text] field distribution and a source wavelet, which are all challenging to perform. The constant [Formula: see text] field and the extracted wavelet obtained from the early part of the seismograms are commonly used as alternatives; consequently, inaccurate AI inversion results will be easily yielded. To deal with this problem, we decompose nonstationary seismic inversion into three subproblems. First, [Formula: see text] curves are estimated at well locations using a novel scheme. Because the Gabor transform can induce approximate factorization of NSCM, the time-varying wavelet amplitude spectrum (TVWAS) can be obtained by dividing the time-frequency amplitude spectrum (TFAS) of the seismic signal by the TFAS of the reflection coefficient. Stable [Formula: see text] curves at well locations then can be estimated by the compensation-based [Formula: see text]-analysis method, whose input data are TVWAS. Second, having an estimation of [Formula: see text] and AI logging information at well locations, the source-wavelet inversion objective function with wavelet transform-domain sparse constraint is proposed based on the time-domain NSCM. This inverse problem is solved efficiently using projected fast iterative soft-thresholding algorithm. We have developed the AI inversion objective function with the low-frequency constraint and isotropic total variation transform-domain sparse constraint based on frequency-domain NSCM and the split Bregman method for minimizing this objective function. Benchmark model synthetic tests and field-data application indicate the effectiveness of our nonstationary AI inversion algorithm by demonstrating the improved accuracy and resolution of the AI inversion results.

Block tensor train decomposition for missing data estimation

We propose a method for imputation of missing values in large scale matrix data based on a low-rank tensor approximation technique called the block tensor train (BTT) decomposition. Given sparsely observed data points, the proposed method iteratively computes the singular value decomposition (SVD) of the underlying data matrix with missing values. The SVD of the matrices is performed based on a low-rank BTT decomposition, by which storage and time complexities can be reduced dramatically for large-scale data matrices admitting a low-rank tensor structure. An iterative soft-thresholding algorithm is implemented for missing data estimation based on an alternating least squares method for BTT decomposition. Experimental results on simulated data and real benchmark data demonstrate that the proposed method can estimate a large amount of missing values accurately compared to a matrix-based standard method. The R source code of the BTT-based imputation method is available at https://github.com/namgillee/BTTSoftImpute .

Super-resolution least-squares prestack Kirchhoff depth migration using the L0-norm

Least-squares migration (LSM) is applied to image subsurface structures and lithology by minimizing the objective function of the observed seismic and reverse-time migration residual data of various underground reflectivity models. LSM reduces the migration artifacts, enhances the spatial resolution of the migrated images, and yields a more accurate subsurface reflectivity distribution than that of standard migration. The introduction of regularization constraints effectively improves the stability of the least-squares offset. The commonly used regularization terms are based on the L2-norm, which smooths the migration results, e.g., by smearing the reflectivities, while providing stability. However, in exploration geophysics, reflection structures based on velocity and density are generally observed to be discontinuous in depth, illustrating sparse reflectance. To obtain a sparse migration profile, we propose the super-resolution least-squares Kirchhoff prestack depth migration by solving the L0-norm-constrained optimization problem. Additionally, we introduce a two-stage iterative soft and hard thresholding algorithm to retrieve the super-resolution reflectivity distribution. Further, the proposed algorithm is applied to complex synthetic data. Furthermore, the sensitivity of the proposed algorithm to noise and the dominant frequency of the source wavelet was evaluated. Finally, we conclude that the proposed method improves the spatial resolution and achieves impulse-like reflectivity distribution and can be applied to structural interpretations and complex subsurface imaging.

Compressively sampled MR image reconstruction using generalized thresholding iterative algorithm

Compressed sensing (CS) is an emerging area of interest in Magnetic Resonance Imaging (MRI). CS is used for the reconstruction of the images from a very limited number of samples in k-space. This significantly reduces the MRI data acquisition time. One important requirement for signal recovery in CS is the use of an appropriate non-linear reconstruction algorithm. It is a challenging task to choose a reconstruction algorithm that would accurately reconstruct the MR images from the under-sampled k-space data. Various algorithms have been used to solve the system of non-linear equations for better image quality and reconstruction speed in CS. In the recent past, iterative soft thresholding algorithm (ISTA) has been introduced in CS-MRI. This algorithm directly cancels the incoherent artifacts produced because of the undersampling in k-space. This paper introduces an improved iterative algorithm based on p-thresholding technique for CS-MRI image reconstruction. The use of p-thresholding function promotes sparsity in the image which is a key factor for CS based image reconstruction. The p-thresholding based iterative algorithm is a modification of ISTA, and minimizes non-convex functions. It has been shown that the proposed p-thresholding iterative algorithm can be used effectively to recover fully sampled image from the under-sampled data in MRI. The performance of the proposed method is verified using simulated and actual MRI data taken at St. Mary’s Hospital, London. The quality of the reconstructed images is measured in terms of peak signal-to-noise ratio (PSNR), artifact power (AP), and structural similarity index measure (SSIM). The proposed approach shows improved performance when compared to other iterative algorithms based on log thresholding, soft thresholding and hard thresholding techniques at different reduction factors.

Enhancement of dynamic myocardial perfusion PET images based on low-rank plus sparse decomposition

Background and objective: The absolute quantification of dynamic myocardial perfusion (MP) PET imaging is challenged by the limited spatial resolution of individual frame images due to division of the data into shorter frames. This study aims to develop a method for restoration and enhancement of dynamic PET images.Methods: We propose that the image restoration model should be based on multiple constraints rather than a single constraint, given the fact that the image characteristic is hardly described by a single constraint alone. At the same time, it may be possible, but not optimal, to regularize the image with multiple constraints simultaneously. Fortunately, MP PET images can be decomposed into a superposition of background vs. dynamic components via low-rank plus sparse (L + S) decomposition. Thus, we propose an L + S decomposition based MP PET image restoration model and express it as a convex optimization problem. An iterative soft thresholding algorithm was developed to solve the problem. Using realistic dynamic 82Rb MP PET scan data, we optimized and compared its performance with other restoration methods.Results: The proposed method resulted in substantial visual as well as quantitative accuracy improvements in terms of noise versus bias performance, as demonstrated in extensive 82Rb MP PET simulations. In particular, the myocardium defect in the MP PET images had improved visual as well as contrast versus noise tradeoff. The proposed algorithm was also applied on an 8-min clinical cardiac 82Rb MP PET study performed on the GE Discovery PET/CT, and demonstrated improved quantitative accuracy (CNR and SNR) compared to other algorithms.Conclusions: The proposed method is effective for restoration and enhancement of dynamic PET images.

A new linear convergence result for the iterative soft thresholding algorithm

The iterative soft thresholding algorithm (ISTA) is one of the most popular optimization algorithms for solving the regularized least squares problem, and its linear convergence has been investigated under the assumption of finite basis injectivity property or strict sparsity pattern. In this paper, we consider the regularized least squares problem in finite- or infinite-dimensional Hilbert space, introduce a weaker notion of orthogonal sparsity pattern (OSP) and establish the Q-linear convergence of ISTA under the assumption of OSP. Examples are provided to illustrate the cases where the linear convergence of ISTA can be established only by our result, but cannot be ensured by any existing result in the literature.

Linearized dynamic warping with ℓ1-norm constraint for multi-component registration

Registration of the travel time of PP waves and PS waves on the same temporal coordinate is critical for joint interpretation of multi-component seismic data. We propose an ℓ1-norm minimization approach to register the corresponding amplitude of reflection events in the PP waves and PS waves data. The dynamic warping function is linearized in the ℓ2-norm fitting term, and a sparsity constrained is used to estimate smooth warping functions. The latter is possible by defining the warping function in terms of a cosine basis. Sparsity is used to minimize the number of basis functions that are used to model the warping function. We have also applied a fast iterative soft thresholding algorithm for solving the non-linear registration optimization problem. We apply the linearized dynamic warping approach with ℓ1-norm regularization to a synthetic example and to a field data set from the Shengli Oilfield, China.

Sparsity-enforcing regularisation and ISTA revisited

About two decades ago, the concept of sparsity emerged in different disciplines such as statistics, imaging, signal processing and inverse problems, and proved to be useful for several applications. Sparsity-enforcing constraints or penalties were then shown to provide a viable alternative to the usual quadratic ones for the regularisation of ill-posed problems. To compute the corresponding regularised solutions, a simple, iterative and provably convergent algorithm was proposed and later on referred to as the iterative soft-thresholding algorithm. This paper provides a brief review of these early results as well as that of the subsequent literature, albeit from the authors’ limited perspective. It also presents the previously unpublished proof of an extension of the original framework.

Projected Iterative Soft-Thresholding Algorithm for Tight Frames in Compressed Sensing Magnetic Resonance Imaging.

Compressed sensing (CS) has exhibited great potential for accelerating magnetic resonance imaging (MRI). In CS-MRI, we want to reconstruct a high-quality image from very few samples in a short time. In this paper, we propose a fast algorithm, called projected iterative soft-thresholding algorithm (pISTA), and its acceleration pFISTA for CS-MRI image reconstruction. The proposed algorithms exploit sparsity of the magnetic resonance (MR) images under the redundant representation of tight frames. We prove that pISTA and pFISTA converge to a minimizer of a convex function with a balanced tight frame sparsity formulation. The pFISTA introduces only one adjustable parameter, the step size, and we provide an explicit rule to set this parameter. Numerical experiment results demonstrate that pFISTA leads to faster convergence speeds than the state-of-art counterpart does, while achieving comparable reconstruction errors. Moreover, reconstruction errors incurred by pFISTA appear insensitive to the step size.

Conjugate gradient acceleration of iteratively re-weighted least squares methods

Iteratively re-weighted least squares (IRLS) is a method for solving minimization problems involving non-quadratic cost functions, perhaps non-convex and non-smooth, which however can be described as the infimum over a family of quadratic functions. This transformation suggests an algorithmic scheme that solves a sequence of quadratic problems to be tackled efficiently by tools of numerical linear algebra. Its general scope and its usually simple implementation, transforming the initial non-convex and non-smooth minimization problem into a more familiar and easily solvable quadratic optimization problem, make it a versatile algorithm. However, despite its simplicity, versatility, and elegant analysis, the complexity of IRLS strongly depends on the way the solution of the successive quadratic optimizations is addressed. For the important special case of compressed sensing and sparse recovery problems in signal processing, we investigate theoretically and numerically how accurately one needs to solve the quadratic problems by means of the conjugate gradient (CG) method in each iteration in order to guarantee convergence. The use of the CG method may significantly speed-up the numerical solution of the quadratic subproblems, in particular, when fast matrix-vector multiplication (exploiting for instance the FFT) is available for the matrix involved. In addition, we study convergence rates. Our modified IRLS method outperforms state of the art first order methods such as Iterative Hard Thresholding (IHT) or Fast Iterative Soft-Thresholding Algorithm (FISTA) in many situations, especially in large dimensions. Moreover, IRLS is often able to recover sparse vectors from fewer measurements than required for IHT and FISTA.