Iterative Thresholding Research Articles

On convergence of iterative thresholding algorithms to approximate sparse solution for composite nonconvex optimization

AbstractThis paper aims to find an approximate true sparse solution of an underdetermined linear system. For this purpose, we propose two types of iterative thresholding algorithms with the continuation technique and the truncation technique respectively. We introduce a notion of limited shrinkage thresholding operator and apply it, together with the restricted isometry property, to show that the proposed algorithms converge to an approximate true sparse solution within a tolerance relevant to the noise level and the limited shrinkage magnitude. Applying the obtained results to nonconvex regularization problems with SCAD, MCP and $$\ell _p$$ ℓ p penalty ($$0\le p \le 1$$ 0 ≤ p ≤ 1 ) and utilizing the recovery bound theory, we establish the convergence of their proximal gradient algorithms to an approximate global solution of nonconvex regularization problems. The established results include the existing convergence theory for $$\ell _1$$ ℓ 1 or $$\ell _0$$ ℓ 0 regularization problems for finding a true sparse solution as special cases. Preliminary numerical results show that our proposed algorithms can find approximate true sparse solutions that are much better than stationary solutions that are found by using the standard proximal gradient algorithm.

Cardinality-Constrained Multi-objective Optimization: Novel Optimality Conditions and Algorithms

In this paper, we consider multi-objective optimization problems with a sparsity constraint on the vector of variables. For this class of problems, inspired by the homonymous necessary optimality condition for sparse single-objective optimization, we define the concept of L-stationarity and we analyze its relationships with other existing conditions and Pareto optimality concepts. We then propose two novel algorithmic approaches: the first one is an iterative hard thresholding method aiming to find a single L-stationary solution, while the second one is a two-stage algorithm designed to construct an approximation of the whole Pareto front. Both methods are characterized by theoretical properties of convergence to points satisfying necessary conditions for Pareto optimality. Moreover, we report numerical results establishing the practical effectiveness of the proposed methodologies.

Efficient and Fast Joint Sparse Constrained Canonical Correlation Analysis for Fault Detection.

The canonical correlation analysis (CCA) has attracted wide attention in fault detection (FD). To improve the detection performance, we propose a new joint sparse constrained CCA (JSCCCA) model that integrates the l2,0 -norm joint sparse constraints into classical CCA. The key idea is that JSCCCA can fully exploit the joint sparse structure to determine the number of extracted variables. We then develop an efficient alternating minimization algorithm using the improved iterative hard thresholding and manifold constrained gradient descent method. More importantly, we establish the convergence guarantee with detailed analysis. Finally, we provide extensive numerical studies on the simulated dataset, the benchmark Tennessee Eastman process, and a practical cylinder-piston process. In some cases, the computing time is reduced by 600 times, and the FD rate is increased by 12.62% compared with classical CCA. The results suggest that the proposed approach is efficient and fast.

Impact load identification of high-speed train scale carbody based on multi-objective optimisation and TwIST sparse regularisation

ABSTRACT Effective identification of dynamic loads on vehicle systems is crucial for evaluating the structural integrity of railway vehicles. The inverse problem method has been proposed for dynamic load identification to determine the impact load on the carbody. Due to the sparse nature of the impact load, the combination of the l 1-norm and TwIST (two-step iterative threshold shrinkage method) methods is used to reconstruct the impact load, taking into account the influence of sensor placement on the identification results. The MOHBA (Multi-objective Optimisation Honey Badger Algorithm) was proposed to simultaneously select the regularisation and iteration parameters. This algorithm aims to improve the accuracy of subsequence load reconstruction by considering the influence of noise levels when selecting the measured data. The impact load experiment was conducted on the scale carbody, and the single-point impact load and multi-point impact load matched well with the actual load. Satisfactory load identification results can still be achieved when the noise level is within 15%. When arranging measurement points on the boundary of the vehicle body, satisfactory load identification results can also be achieved.

A Systematic Evaluation of Recurrent Neural Network Models for Edge Intelligence and Human Activity Recognition Applications

The Recurrent Neural Networks (RNNs) are an essential class of supervised learning algorithms. Complex tasks like speech recognition, machine translation, sentiment classification, weather prediction, etc., are now performed by well-trained RNNs. Local or cloud-based GPU machines are used to train them. However, inference is now shifting to miniature, mobile, IoT devices and even micro-controllers. Due to their colossal memory and computing requirements, mapping RNNs directly onto resource-constrained platforms is arcane and challenging. The efficacy of edge-intelligent RNNs (EI-RNNs) must satisfy both performance and memory-fitting requirements at the same time without compromising one for the other. This study’s aim was to provide an empirical evaluation and optimization of historic as well as recent RNN architectures for high-performance and low-memory footprint goals. We focused on Human Activity Recognition (HAR) tasks based on wearable sensor data for embedded healthcare applications. We evaluated and optimized six different recurrent units, namely Vanilla RNNs, Long Short-Term Memory (LSTM) units, Gated Recurrent Units (GRUs), Fast Gated Recurrent Neural Networks (FGRNNs), Fast Recurrent Neural Networks (FRNNs), and Unitary Gated Recurrent Neural Networks (UGRNNs) on eight publicly available time-series HAR datasets. We used the hold-out and cross-validation protocols for training the RNNs. We used low-rank parameterization, iterative hard thresholding, and spare retraining compression for RNNs. We found that efficient training (i.e., dataset handling and preprocessing procedures, hyperparameter tuning, and so on, and suitable compression methods (like low-rank parameterization and iterative pruning) are critical in optimizing RNNs for performance and memory efficiency. We implemented the inference of the optimized models on Raspberry Pi.

Sparse Identification of Fractional Chaotic Systems based on the time-domain data

The complexity inherent in defining and computing fractional derivatives renders the development of a data-driven modeling framework for fractional-order systems more formidable than for their integer-order counterparts. This study introduces a methodology for sparsely identifying fractional chaotic systems by leveraging time-domain data to ascertain the system’s governing equations. Initially, we establish a sparse identification framework specifically tailored for fractional order systems and introduce a joint iterative thresholding method designed to identify these systems’ fractional and integer order components concurrently. Moreover, this study conducts a comparative analysis of two error criteria for model selection. It introduces a strategy predicated on minimizing the distance to balance sparsity and model fitting error optimally. To validate the efficacy and precision of the proposed methodology, numerical simulations were conducted on fractional order simplified Lorenz and Chua’s circuits, affirming the robustness and accuracy of our approach.

Damage imaging in plates by evaluating local entropy in guided wavefield data

This paper presents a new algorithm called broadband entropy-based two-stage damage imaging (BE2DI) for the detection and imaging of defects in multi-layer plates.The method differentiates defects from either local scattering phenomena of propagating Lamb waves or locally increased vibrational activity for stimulated global plate resonances by exploiting the higher vibrational activity of full-field broadband vibration signals at defective regions and the distinct spatial behavior of defects and artifacts. Using these features, first, a candidate damage map is constructed by means of iterative thresholding based on descriptive statistics of broadband data. Afterwards, an adaptive regional damage index based on local entropy is introduced, which utilizes operational deflection shapes to process previously identified candidate regions and subsequently reveal both shallow and deep defects. BE2DI can be seen as an automated local defect resonance identification approach in the case of shallow defects. Multiple plates having various defect types with different sizes and depths are investigated. Broadband chirp signals are used to excite the plates using low-power piezoelectric actuators, and vibrational data is registered by 3D scanning laser Doppler vibrometry. The obtained results quantitatively and qualitatively demonstrate that the proposed method can effectively detect different kinds of damage, regardless of being shallow or deep.

Low-Cost Plant-Protection Unmanned Ground Vehicle System for Variable Weeding Using Machine Vision.

This study presents a machine vision-based variable weeding system for plant- protection unmanned ground vehicles (UGVs) to address the issues of pesticide waste and environmental pollution that are readily caused by traditional spraying agricultural machinery. The system utilizes fuzzy rules to achieve adaptive modification of the Kp, Ki, and Kd adjustment parameters of the PID control algorithm and combines them with an interleaved period PWM controller to reduce the impact of nonlinear variations in water pressure on the performance of the system, and to improve the stability and control accuracy of the system. After testing various image threshold segmentation and image graying algorithms, the normalized super green algorithm (2G-R-B) and the fast iterative threshold segmentation method were adopted as the best combination. This combination effectively distinguished between the vegetation and the background, and thus improved the accuracy of the pixel extraction algorithm for vegetation distribution. The results of orthogonal testing by selected four representative spraying duty cycles-25%, 50%, 75%, and 100%-showed that the pressure variation was less than 0.05 MPa, the average spraying error was less than 2%, and the highest error was less than 5% throughout the test. Finally, the performance of the system was comprehensively evaluated through field trials. The evaluation showed that the system was able to adjust the corresponding spraying volume in real time according to the vegetation distribution under the decision-making based on machine vision algorithms, which proved the low cost and effectiveness of the designed variable weed control system.

Deconvolution of JWST/MIRI Images: Applications to an Active Galactic Nucleus Model and GATOS Observations of NGC 5728

The superb image quality, stability, and sensitivity of JWST permit deconvolution techniques to be pursued with a fidelity unavailable to ground-based observations. We present an assessment of several deconvolution approaches to improve image quality and mitigate the effects of the complex JWST point-spread function (PSF). The optimal deconvolution method is determined by using WebbPSF to simulate JWST’s complex PSF and MIRISim to simulate multiband JWST/Mid-Infrared Imager Module (MIRIM) observations of a toy model of an active galactic nucleus (AGN). Five different deconvolution algorithms are tested: (1) Kraken deconvolution, (2) Richardson–Lucy, (3) the adaptive imaging deconvolution algorithm, (4) sparse regularization with the Condat–Vũ algorithm, and (5) iterative Wiener filtering and thresholding. We find that Kraken affords the greatest FWHM reduction of the nuclear source of our MIRISim observations for the toy AGN model while retaining good photometric integrity across all simulated wave bands. Applying Kraken to Galactic Activity, Torus, and Outflow Survey (GATOS) multiband JWST/MIRIM observations of the Seyfert 2 galaxy NGC 5728, we find that the algorithm reduces the FWHM of the nuclear source by a factor of 1.6–2.2 across all five filters. Kraken images facilitate detection of extended nuclear emission ∼2.″5 (∼470 pc, position angle ≃ 115°) in the SE–NW direction, especially at the longest wavelengths. We demonstrate that Kraken is a powerful tool to enhance faint features otherwise hidden in the complex JWST PSF.

SWISTA-Nets: Subband-adaptive wavelet iterative shrinkage thresholding networks for image reconstruction

Robust and interpretable image reconstruction is central to imageology applications in clinical practice. Prevalent deep networks, with strong learning ability to extract implicit information from data manifold, are still lack of prior knowledge introduced from mathematics or physics, leading to instability, poor structure interpretability and high computation cost. As to this issue, we propose two prior knowledge-driven networks to combine the good interpretability of mathematical methods and the powerful learnability of deep learning methods. Incorporating different kinds of prior knowledge, we propose subband-adaptive wavelet iterative shrinkage thresholding networks (SWISTA-Nets), where almost every network module is in one-to-one correspondence with each step involved in the iterative algorithm. By end-to-end training of proposed SWISTA-Nets, implicit information can be extracted from training data and guide the tuning process of key parameters that possess mathematical definition. The inverse problems associated with two medical imaging modalities, i.e., electromagnetic tomography and X-ray computational tomography are applied to validate the proposed networks. Both visual and quantitative results indicate that the SWISTA-Nets outperform mathematical methods and state-of-the-art prior knowledge-driven networks, especially with fewer training parameters, interpretable network structures and well robustness. We assume that our analysis will support further investigation of prior knowledge-driven networks in the field of ill-posed image reconstruction.

Sufficient condition based on nearly optimal order RIC for IHT algorithm

Iterative hard thresholding (IHT) is one class of sparse signal recovery algorithms widely used in compressed sensing. The well-known restricted isometry constant (RIC) is one of the most popular frameworks used to ensure the convergence of recovery algorithms. In previous literature, sufficient conditions to guarantee the recovery performance of the IHT algorithm have usually been established based on RICs with orders 3 s and 2 s , while no such condition in terms of RIC with order less than 2 s has been reported yet. In this study, based on the theoretical optimal step-length, we develop the RIC condition γ < ( 5 − 1 ) / 8 ≈ 0.1545 for successful recovery via the IHT algorithm. Here, γ = δ s if s is an even number, otherwise γ = δ s + 1 . To the best of our knowledge, this is the first result based on RIC with nearly optimal order for the IHT algorithm.

A Fast Intersection of Confidence Intervals Method-Based Adaptive Thresholding for Sparse Image Reconstruction Using the Matrix Form of the Wavelet Transform

One of the frequently used classes of sparse reconstruction algorithms is based on the iterative shrinkage/thresholding procedure, in which the thresholding parameter controls a trade-off between the algorithm’s accuracy and execution time. In order to avoid this trade-off, we propose using a fast intersection of confidence intervals method in order to adaptively control the threshold value throughout the iterations of the reconstruction algorithm. We have upgraded the two-step iterative shrinkage thresholding algorithm with a such procedure, improving its performance. The proposed algorithm, denoted as the FICI-TwIST, along with a few selected state-of-the-art sparse reconstruction algorithms, has been tested on the classical problem of image recovery by emphasizing the image sparsity in the discrete cosine and the discrete wavelet domain. Furthermore, we have derived a single wavelet transformation matrix which avoids wrapping effects, thereby achieving significantly faster execution times as compared to a more traditional function-based transformation. The obtained results indicate the competitive performance of the proposed algorithm, even in cases where all algorithm parameters have been individually fine-tuned for best performance.

MsDC-DEQ-Net: Deep Equilibrium Model (DEQ) with Multiscale Dilated Convolution for Image Compressive Sensing (CS)

Compressive sensing (CS) is a technique that enables the recovery of sparse signals using fewer measurements than traditional sampling methods. To address the computational challenges of CS reconstruction, our objective is to develop an interpretable and concise neural network model for reconstructing natural images using CS. We achieve this by mapping one step of the iterative shrinkage thresholding algorithm (ISTA) to a deep network block, representing one iteration of ISTA. To enhance learning ability and incorporate structural diversity, we integrate aggregated residual transformations (ResNeXt) and squeeze-and-excitation mechanisms into the ISTA block. This block serves as a deep equilibrium layer connected to a semi-tensor product network for convenient sampling and providing an initial reconstruction. The resulting model, called MsDC-DEQ-Net, exhibits competitive performance compared to state-of-the-art network-based methods. It significantly reduces storage requirements compared to deep unrolling methods, using only one iteration block instead of multiple iterations. Unlike deep unrolling models, MsDC-DEQ-Net can be iteratively used, gradually improving reconstruction accuracy while considering computation tradeoffs. Additionally, the model benefits from multiscale dilated convolutions, further enhancing performance.

PISTA: Preconditioned Iterative Soft Thresholding Algorithm for Graphical Lasso

pISTA: Preconditioned Iterative Soft Thresholding Algorithm for Graphical Lasso

Defect imaging and identification in asphalt materials using coplanar capacitance sensors with single-pair electrodes

Coplanar capacitance imaging technology (CCIT) is a non-destructive method with intuitive and accurate identification. This paper aims to realize defect identification in asphalt materials based on the CCIT using coplanar single-pair electrode capacitance sensor (CSCS). Firstly, the Linear Back Projection (LBP) algorithm, the Landweber algorithm, and the Kalman-Filter (KF) algorithm, are compared and evaluated. Then, the principal component analysis (PCA) method is utilized to fuse the reconstructed images. In addition, the OTSU method, the iterative threshold (IT) method, and the genetic algorithm (OA) method, is compared and utilized to quantify the defective region. Finally, this investigation analyzes the reconstructed and segmented images of defects in asphalt materials. It is concluded that the KL algorithm is the most suitable algorithm to reconstruct the defective images. The PCA method improve the quality of the reconstructed defective images. The defective region is determined by the OTSU method, which is the most approximate imaging segment method. It is found that the CCIT can detect the invisible defect depth in asphalt materials. The different defect mediums in various asphalt materials can be identified by the CCIT. The segmented defective region error in asphalt materials is less than 13%, demonstrating that the CCIT is effective for the identification of defect shape details in asphalt materials. The segmented imaging precision of square defects in asphalt materials is the highest; circular defects are the next lowest; and triangular defects are the lowest. The outcomes of this research can assist engineers in realizing intuitive and high-accuracy identification of various invisible defects in shallow asphalt layers in bridge deck asphalt pavement utilizing the CCIT.

Iterative Thresholding and Projection Algorithms and Model-Based Deep Neural Networks for Sparse LQR Control Design

Iterative Thresholding and Projection Algorithms and Model-Based Deep Neural Networks for Sparse LQR Control Design

Self-Supervised Medical Image Denoising Based on WISTA-Net for Human Healthcare in Metaverse.

Medical image processing plays an important role in the interaction of real world and metaverse for healthcare. Self-supervised denoising based on sparse coding methods, without any prerequisite on large-scale training samples, has been attracting extensive attention for medical image processing. Whereas, existing self-supervised methods suffer from poor performance and low efficiency. In this paper, to achieve state-of-the-art denoising performance on the one hand, we present a self-supervised sparse coding method, named the weighted iterative shrinkage thresholding algorithm (WISTA). It does not rely on noisy-clean ground-truth image pairs to learn from only a single noisy image. On the other hand, to further improve denoising efficiency, we unfold the WISTA to construct a deep neural network (DNN) structured WISTA, named WISTA-Net. Specifically, in WISTA, motivated by the merit of the lp-norm, WISTA-Net has better denoising performance than the classical orthogonal matching pursuit (OMP) algorithm and the ISTA. Moreover, leveraging the high-efficiency of DNN structure in parameter updating, WISTA-Net outperforms the compared methods in denoising efficiency. In detail, for a 256 by 256 noisy image, the running time of WISTA-Net is 4.72 s on the CPU, which is much faster than WISTA, OMP, and ISTA by 32.88 s, 13.06 s, and 6.17 s, respectively.

Lasso-based state estimation for cyber-physical systems under sensor attacks

The development of algorithms for secure state estimation in vulnerable cyber-physical systems has been gaining attention in the last years. A consolidated assumption is that an adversary can tamper a relatively small number of sensors. In the literature, block-sparsity methods exploit this prior information to recover the attack locations and the state of the system. In this paper, we propose an alternative, Lasso-based approach and we analyse its effectiveness. In particular, we theoretically derive conditions that guarantee successful attack/state recovery, independently of established time sparsity patterns. Furthermore, we develop a sparse state observer, by starting from the iterative soft thresholding algorithm for Lasso, to perform online estimation. Through several numerical experiments, we compare the proposed methods to the state-of-the-art algorithms.

Iterative singular tube hard thresholding algorithms for tensor recovery

Iterative singular tube hard thresholding algorithms for tensor recovery

Infrared Small Target Detection Based on Adaptive Region Growing Algorithm With Iterative Threshold Analysis

Infrared Small Target Detection Based on Adaptive Region Growing Algorithm With Iterative Threshold Analysis