Group Sparsity Regularization Research Articles

Flexible Krylov methods for group sparsity regularization

Abstract This paper introduces new solvers for efficiently computing solutions to large-scale inverse problems with group sparsity regularization, including both non-overlapping and overlapping groups. Group sparsity regularization refers to a type of structured sparsity regularization, where the goal is to impose additional structure in the regularization process by assigning variables to predefined groups that may represent graph or network structures. Special cases of group sparsity regularization include ℓ 1 and isotropic total variation regularization. In this work, we develop hybrid projection methods based on flexible Krylov subspaces, where we first recast the group sparsity regularization term as a sequence of 2-norm penalization terms using adaptive regularization matrices in an iterative reweighted norm fashion. Then we exploit flexible preconditioning techniques to efficiently incorporate the weight updates. The main advantages of these methods are that they are computationally efficient (leveraging the advantages of flexible methods), they are general (and therefore very easily adaptable to new regularization term choices), and they are able to select the regularization parameters automatically and adaptively (exploiting the advantages of hybrid methods). Extensions to multiple regularization terms and solution decomposition frameworks (e.g. for anomaly detection) are described, and a variety of numerical examples demonstrate both the efficiency and accuracy of the proposed approaches compared to existing solvers.

Overlapping Group Sparse Regularization Technique for Vibration-Based Regional Damage Detection of Structures

In structural damage detection (SDD) studies, regularization techniques have shown potential for improving ill-posedness. However, existing methods based on regularization techniques cannot yield satisfactory results for the SDD problem involving regional damages. Based on the sparse distribution characteristics of regional damages in the finite element (FE) model, this study proposes a novel SDD method that integrates sensitivity analysis of modal parameters with overlapping group sparse regularization. First, the relationship between modal parameters and structural damages is established based on sensitivity analysis. Then, the overlapping group sparsity of regional damages is analyzed and the objective function for SDD can be defined. Finally, a grouping matrix is introduced to transform the objective function from overlapping to non-overlapping group sparse regularization. To evaluate the effectiveness of the proposed method, both numerical simulations and experimental studies are employed, and comparative studies are conducted with the SDD method based on the [Formula: see text]-norm regularization. The identification results indicate that the proposed method can handle regional damages in the FE model and identify both single and multiple damages well.

Robust hyperspectral unmixing using weighted group sparsity and minimum volume regularization

ABSTRACT Hyperspectral unmixing (HU) has been a vital technique in hyperspectral image processing. Over the past years, many algorithms have been developed based on the nonnegative matrix factorization (NMF) framework. However, the conventional methods based on the NMF need prior information of the number of endmembers, which is usually estimated by a separate procedure. Therefore, the precision of estimating the number of endmembers will impact the subsequent unmixing effect. Thus, collaborating between the two processes is desired to avoid the propagation of errors. This paper proposes a new NMF-based method, which can estimate the number of endmembers and extract the endmembers and the abundance maps. Firstly, we introduce the weighted group sparsity regularization to adaptively eliminate the abundances corresponding to the redundant endmembers, while the abundances of the true endmembers are preserved. Then the number of endmembers can be determined by the reconstruction error function and the l 2 -norm of the abundance vectors. Secondly, since the simplex determined by the endmembers has the minimum volume among all simplexes that circumscribe the data points, we impose the total variational regularization to force the endmembers to be close to each other to minimize the simplex volume. And because the total variational regularization avoids the use of noisy pixels, it can decrease the influence of noise and improve the robustness of the solution. For the proposed model, we design an effective algorithm to solve it. Also, the experimental results based on the synthetic data and real data demonstrate that the proposed method can identify the endmembers number well and achieve robust unmixing results.

Hyperspectral-multispectral image fusion using subspace decomposition and Elastic Net Regularization

ABSTRACT The fusion of hyperspectral and multispectral images presents a challenge as it involves blending a low-resolution hyperspectral image (HSI) with a corresponding multispectral image (MSI) to produce a high-resolution hyperspectral image (HRI). A number of existing techniques have limitations; for instance, matrix decomposition-based approaches fail to retain adequate spatial and spectral image information during fusion, while tensor decomposition-based processes have high computational overhead. In this paper, we propose a novel method for fusing hyperspectral and multispectral images. Our method leverages the strong correlation among the spectral bands of hyperspectral images and employs the SVD technique to extract spectral feature subspaces. This approach results in a more compact and representative feature space for fusion. Secondly, the proposed method utilizes Elastic Net regularization in combination with L 1 and L 2 regularization for effective feature selection of highly covariate features. Weighted group sparse regularization is employed to enhance the fusion effect, enabling better representation of the image’s structure and features. The algorithm is subsequently evaluated on multiple datasets to confirm its effectiveness. The results of the experiment suggest that the suggested algorithm greatly enhances the spatial resolution and visual clarity of HSI images while preserving the spectral characteristics when compared to conventional methods for fusion of hyperspectral images. Additionally, the constraint for regulation can competently reduce any noise or artefacts, thereby boosting image discrimination. To summarize, the utilization of a sparse tensor-based hyperspectral image fusion algorithm with subspace learning provides an efficacious approach for processing hyperspectral imagery. This method is capable of improving spatial resolution, extracting advantageous features from hyperspectral imagery, and ultimately supports the process of remote sensing image analysis and application.

A proximal-gradient method for problems with overlapping group-sparse regularization: support identification complexity

We consider the proximal-gradient method for minimizing the sum of a smooth function and a convex non-smooth overlapping group- ℓ 1 regularizer, which is known to promote sparse solutions with respect to its groups. A feature that distinguishes our work from most in the literature is that the proximal operator for the overlapping group- ℓ 1 function does not admit a closed-form solution, which introduces challenges when proving convergence of the iterates and especially when proving that the iterates correctly identify the group-sparse structure of the optimal solution. To address these challenges, we present an implementable termination condition for the proximal-gradient algorithm, and a specialized primal-dual subproblem solver that is designed to ensure that a group-sparse structure identification property holds. In particular, we give an upper bound on the maximum number of iterations before the correct support (i.e. group-sparse structure) of an optimal solution is identified.

Broad Multitask Learning System With Group Sparse Regularization.

The broad learning system (BLS) featuring lightweight, incremental extension, and strong generalization capabilities has been successful in its applications. Despite these advantages, BLS struggles in multitask learning (MTL) scenarios with its limited ability to simultaneously unravel multiple complex tasks where existing BLS models cannot adequately capture and leverage essential information across tasks, decreasing their effectiveness and efficacy in MTL scenarios. To address these limitations, we proposed an innovative MTL framework explicitly designed for BLS, named group sparse regularization for broad multitask learning system using related task-wise (BMtLS-RG). This framework combines a task-related BLS learning mechanism with a group sparse optimization strategy, significantly boosting BLS's ability to generalize in MTL environments. The task-related learning component harnesses task correlations to enable shared learning and optimize parameters efficiently. Meanwhile, the group sparse optimization approach helps minimize the effects of irrelevant or noisy data, thus enhancing the robustness and stability of BLS in navigating complex learning scenarios. To address the varied requirements of MTL challenges, we presented two additional variants of BMtLS-RG: BMtLS-RG with sharing parameters of feature mapped nodes (BMtLS-RGf), which integrates a shared feature mapping layer, and BMtLS-RGf and enhanced nodes (BMtLS-RGfe), which further includes an enhanced node layer atop the shared feature mapping structure. These adaptations provide customized solutions tailored to the diverse landscape of MTL problems. We compared BMtLS-RG with state-of-the-art (SOTA) MTL and BLS algorithms through comprehensive experimental evaluation across multiple practical MTL and UCI datasets. BMtLS-RG outperformed SOTA methods in 97.81% of classification tasks and achieved optimal performance in 96.00% of regression tasks, demonstrating its superior accuracy and robustness. Furthermore, BMtLS-RG exhibited satisfactory training efficiency, outperforming existing MTL algorithms by 8.04-42.85 times.

Sparse smooth group [formula omitted] regularization method for convolutional neural networks

Deep convolutional neural networks (CNNs) have successfully addressed numerous challenging real-world problems and are known to be more complex and powerful classifiers. However, finding a simple CNN architecture to properly estimate an unknown solution is still a challenging problem. With the objective of regularizing networks at the group level instead of individual ones, we propose the sparse smooth Group L0∘L1/2 regularization method, which is formulated as a composition of L0 and L1/2 regularizers. In doing so, both the L0 and L1/2 regularization terms are nonsmooth, which leads to difficulties in the training process. To overcome this issue, we propose a novel smooth function approximating the L1/2 regularizer. Additionally, we use an existing smooth indicator function to approximate the L0 regularizer. Numerical simulation results on a range of benchmark datasets and various CNN architectures demonstrate the effectiveness of the proposed method. It outperforms well-known group sparse regularization algorithms in several aspects, particularly in achieving superior group and weight sparsity.

Expectile trace regression via low-rank and group sparsity regularization

Trace regression has received a lot of attention due to its ability to account for matrix-type covariates, including panel data, images, and genomic microarrays as special cases. However, most of its existing research focuses on the case of mean regression. In this paper, we consider the expectile trace regression, which can provide a more diversified picture of the regression relationship at different expectiles, via the low-rank and group sparsity regularization. The upper bound for the statistical rate of convergence of the regularized estimator is established under some mild conditions. Some simulations, as well as a real data example, are also provided to illustrate the finite sample performance of the developed expectile trace regression.

Hyperspectral Image Super-Resolution via Adaptive Factor Group Sparsity Regularization-Based Subspace Representation

Hyperspectral image (HSI) super-resolution is a vital technique that generates high spatial-resolution HSI (HR-HSI) by integrating information from low spatial-resolution HSI with high spatial-resolution multispectral image (MSI). However, existing subspace representation-based methods face challenges, including adaptive subspace dimension determination, inadequate spectral correlation capture, and expensive computation. In this paper, we propose a novel factor group sparsity regularized subspace representation (FGSSR)-based method for HSI super-resolution that can simultaneously address these issues encountered in previous methods. Specifically, by incorporating the factor group sparsity regularization into the subspace representation model, we first propose an FGSSR model to capture the spectral correlation property of the HR-HSI. The key advantage of FGSSR lies in its equivalence to the Schatten-p norm and its adaptive determination of the accurate subspace dimension, enabling it to capture spectral correlation more effectively. To preserve the spatial self-similarity prior in the HR-HSI, the tensor nuclear norm regularization on the low-dimensional coefficients is also incorporated into the proposed FGSSR-based model. Finally, an effective proximal alternating minimization-based algorithm is developed to solve the FGSSR-based model. Experimental results on the simulated and real datasets demonstrate that the proposed FGSSR-based method outperforms several state-of-the-art fusion methods with significant improvements.

Weighted Group Sparse Regularized Tensor Decomposition for Hyperspectral Image Denoising

Hyperspectral imaging (HSI) has been used in a wide range of applications in recent years. But in the process of image acquisition, hyperspectral images are subject to various types of noise interference. Noise reduction algorithms can be used to enhance the quality of images and make it easier to detect and analyze features of interest. To realize better image recovery, we propose a weighted group sparsity-regularized low-rank tensor ring decomposition (LRTRDGS) method for hyperspectral image recovery. Tensor ring decomposition can be utilized by this approach to investigate self-similarity and global spectral correlation. Furthermore, weighted group sparsity regularization can be employed to depict the sparsity structure of the group along the spectral dimension of the spatial difference image. Moreover, we solve the proposed model using a symmetric alternating direction method multiplier with the addition of a proximity term. The experimental data verify the effectiveness of our proposed method.

Analytical HDR prostate brachytherapy planning with automatic catheter and isotope selection.

High dose rate (HDR) brachytherapy is commonly used to treat prostate cancer. Existing HDR planning systems solve the dwell time problem for predetermined catheters and a single energy source. Additional degrees of freedom can be obtained by relaxing the catheters' pre-designation and introducing more source types, and may have a dosimetric benefit, particularly in improving conformality to spare the urethra. This study presents a novel analytical approach to solving the corresponding HDR planning problem. The catheter and dual-energy source selection problem was formulated as a constrained optimization problem with a non-convex group sparsity regularization. The optimization problem was solved using the fast-iterative shrinkage-thresholding algorithm (FISTA). Two isotopes were considered. The dose rates for the HDR 4140 Ytterbium (Yb-169) source and the Elekta Iridium (Ir-192) HDR Flexisource were modeled according to the TG-43U1 formalism and benchmarked accordingly. Twenty-two retrospective HDR prostate brachytherapy patients treated with Ir-192 were considered. An Ir-192 only (IRO), Yb-169 only (YBO), and dual-source (DS) plan with optimized catheter location was created for each patient with N catheters, where N is the number of catheters used in the clinically delivered plans. The DS plans jointly optimized Yb-169 and Ir-192 dwell times. All plans and the clinical plans were normalized to deliver a 15Gy prescription (Rx) dose to 95% of the clinical treatment volume (CTV) and evaluated for the CTV D90%, V150%, and V200%, urethra D0.1cc and D1cc, bladder V75%, and rectum V75%. Dose-volume histograms (DVHs) were generated for each structure. The DS plans ubiquitously selected Ir-192 as the only treatment source. IRO outperformed YBO in organ at risk (OARs) OAR sparing, reducing the urethra D0.1cc and D1cc by 0.98% ( ) and 1.09% ( ) of the Rx dose, respectively, and reducing the bladder and rectum V75% by 0.09 ( ) and 0.13 cubic centimeters (cc) ( ), respectively. The YBO plans delivered a more homogenous dose to the CTV, with a smaller V150% and V200% by 3.20 ( ) and 1.91 cc ( ), respectively, and a lower CTV D90% by 0.49% ( ) of the prescription dose. The IRO plans reduce the urethral D1cc by 2.82% ( ) of the Rx dose compared to the clinical plans, at the cost of increased bladder and rectal V75% by 0.57 ( ) and 0.21 cc ( ), respectively, and increased CTV V150% by a mean of 1.46 cc ( ) and CTV D90% by an average of 1.40% of the Rx dose ( ). While these differences are statistically significant, the clinical differences between the plans are minimal. The proposed analytical HDR planning algorithm integrates catheter and isotope selection with dwell time optimization for varying clinical goals, including urethra sparing. The planning method can guide HDR implants and identify promising isotopes for specific HDR clinical goals, such as target conformality or OAR sparing.

Image Restoration with Fractional-Order Total Variation Regularization and Group Sparsity

In this paper, we present a novel image denoising algorithm, specifically designed to effectively restore both the edges and texture of images. This is achieved through the use of an innovative model known as the overlapping group sparse fractional-order total variation regularization model (OGS-FOTVR). The OGS-FOTVR model ingeniously combines the benefits of the fractional-order (FO) variation domain with an overlapping group sparsity measure, which acts as its regularization component. This is further enhanced by the inclusion of the well-established L2-norm, which serves as the fidelity term. To simplify the model, we employ the alternating direction method of multipliers (ADMM), which breaks down the model into a series of more manageable sub-problems. Each of these sub-problems can then be addressed individually. However, the sub-problem involving the overlapping group sparse FO regularization presents a high level of complexity. To address this, we construct an alternative function for this sub-problem, utilizing the mean inequality principle. Subsequently, we employ the majorize-minimization (MM) algorithm to solve it. Empirical results strongly support the effectiveness of the OGS-FOTVR model, demonstrating its ability to accurately recover texture and edge information in images. Notably, the model performs better than several advanced variational alternatives, as indicated by superior performance metrics across three image datasets, PSNR, and SSIM.

Fast Regularized Discrete Optimal Transport with Group-Sparse Regularizers

Regularized discrete optimal transport (OT) is a powerful tool to measure the distance between two discrete distributions that have been constructed from data samples on two different domains. While it has a wide range of applications in machine learning, in some cases the sampled data from only one of the domains will have class labels such as unsupervised domain adaptation. In this kind of problem setting, a group-sparse regularizer is frequently leveraged as a regularization term to handle class labels. In particular, it can preserve the label structure on the data samples by corresponding the data samples with the same class label to one group-sparse regularization term. As a result, we can measure the distance while utilizing label information by solving the regularized optimization problem with gradient-based algorithms. However, the gradient computation is expensive when the number of classes or data samples is large because the number of regularization terms and their respective sizes also turn out to be large. This paper proposes fast discrete OT with group-sparse regularizers. Our method is based on two ideas. The first is to safely skip the computations of the gradients that must be zero. The second is to efficiently extract the gradients that are expected to be nonzero. Our method is guaranteed to return the same value of the objective function as that of the original approach. Experiments demonstrate that our method is up to 8.6 times faster than the original method without degrading accuracy.

Beam angle optimization for proton therapy via group-sparsity based angle generation method.

In treatment planning, beam angle optimization (BAO) refers to the selection of a subset with a given number of beam angles from all available angles that provides the best plan quality. BAO is a NP-hard combinatorial problem. Although exhaustive search (ES) can exactly solve BAO by exploring all possible combinations, ES is very time-consuming and practically infeasible. To the best of our knowledge, (1) no optimization method has been demonstrated that can provide the exact solution to BAO, and (2) no study has validated an optimization method for solving BAO by benchmarking with the optimal BAO solution (e.g., via ES), both of which will be addressed by this work. This work considers BAO for proton therapy, for example, the selection of 2-4 beam angles for IMPT. The optimal BAO solution is obtained via ES and serves as the ground truth. A new BAO algorithm, namely angle generation (AG) method, is proposed, and demonstrated to provide nearly-exact solutions for BAO in reference to the ES solution. AG iteratively optimizes the angular set via group-sparsity (GS) regularization, until the planning objective does not decrease further. Since GS alone can also solve BAO, AG was validated and compared with GS for 2-angle brain, 3-angle lung, and 4-angle brain cases, in reference to the optimal BAO solutions obtained by ES: the AG solution had the rank (1/276, 1/2024, 4/10626), while the GS solution had the rank (42/276, 279/2024, 4328/10626). A new BAO algorithm called AG is proposed and shown to provide substantially improved accuracy for BAO from current methods with nearly-exact solutions to BAO, in reference to the ground truth of optimal BAO solution via ES.

Remote sensing image destriping based on variable weight and group sparse regularization

Striping effects are common phenomena in remote sensing images, and they significantly limit subsequent applications. Although many destriping approaches have been developed, there aren’t many that can completely eliminate complex stripes with varying levels of strength. To address this issue, we propose a stripe removal model based on a variable weight coefficient and group sparse regularization. Specifically, rather than a single scalar for the stripes in most approaches, different weights are set for different stripe rows to estimate the stripes with varying intensities. An adaptive method to estimate the weight matrix is proposed. On the other hand, group sparsity regularization is employed to constrain the entire stripe. In addition, region weights are designed for regions with different stripe characteristics. The alternating direction multiplier method is employed to solve the proposed model by alternating minimization. Experimental results based on simulation and real data demonstrate that the proposed model outperforms other advanced methods in terms of stripe noise removal and image detail preservation.

Diagnosis of Alzheimer’s disease using hypergraph [formula omitted]-Laplacian regularized multi-task feature learning

Multimodal data-based classification methods have been widely used in the diagnosis of Alzheimer’s disease (AD) and have achieved better performance than single-modal-based methods. However, most classification methods based on multimodal data tend to consider only the correlation between different modal data and ignore the inherent non-linear higher-order relationships between similar data, which can improve the robustness of the model. Therefore, this study proposes a hypergraph p-Laplacian regularized multi-task feature selection (HpMTFS) method for AD classification. Specifically, feature selection for each modal data is considered as a distinct task and the common features of multimodal data are extracted jointly by group-sparsity regularizer. In particular, two regularization terms are introduced in this study, namely (1) a hypergraph p-Laplacian regularization term to retain higher-order structural information for similar data, and (2) a Frobenius norm regularization term to improve the noise immunity of the model. Finally, using a multi-kernel support vector machine to fuse multimodal features and perform the final classification. We used baseline sMRI, FDG-PET, and AV-45 PET imaging data from 528 subjects in the Alzheimer’s Disease Neuroimaging Initiative (ADNI) to evaluate our approach. Experimental results show that our HpMTFS method outperforms existing multimodal-based classification methods.

Tensorial Multiview Representation for Saliency Detection via Nonconvex Approach.

In the study of salient object detection, multiview features play an important role in identifying various underlying salient objects. As to current common patch-based methods, all different features are handled directly by stacking them into a high-dimensional vector to represent related image patches. These approaches ignore the correlations inhering in the original spatial structure, which may lead to the loss of certain underlying characterization such as view interaction. In this article, different from currently available approaches, a tensorial feature representation framework is developed for the salient object detection in order to better explore the complementary information of multiview features. Under the tensor framework, a tensor low-rank constraint is applied to the background to capture its intrinsic structure, a tensor group sparsity regularization is posed on the salient part, and a tensorial sliced Laplacian regularization is then introduced to enlarge the gap between the subspaces of the background and salient object. Moreover, a nonconvex tensor Log-determinant function, instead of the tensor nuclear norm, is adopted to approximate the tensor rank for effectively suppressing the confusing information resulted from underlying complex backgrounds. Further, we have deduced the closed-form solution of this nonconvex minimization problem and established a feasible algorithm whose convergence is mathematically proven. Experiments on five well-known public datasets are provided and the simulations demonstrate that our method outperforms the latest unsupervised handcrafted features-based methods in the literature. Furthermore, our model is flexible with various deep features and is competitive with the state-of-the-art approaches.

Fabric defect detection based on low-rank decomposition with factor group-sparse regularizer

Recently, many low-rank-based methods in terms of detecting defects in fabric images have been proposed. However, there are two disadvantages of these methods. First, current low-rank based methods use the nuclear norm as the surrogate of rank, which causes inefficient optimization process and sub-optimal performance. Second, low-rank defective regions cannot be detected by low-rank based models. Thus, we propose a factor group-sparse regularized low-rank decomposition model (FGSRLRD) to solve these problems. This method takes the factor group-sparse regularizer as the surrogate of rank, which is more efficient as singular value decomposition (SVD) is not applied in the optimization process. Better performance is achieved as the factor group-sparse regularizer is a more accurate approximation of the rank. In addition, the weight matrix generated by the lightweight autoencoder is incorporated into the object function of FGSRLRD to guide locating defective regions. Besides, as low-rank defective regions cannot be segmented by low-rank models, this method constructs a fusion image of the prior image and the sparse image to highlight the defective regions. The performance of the proposed method is evaluated on two standard datasets, and the results indicate that the suggested method outperforms the existing state-of-the-art methods in locating the defective regions on fabric images.

Cellular Binary Neural Network for Accurate Image Classification and Semantic Segmentation

This paper presents the Cellular Binary Neural Network (CBNN), which is an efficient deep neural network with binary weights and activations. To address the challenge of performance drop caused by low-precision representation, the CBNN adopts multiple subnets which are connected via learnable global lateral paths. The introduced lateral connections are assumed to be sparse and grouped with respect to different source layers. The inter-network lateral connections and inner-network parameters are simultaneously optimized by the distributional loss, classification loss and the group sparse regularization term. Experiments on the CIFAR-10 and ImageNet datasets showed that, by incorporating optimized group-sparse lateral paths, the CBNN outperformed many state-of-the-art binary neural networks in terms of classification accuracy. Besides, to verify the generalization of the proposed binary model, we extended the CBNN on semantic segmentation task. CBNN takes advantage of the multiple subnets to derive the more informative feature maps which are computed by the parallel aggregation in the last convolution block. Experiments on PASCAL VOC segmentation dataset demonstrated that, under the same segmentation settings, the proposed method achieved the superior performance over other compared networks and even the full-precision counterpart.

Hyperspectral Image Denoising via Robust Subspace Estimation and Group Sparsity Constraint

Hyperspectral cameras capture electromagnetic information within hundreds of narrow spectral bands, producing hyperspectral images (HSIs) with the capability to accurately characterize the attribute information of objects. However, mixed noise induced by instrument and atmospheric effects hinders the interpretations and applications of the HSIs. In this paper, we propose a novel subspace representation based mixed noise removal method for hyperspectral images via Robust Subspace Estimation and weighted Group Sparsity constraint (RoSEGS). An outlier detection method is proposed to effectively detect sparse noise and replace the sparse noise with new estimates. A subspace estimation strategy, which is robust to mixed noise, is proposed. The subspace is first estimated after sparse noise detection and then optimized iteratively. In addition to the introduction of a state-of-the-art denoiser based on the plug-and-play technique to exploit self-similarity characteristics of the eigen-images, we impose a weighted group sparse regularization on the eigen-images to better promote the group sparsity of the spatial differences between the eigen-images, which further improves the denoising performance. We performed extensive experiments on two simulated and two real HSIs to fully demonstrate the effectiveness of the proposed method in comparison with seven state-of-the-art competitors.