Group Lasso Regularization Research Articles

Improved Selection of Pairwise Interaction Effects via Automated Calibration of Hierarchical Group-LASSO Regularization in a Stability Selection Framework.

Improved Selection of Pairwise Interaction Effects via Automated Calibration of Hierarchical Group-LASSO Regularization in a Stability Selection Framework.

Embedded feature selection approach based on TSK fuzzy system with sparse rule base for high-dimensional classification problems

In high-dimensional problems of fuzzy rule-based embedded feature selection, the challenges include loss of interpretability, curse of dimensionality, and arithmetic underflow, among others. The primary reason for these problems is the exponential increase in the number of fuzzy rules with an increase in input dimension. In this study, an embedded feature selection approach, the Takagi–Sugeno–Kang (TSK) fuzzy system with sparse rule base (TSK-SRB), is proposed for high-dimensional data. Based on clustering, a broader initial rule base is designed with a suitable number of rules. In the rule layer, refined softmin (Ref-softmin) is introduced to calculate the firing strength, which can approximate the minimum T-norm while avoiding arithmetic underflow. Two Group Lasso regularization terms are used to realize feature and rule selection. In addition, an automatic threshold segmentation is introduced to determine the appropriate number of selected features/rules. Extensive experiments on 17 classification datasets showed that TSK-SRB is effective and competitive in high-dimensional feature selection.

First-Order Sparse TSK Nonstationary Fuzzy Neural Network Based on the Mean Shift Algorithm and the Group Lasso Regularization

Nonstationary fuzzy inference systems (NFIS) are able to tackle uncertainties and avoid the difficulty of type-reduction operation. Combining NFIS and neural network, a first-order sparse TSK nonstationary fuzzy neural network (SNFNN-1) is proposed in this paper to improve the interpretability/translatability of neural networks and the self-learning ability of fuzzy rules/sets. The whole architecture of SNFNN-1 can be considered as an integrated model of multiple sub-networks with a variation in center, variation in width or variation in noise. Thus, it is able to model both “intraexpert” and “interexpert” variability. There are two techniques adopted in this network: the Mean Shift-based fuzzy partition and the Group Lasso-based rule selection, which can adaptively generate a suitable number of clusters and select important fuzzy rules, respectively. Quantitative experiments on six UCI datasets demonstrate the effectiveness and robustness of the proposed model.

A Novel Learning Approach to Remove Oscillations in First‐Order Takagi–Sugeno Fuzzy System: Gradient Descent‐Based Neuro‐Fuzzy Algorithm Using Smoothing Group Lasso Regularization

AbstractAs a universal approximator, the first order Takagi–Sugeno fuzzy system possesses the capability to approximate widespread nonlinear systems through a group of IF THEN fuzzy rules. Although group lasso regularization has the advantage of inducing group sparsity and handling variable selection issues, it can lead to numerical oscillations and theoretical challenges in calculating the gradient at the origin when employed directly during training. The paper addresses the aforementioned obstacle by invoking a smoothing function to approximate group lasso regularization. On this basis, a gradient‐based neuro fuzzy learning algorithm with smoothing group lasso regularization for the first order Takagi–Sugeno fuzzy system is proposed. The convergence of the proposed algorithm is rigorously proved under gentle conditions. In addition, experimental outcomes acquired on two approximations and two classification simulations demonstrate that the proposed algorithm outperforms the algorithm with original group lasso regularization andL2regularization in terms of error, pruned neurons, and accuracy. This is particularly evident in significant advancements in pruned neurons due to group sparsity. In comparison to the algorithm withL2regularization, the proposed algorithm exhibits improvements of 6.3, 5.3, and 142.6 in pruned neurons during sin function, Gabor function, and Sonar benchmark dataset simulations, respectively.

A smoothing Group Lasso based interval type-2 fuzzy neural network for simultaneous feature selection and system identification

Inspired by the life philosophy, an ingenious gate (membership) function, which can mimic the open and close of the gate in the real world, is proposed to realize feature selection (FS) for interval type-2 fuzzy neural network. Based on the widths of gate functions, a smoothing integrated learning strategy is designed, and in the regularized FS error function the gate widths corresponding to one feature are grouped as one Group Lasso (GL) regularization term. During the training procedure of feature selection, all the group terms with respect to bad features will be punished, but for the good features the corresponding group terms converge to different nonzero values. When feature selection is finished, using the selected features, the interpretable clustering algorithm based initial interval type-2 rule generation method is detailedly introduced firstly, then a simple and efficient memory-based gradient method is presented for the tuning of rule structure parameters. Lastly, employing the simulation results of four regression problems and four classification problems, the effectiveness on simultaneous feature selection and system identification of our proposed model, i.e., the smoothing Group Lasso based interval type-2 fuzzy neural network (SGLIT2FNN), is proved, which illustrates that SGLIT2FNN not only simplifies the model structure by deleting the bad features, but also maintains the performance.

Dual-modal Image Reconstruction for Electrical Impedance Tomography with Overlapping Group Lasso and Laplacian Regularization.

Electrical Impedance Tomography (EIT) is a promising biomedical imaging modality, yet EIT image reconstruction remains an open challenge due to its severe ill-posedness. High-quality EIT image reconstruction algorithms are desired. This paper reports a segmentation-free dual-modal EIT image reconstruction algorithm that uses Overlapping Group Lasso and Laplacian (OGLL) regularization. An overlapping group lasso penalty is constructed based on conductivity change properties and encodes the imaging targets' structural information obtained from an auxiliary imaging modality that provides structural images of the sensing region. We introduce Laplacian regularization to alleviate the artifacts caused by group overlapping. The performance of OGLL is evaluated and compared with single-modal and dual-modal image reconstruction algorithms using simulation and real-world data. Quantitative metrics and visualized images confirm the superiority of the proposed method in terms of structure preservation, background artifact (BA) suppression, and conductivity contrast differentiation. This work proves the effectiveness of OGLL in improving EIT image quality. This study demonstrates that EIT has the potential to be adopted in quantitative tissue analysis by using such dual-modal imaging approaches.

Nutritional Modulation of Associations between Prenatal Exposure to Persistent Organic Pollutants and Childhood Obesity: A Prospective Cohort Study.

Prenatal exposure to persistent organic pollutants (POPs) may contribute to the development of childhood obesity and metabolic disorders. However, little is known about whether the maternal nutritional status during pregnancy can modulate these associations. The main objective was to characterize the joint associations and interactions between prenatal levels of POPs and nutrients on childhood obesity. We used data from to the Spanish INfancia y Medio Ambiente-Environment and Childhood (INMA) birth cohort, on POPs and nutritional biomarkers measured in maternal blood collected at the first trimester of pregnancy and child anthropometric measurements at 7 years of age. Six organochlorine compounds (OCs) [dichlorodiphenyldichloroethylene, hexachlorobenzene (HCB), () and polychlorinated biphenyls 138, 153, 180] and four per- and polyfluoroalkyl substances (PFAS) were measured. Nutrients included vitamins (D, B12, and folate), polyunsaturated fatty acids (PUFAs), and dietary carotenoids. Two POPs-nutrients mixtures data sets were established: a) OCs, PFAS, vitamins, and carotenoids (), and b) OCs, PUFAs, and vitamins (). Joint associations of mixtures on obesity were characterized using Bayesian kernel machine regression (BKMR). Relative importance of biomarkers and two-way interactions were identified using gradient boosting machine, hierarchical group lasso regularization, and BKMR. Interactions were further characterized using multivariate regression models in the multiplicative and additive scale. Forty percent of children had overweight or obesity. We observed a positive overall joint association of both POPs-nutrients mixtures on overweight/obesity risk, with HCB and vitamin B12 the biomarkers contributing the most. Recurrent interactions were found between HCB and vitamin B12 across screening models. Relative risk for a natural log increase of HCB was 1.31 (95% CI: 1.11, 1.54, ) in the tertile 2 of vitamin B12 and in the additive scale a relative excess risk due to interaction of 0.11 (95% CI: 0.02, 0.20) was found. Interaction between perfluorooctane sulfonate and suggested a protective effect of the antioxidant on overweight/obesity risk. These results support that maternal nutritional status may modulate the effect of prenatal exposure to POPs on childhood overweight/obesity. These findings may help to develop a biological hypothesis for future toxicological studies and to better interpret inconsistent findings in epidemiological studies. https://doi.org/10.1289/EHP11258.

Complex Bayesian group Lasso for defect imaging with guided waves

The defect imaging based on guided wave provides an intuitive way for defect localization. Recently, sparse representation methods based on the damage sparsity assumption have been developed for defect imaging, where few sensors are used in these methods. However, these sparse imaging methods need repeatedly tuning the regularization parameter to obtain a good imaging performance. In this paper, an adaptive method based on complex Bayesian group Lasso is developed for localizing the damage. A group Lasso model is constructed to represent the defect imaging problem, and formulated by a sparse Bayesian learning (SBL) framework, where a hierarchical model of a Laplace prior is built to represent the group Lasso regularization. Estimations of the model variables are derived by using variational inference. In the proposed method, the model parameters are automatically updated without needing priori information. The effectiveness of the proposed method is verified by analyzing an experimental data.

Pooled Cohort Probability Score for Subclinical Airflow Obstruction.

Rationale: Early detection of chronic obstructive pulmonary disease (COPD) is a public health priority. Airflow obstruction is the single most important risk factor for adverse COPD outcomes, but spirometry is not routinely recommended for screening. Objectives: To describe the burden of subclinical airflow obstruction (SAO) and to develop a probability score for SAO to inform potential detection and prevention programs. Methods: Lung function and clinical data were harmonized and pooled across nine U.S. general population cohorts. Adults with respiratory symptoms, inhaler use, or prior diagnosis of COPD or asthma were excluded. A probability score for prevalent SAO (forced expiratory volume in 1 second/forced vital capacity < 0.70) was developed via hierarchical group-lasso regularization from clinical variables in strata of sex and smoking status, and its discriminative accuracy for SAO was assessed in the pooled cohort as well as in an external validation cohort (NHANES [National Health and Nutrition Examination Survey] 2011-2012). Incident hospitalizations and deaths due to COPD (respiratory events) were defined by adjudication or administrative criteria in four of nine cohorts. Results: Of 33,546 participants (mean age 52 yr, 54% female, 44% non-Hispanic White), 4,424 (13.2%) had prevalent SAO. The incidence of respiratory events (Nat-risk = 14,024) was threefold higher in participants with SAO versus those without (152 vs. 39 events/10,000 person-years). The probability score, which was based on six commonly available variables (age, sex, race and/or ethnicity, body mass index, smoking status, and smoking pack-years) was well calibrated and showed excellent discrimination in both the testing sample (C-statistic, 0.81; 95% confidence interval [CI], 0.80-0.82) and in NHANES (C-statistic, 0.83; 95% CI, 0.80-0.86). Among participants with predicted probabilities ⩾ 15%, 3.2 would need to undergo spirometry to detect one case of SAO. Conclusions: Adults with SAO demonstrate excess respiratory hospitalization and mortality. A probability score for SAO using commonly available clinical risk factors may be suitable for targeting screening and primary prevention strategies.

Two-stage improved Grey Wolf optimization algorithm for feature selection on high-dimensional classification

In recent years, evolutionary algorithms have shown great advantages in the field of feature selection because of their simplicity and potential global search capability. However, most of the existing feature selection algorithms based on evolutionary computation are wrapper methods, which are computationally expensive, especially for high-dimensional biomedical data. To significantly reduce the computational cost, it is essential to study an effective evaluation method. In this paper, a two-stage improved gray wolf optimization (IGWO) algorithm for feature selection on high-dimensional data is proposed. In the first stage, a multilayer perceptron (MLP) network with group lasso regularization terms is first trained to construct an integer optimization problem using the proposed algorithm for pre-selection of features and optimization of the hidden layer structure. The dataset is compressed using the feature subset obtained in the first stage. In the second stage, a multilayer perceptron network with group lasso regularization terms is retrained using the compressed dataset, and the proposed algorithm is employed to construct the discrete optimization problem for feature selection. Meanwhile, a rapid evaluation strategy is constructed to mitigate the evaluation cost and improve the evaluation efficiency in the feature selection process. The effectiveness of the algorithm was analyzed on ten gene expression datasets. The experimental results show that the proposed algorithm not only removes almost more than 95.7% of the features in all datasets, but also has better classification accuracy on the test set. In addition, the advantages of the proposed algorithm in terms of time consumption, classification accuracy and feature subset size become more and more prominent as the dimensionality of the feature selection problem increases. This indicates that the proposed algorithm is particularly suitable for solving high-dimensional feature selection problems.

Minimum Average Variance Estimation with group Lasso for the multivariate response Central Mean Subspace

The Minimum Average Variance Estimation (MAVE) method and its variants have proven to be effective approaches to the dimension reduction problems. However, as far as we know, using MAVE to estimate the Central Mean Subspace (CMS) with multivariate response receives little attention. This paper proposes a weighted version of MAVE for the CMS with multivariate response. The proposed weighted MAVE method takes account of the correlations among the responses and is a natural extension of the original MAVE method that only targets the univariate response CMS. The algorithm to implement the weighted MAVE method is provided. Asymptotic distribution of the MAVE estimator under the multivariate response setting is also derived. In addition, for the goal of variable selection, we propose incorporating the adaptive group lasso regularization method to induce sparse solutions. We show that the resulting sparse estimator can be calculated in a computationally efficient manner, and the associated BIC-type criterion can consistently select relevant variables. Experimental simulations and a real data analysis demonstrate the effectiveness and usefulness of the proposed methods.

Feature Selection Using a Neural Network With Group Lasso Regularization and Controlled Redundancy.

We propose a neural network-based feature selection (FS) scheme that can control the level of redundancy in the selected features by integrating two penalties into a single objective function. The Group Lasso penalty aims to produce sparsity in features in a grouped manner. The redundancy-control penalty, which is defined based on a measure of dependence among features, is utilized to control the level of redundancy among the selected features. Both the penalty terms involve the L2,1 -norm of weight matrix between the input and hidden layers. These penalty terms are nonsmooth at the origin, and hence, one simple but efficient smoothing technique is employed to overcome this issue. The monotonicity and convergence of the proposed algorithm are specified and proved under suitable assumptions. Then, extensive experiments are conducted on both artificial and real data sets. Empirical results explicitly demonstrate the ability of the proposed FS scheme and its effectiveness in controlling redundancy. The empirical simulations are observed to be consistent with the theoretical results.

Spline-rule ensemble classifiers with structured sparsity regularization for interpretable customer churn modeling

An important business domain that relies heavily on advanced statistical- and machine learning algorithms to support operational decision-making is customer retention management. Customer churn prediction is a crucial tool to support customer retention. It allows an early identification of customers who are at risk to abandon the company and provides the ability to gain insights into why customers are at risk. Hence, customer churn prediction models should complement predictive performance with model insights. Inspired by their ability to reconcile strong predictive performance and interpretability, this study introduces rule ensembles and their extension, spline-rule ensembles, as a promising family of classification algorithms to the customer churn prediction domain. Spline-rule ensembles combine the flexibility of a tree-based ensemble classifier with the simplicity of regression analysis. They do, however, neglect the relatedness between potentially conflicting model components which can introduce unnecessary complexity in the models and compromises model interpretability. To tackle this issue, a novel algorithmic extension, spline-rule ensembles with sparse group lasso regularization (SRE-SGL) is proposed to enhance interpretability through structured regularization. Experiments on fourteen real-world customer churn data sets in different industries (i) demonstrate the superior predictive performance of spline-rule ensembles with sparse group lasso over a set well yet powerful benchmark methods in terms of AUC and top decile lift; (ii) show that spline-rule ensembles with sparse group lasso regularization significantly outperform conventional rule ensembles whilst performing at least as well as conventional spline-rule ensembles; and (iii) illustrate the interpretable nature of a spline-rule ensemble model and the advantage of structured regularization in SRE-SGL by means of a case study on customer churn prediction for a telecommunications company.

Estimation for Varying Coefficient Models with Hierarchical Structure

The varying coefficient (VC) model is a generalization of ordinary linear model, which can not only retain strong interpretability but also has the flexibility of the nonparametric model. In this paper, we investigate a VC model with hierarchical structure. A unified variable selection method for VC model is proposed, which can simultaneously select the nonzero effects and estimate the unknown coefficient functions. Meanwhile, the selected model enforces the hierarchical structure, that is, interaction terms can be selected into the model only if the corresponding main effects are in the model. The kernel method is employed to estimate the varying coefficient functions, and a combined overlapped group Lasso regularization is introduced to implement variable selection to keep the hierarchical structure. It is proved that the proposed penalty estimators have oracle properties, that is, the coefficients are estimated as well as if the true model were known in advance. Simulation studies and a real data analysis are carried out to examine the performance of the proposed method in finite sample case.

LASSO Regularization within the LocalGLMnet Architecture

Deep learning models have been very successful in the application of machine learning methods, often out-performing classical statistical models such as linear regression models or generalized linear models. On the other hand, deep learning models are often criticized for not being explainable nor allowing for variable selection. There are two different ways of dealing with this problem, either we use post-hoc model interpretability methods or we design specific deep learning architectures that allow for (more) easy interpretation and explanation. This paper builds on our previous work on the LocalGLMnet architecture that gives an interpretable deep learning architecture. In the present paper, we show how group LASSO regularization (and other regularization schemes) can be implemented within the LocalGLMnet architecture so that we receive feature sparsity for variable selection. We benchmark our approach with the recently developed LassoNet of Lemhadri et al.

Convergence of a Gradient-Based Learning Algorithm With Penalty for Ridge Polynomial Neural Networks

Recently there have been renewed interests in high order neural networks (HONNs) for its powerful mapping capability. Ridge polynomial neural network (RPNN) is an important kind of HONNs, which always occupies a key position as an efficient instrument in the tasks of classification or regression. In order to make the convergence speed faster and the network generalization ability stronger, we introduce a regularization model for RPNN with Group Lasso penalty, which deals with the structural sparse problem at the group level in this paper. Nevertheless, there are two main obstacles for introducing the Group Lasso penalty, one is numerical oscillation and the other is convergence analysis challenge. In doing so, we adopt smoothing function to approximate the Group Lasso penalty to overcome these drawbacks. Meanwhile, strong and weak convergence theorems, and monotonicity theorems are provided for this novel algorithm. We also demonstrate the efficiency of our proposed algorithm by numerical experiments, and compare it to the no regularizer, $L_{2}$ regularizer, $L_{1/2}$ regularizer, smoothing $L_{1/2}$ regularizer, and the Group Lasso regularizer, and also the relevant theoretical analysis has been verified.

Deterministic convergence analysis via smoothing group Lasso regularization and adaptive momentum for Sigma-Pi-Sigma neural network

In this paper, we propose a sparse and accelerated method for Sigma-Pi-Sigma neural network training based on smoothing group lasso regularization and adaptive momentum. It is shown that group sparsity can more efficiently sparsity the network structure at the group level, and the adaptive momentum term can speed up the learning convergence during the iteration process. Also, another important contribution lies in the analysis of theoretical results. However, the group lasso regularization is not differentiable at the origin. This leads to oscillations observed in numerical experiments and poses a challenge to theoretical analysis. So, we overcome those problems by smoothing techniques. Under suitable assumptions, we strictly proved the monotonicity, and weak and strong convergence theorems of the new algorithm. Finally, the numerical experiments are presented to support our theoretical findings.

Enhancing incremental deep learning for FCCU end-point quality prediction

In the refining process, Fluid Catalytic Cracking Unit (FCCU) end-point quality prediction plays an important role in real-time quality monitoring, optimization and control. Due to the process uncertainties (such as changes of raw materials, abrasion of mechanic components, catalyst deactivation, changes in the external environment, etc), data distribution of process data in FCCU is time-varying, which leads to accuracy degradation of the quality prediction model. Therefore, to deal with the time varying characteristic of process data and avoid accuracy degradation of the quality prediction model, real-time processing should be considered. In this paper, an enhancing incremental deep learning approach is proposed for the online quality prediction of the absorption-stabilization system in FCCU. First, the offline model is built by Stacked Auto Encoder-Deep Neural Network (SAE-DNN). To determine whether the data distribution has changed and model modification is needed, a concept drift detector is proposed for the regression problem by defining an error bound in the Statistical Test of Equal Proportions (STEPD). If the model modification is needed, then the top layer of the offline SAE-DNN model is expanded by Random Vector Functional Link (RVFL) structure, and the parameters in the expansion layer is dynamically assigned by the new coming data with the group lasso regularization and the L2 regularization. The proposed approach is validated by predicting the Saturated Vapor Pressure (SVP) of stabilized gasoline in the FCCU. The experimental results show that the proposed approach can deal with the time-varying characteristic of process data and avoid accuracy degradation under process uncertainties.

Seq2Sick: Evaluating the Robustness of Sequence-to-Sequence Models with Adversarial Examples

Crafting adversarial examples has become an important technique to evaluate the robustness of deep neural networks (DNNs). However, most existing works focus on attacking the image classification problem since its input space is continuous and output space is finite. In this paper, we study the much more challenging problem of crafting adversarial examples for sequence-to-sequence (seq2seq) models, whose inputs are discrete text strings and outputs have an almost infinite number of possibilities. To address the challenges caused by the discrete input space, we propose a projected gradient method combined with group lasso and gradient regularization. To handle the almost infinite output space, we design some novel loss functions to conduct non-overlapping attack and targeted keyword attack. We apply our algorithm to machine translation and text summarization tasks, and verify the effectiveness of the proposed algorithm: by changing less than 3 words, we can make seq2seq model to produce desired outputs with high success rates. We also use an external sentiment classifier to verify the property of preserving semantic meanings for our generated adversarial examples. On the other hand, we recognize that, compared with the well-evaluated CNN-based classifiers, seq2seq models are intrinsically more robust to adversarial attacks.

Feature Selection for Neural Networks Using Group Lasso Regularization

We propose an embedded/integrated feature selection method based on neural networks with Group Lasso penalty. Group Lasso regularization is considered to produce sparsity on the inputs to the network, i.e., for selection of useful features. Lasso based feature selection using a multi-layer perceptron usually requires an additional set of weights, while our Group Lasso formulation does not require that. However, Group Lasso penalty is non-differentiable at the origin. This may lead to oscillations in numerical simulations and make it difficult to analyze theoretically. To address this issue, four smoothing Group Lasso penalties are introduced. A rigorous proof for the convergence of the proposed algorithm is presented under suitable assumptions. To verify the effectiveness, a three-step algorithmic architecture is adopted in implementation. Experimental results on several datasets validate the theoretical results and demonstrate the competitive performance of the proposed method.