Sparse Group Lasso Research Articles

Variable selection in social-environmental data: sparse regression and tree ensemble machine learning approaches

BackgroundSocial-environmental data obtained from the US Census is an important resource for understanding health disparities, but rarely is the full dataset utilized for analysis. A barrier to incorporating the full data is a lack of solid recommendations for variable selection, with researchers often hand-selecting a few variables. Thus, we evaluated the ability of empirical machine learning approaches to identify social-environmental factors having a true association with a health outcome.MethodsWe compared several popular machine learning methods, including penalized regressions (e.g. lasso, elastic net), and tree ensemble methods. Via simulation, we assessed the methods’ ability to identify census variables truly associated with binary and continuous outcomes while minimizing false positive results (10 true associations, 1000 total variables). We applied the most promising method to the full census data (p = 14,663 variables) linked to prostate cancer registry data (n = 76,186 cases) to identify social-environmental factors associated with advanced prostate cancer.ResultsIn simulations, we found that elastic net identified many true-positive variables, while lasso provided good control of false positives. Using a combined measure of accuracy, hierarchical clustering based on Spearman’s correlation with sparse group lasso regression performed the best overall. Bayesian Adaptive Regression Trees outperformed other tree ensemble methods, but not the sparse group lasso. In the full dataset, the sparse group lasso successfully identified a subset of variables, three of which replicated earlier findings.ConclusionsThis analysis demonstrated the potential of empirical machine learning approaches to identify a small subset of census variables having a true association with the outcome, and that replicate across empiric methods. Sparse clustered regression models performed best, as they identified many true positive variables while controlling false positive discoveries.

Convex clustering method for compositional data via sparse group lasso

High-dimensional sparse clustering with compositional data is of great practical importance, as exemplified by applications in high-throughput gene expression profiles analysis. In this paper, we develop a compositional clustering framework based on convex clustering, which is a convex relaxation of hierarchical clustering that incorporates a fused penalty term on the cluster prototypes. To explicitly deal with the issue of high dimensionality and sparsity, we propose the Compositional Convex Clustering with Sparse Group Lasso (CCC-SGL). The isometric logratio (ilr) transformation is first applied to transform the composition in the simplex space to the standard Euclidean geometry. Then, a group lasso penalty and a lasso penalty are imposed on the cluster centers, which effectively selects informative features and promotes within-feature sparsity. The proposed convex clustering formulation is numerically and efficiently solved with the proximal gradient descent algorithm within the Alternating Direction Method of Multipliers (ADMM) framework. Simulation studies are carried out to evaluate the performance of the proposed methodology and also a real data set in microbiome sequencing is analyzed.

Granger causality detection in high-dimensional systems using feedforward neural networks

This paper proposes a novel methodology to detect Granger causality on average in vector autoregressive settings using feedforward neural networks. The approach accommodates unknown dependence structures between elements of high-dimensional multivariate time series with weak and strong persistence. To do this, we propose a two-stage procedure: first, we maximize the transfer of information between input and output variables in the network in order to obtain an optimal number of nodes in the intermediate hidden layers. Second, we apply a novel sparse double group lasso penalty function in order to identify the variables that have the predictive ability and, hence, indicate that Granger causality is present in the others. The penalty function inducing sparsity is applied to the weights that characterize the nodes of the neural network. We show the correct identification of these weights so as to increase sample sizes. We apply this method to the recently created Tobalaba network of renewable energy companies and show the increase in connectivity between companies after the creation of the network using Granger causality measures to map the connections.

Sparse Multicategory Generalized Distance Weighted Discrimination in Ultra-High Dimensions.

Distance weighted discrimination (DWD) is an appealing classification method that is capable of overcoming data piling problems in high-dimensional settings. Especially when various sparsity structures are assumed in these settings, variable selection in multicategory classification poses great challenges. In this paper, we propose a multicategory generalized DWD (MgDWD) method that maintains intrinsic variable group structures during selection using a sparse group lasso penalty. Theoretically, we derive minimizer uniqueness for the penalized MgDWD loss function and consistency properties for the proposed classifier. We further develop an efficient algorithm based on the proximal operator to solve the optimization problem. The performance of MgDWD is evaluated using finite sample simulations and miRNA data from an HIV study.

Cancer Diagnosis and Disease Gene Identification via Statistical Machine Learning

Diagnosing cancer and identifying the disease gene by using DNA microarray gene expression data are the hot topics in current bioinformatics. This paper is devoted to the latest development in cancer diagnosis and gene selection via statistical machine learning. A support vector machine is firstly introduced for the binary cancer diagnosis. Then, 1-norm support vector machine, doubly regularized support vector machine, adaptive huberized support vector machine and other extensions are presented to improve the performance of gene selection. Lasso, elastic net, partly adaptive elastic net, group lasso, sparse group lasso, adaptive sparse group lasso and other sparse regression methods are also introduced for performing simultaneous binary cancer classification and gene selection. In addition to introducing three strategies for reducing multiclass to binary, methods of directly considering all classes of data in a learning model (multi_class support vector, sparse multinomial regression, adaptive multinomial regression and so on) are presented for performing multiple cancer diagnosis. Limitations and promising directions are also discussed.

A Novel Convex Clustering Method for High-Dimensional Data Using Semiproximal ADMM

Clustering is an important ingredient of unsupervised learning; classical clustering methods include K-means clustering and hierarchical clustering. These methods may suffer from instability because of their tendency prone to sink into the local optimal solutions of the nonconvex optimization model. In this paper, we propose a new convex clustering method for high-dimensional data based on the sparse group lasso penalty, which can simultaneously group observations and eliminate noninformative features. In this method, the number of clusters can be learned from the data instead of being given in advance as a parameter. We theoretically prove that the proposed method has desirable statistical properties, including a finite sample error bound and feature screening consistency. Furthermore, the semiproximal alternating direction method of multipliers is designed to solve the sparse group lasso convex clustering model, and its convergence analysis is established without any conditions. Finally, the effectiveness of the proposed method is thoroughly demonstrated through simulated experiments and real applications.

Seagull: lasso, group lasso and sparse-group lasso regularization for linear regression models via proximal gradient descent

BackgroundStatistical analyses of biological problems in life sciences often lead to high-dimensional linear models. To solve the corresponding system of equations, penalization approaches are often the methods of choice. They are especially useful in case of multicollinearity, which appears if the number of explanatory variables exceeds the number of observations or for some biological reason. Then, the model goodness of fit is penalized by some suitable function of interest. Prominent examples are the lasso, group lasso and sparse-group lasso. Here, we offer a fast and numerically cheap implementation of these operators via proximal gradient descent. The grid search for the penalty parameter is realized by warm starts. The step size between consecutive iterations is determined with backtracking line search. Finally, seagull -the R package presented here- produces complete regularization paths.ResultsPublicly available high-dimensional methylation data are used to compare seagull to the established R package SGL. The results of both packages enabled a precise prediction of biological age from DNA methylation status. But even though the results of seagull and SGL were very similar (R2 > 0.99), seagull computed the solution in a fraction of the time needed by SGL. Additionally, seagull enables the incorporation of weights for each penalized feature.ConclusionsThe following operators for linear regression models are available in seagull: lasso, group lasso, sparse-group lasso and Integrative LASSO with Penalty Factors (IPF-lasso). Thus, seagull is a convenient envelope of lasso variants.

The sparse group lasso for high-dimensional integrative linear discriminant analysis with application to alzheimer's disease prediction

In the big data era, high-dimensional data from multi-type data sources are ubiquitous and pose great challenge for statistical analysis. In this paper, we focus on two-class classification problem for this kind of data. Motivated by the least squares formulation of linear discriminant analysis, we propose an integrative discrimination analysis method by virtue of sparse group Lasso penalty. Thorough numerical studies are conducted and the results show that the proposed method compares favourably with other popular sparse discriminant proposals in terms of both variable selection accuracy and classification error. A real-data example is also given to illustrate its advantages.

Adaptive sparse group LASSO in quantile regression

This paper studies the introduction of sparse group LASSO (SGL) to the quantile regression framework. Additionally, a more flexible version, an adaptive SGL is proposed based on the adaptive idea, this is, the usage of adaptive weights in the penalization. Adaptive estimators are usually focused on the study of the oracle property under asymptotic and double asymptotic frameworks. A key step on the demonstration of this property is to consider adaptive weights based on a initial $$\sqrt{n}$$ -consistent estimator. In practice this implies the usage of a non penalized estimator that limits the adaptive solutions to low dimensional scenarios. In this work, several solutions, based on dimension reduction techniques PCA and PLS, are studied for the calculation of these weights in high dimensional frameworks. The benefits of this proposal are studied both in synthetic and real datasets.

Sparse Elitist Group Lasso Denoising in Frequency Domain for Bearing Fault Diagnosis

The fault-induced impulse responses of localized bearing fault are usually interfered by the background noise and other harmonic components. They are strongly coupled together and are hard to be separated. It is crucial to develop a fast and reliable method to extract the impulse-based feature for online bearing fault diagnosis in the industry application. In this article, we propose a new sparse elitist group lasso denoising (SEGLD) algorithm in frequency domain to detect the incipient impulse-based fault feature, which is free of utilizing the prior knowledge. We first reveal the sparse characteristics of the bearing fault signals in frequency domain. Then, a tailored denoising model is proposed. To obtain a satisfactory analytical stationary solution, the Douglas-Rachford splitting solver is employed for the denoising model. Moreover, we explore the relationship between the best regularization parameters, the periodic information and the normalization estimated noise of the rolling bearing fault signal. A rule of adaptively selecting the best regularization parameters is demonstrated. Finally, the robustness and effectiveness of the proposed SEGLD algorithm are profoundly verified by the numerical simulation and two evaluation experiments under the conditions of early fault stage and low speed scenario. Also, it is demonstrated that the proposed approach outperforms the state-of-the-art method for extracting the weak fault feature.

Seismic Absorption Qualitative Indicator via Sparse Group-Lasso-Based Time–Frequency Representation

Time-frequency (TF) analysis is an available tool to estimate seismic absorption qualitatively. The high TF concentration is a key factor for the seismic attenuation qualitative estimation. To obtain a more concentrated TF representation, we propose a sparse TF method based on sparse representation (SR) and sparse Group-Lasso (GL) penalty function. Based on the SR theory, TF representation can be regarded as an inverse problem, and thus, sparse GL penalty function can be added in this inverse problem to enhance the TF concentration. Sparse GL penalty function, including <i xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">l</i> <sub xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">1</sub> penalty and <i xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">l</i> <sub xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">2,1</sub> penalty, can provide group-wise and within-group sparsity for TF coefficients. Using the proposed sparse GL-based TF (GLTF) method, we develop a workflow to characterize seismic attenuation qualitatively. Finally, a synthetic data of viscoacoustic model and a 2-D field data are applied to test the validity and effectiveness of the proposed workflow for indicating the gas and oil reservoirs.

Accounting for grouped predictor variables or pathways in high-dimensional penalized Cox regression models

BackgroundThe standard lasso penalty and its extensions are commonly used to develop a regularized regression model while selecting candidate predictor variables on a time-to-event outcome in high-dimensional data. However, these selection methods focus on a homogeneous set of variables and do not take into account the case of predictors belonging to functional groups; typically, genomic data can be grouped according to biological pathways or to different types of collected data. Another challenge is that the standard lasso penalisation is known to have a high false discovery rate.ResultsWe evaluated different penalizations in a Cox model to select grouped variables in order to further penalize variables that, in addition to having a low effect, belong to a group with a low overall effect; and to favor the selection of variables that, in addition to having a large effect, belong to a group with a large overall effect. We considered the case of prespecified and disjoint groups and proposed diverse weights for the adaptive lasso method. In particular we proposed the product Max Single Wald by Single Wald weighting (MSW*SW) which takes into account the information of the group to which it belongs and of this biomarker. Through simulations, we compared the selection and prediction ability of our approach with the standard lasso, the composite Minimax Concave Penalty (cMCP), the group exponential lasso (gel), the Integrative L1-Penalized Regression with Penalty Factors (IPF-Lasso), and the Sparse Group Lasso (SGL) methods. In addition, we illustrated the methods using gene expression data of 614 breast cancer patients.ConclusionsThe adaptive lasso with the MSW*SW weighting method incorporates both the information in the grouping structure and the individual variable. It outperformed the competitors by reducing the false discovery rate without severely increasing the false negative rate.

High-dimensional penalized arch processes

We introduce a general methodology to consistently estimate multidimensional ARCH models equation-by-equation, possibly with a very large number of parameters through penalization (Sparse Group Lasso). Some families of multidimensional ARCH models are proposed to tackle homogeneous or heterogeneous portfolios of assets. The corresponding conditions of stationarity and of positive definiteness are studied. We evaluate the relevance of such a strategy by simulation. The relative forecasting performances of our models are compared through the management of financial portfolios.

Multi-task learning sparse group lasso: a method for quantifying antigenicity of influenza A(H1N1) virus using mutations and variations in glycosylation of Hemagglutinin

BackgroundIn addition to causing the pandemic influenza outbreaks of 1918 and 2009, subtype H1N1 influenza A viruses (IAVs) have caused seasonal epidemics since 1977. Antigenic property of influenza viruses are determined by both protein sequence and N-linked glycosylation of influenza glycoproteins, especially hemagglutinin (HA). The currently available computational methods are only considered features in protein sequence but not N-linked glycosylation.ResultsA multi-task learning sparse group least absolute shrinkage and selection operator (LASSO) (MTL-SGL) regression method was developed and applied to derive two types of predominant features including protein sequence and N-linked glycosylation in hemagglutinin (HA) affecting variations in serologic data for human and swine H1N1 IAVs. Results suggested that mutations and changes in N-linked glycosylation sites are associated with the rise of antigenic variants of H1N1 IAVs. Furthermore, the implicated mutations are predominantly located at five reported antibody-binding sites, and within or close to the HA receptor binding site. All of the three N-linked glycosylation sites (i.e. sequons NCSV at HA 54, NHTV at HA 125, and NLSK at HA 160) identified by MTL-SGL to determine antigenic changes were experimentally validated in the H1N1 antigenic variants using mass spectrometry analyses. Compared with conventional sparse learning methods, MTL-SGL achieved a lower prediction error and higher accuracy, indicating that grouped features and MTL in the MTL-SGL method are not only able to handle serologic data generated from multiple reagents, supplies, and protocols, but also perform better in genetic sequence-based antigenic quantification.ConclusionsIn summary, the results of this study suggest that mutations and variations in N-glycosylation in HA caused antigenic variations in H1N1 IAVs and that the sequence-based antigenicity predictive model will be useful in understanding antigenic evolution of IAVs.

Penalized models for analysis of multiple mediators.

Mediation analysis attempts to determine whether the relationship between an independent variable (e.g., exposure) and an outcome variable can be explained, at least partially, by an intermediate variable, called a mediator. Most methods for mediation analysis focus on one mediator at a time, although multiple mediators can be jointly analyzed by structural equation models (SEMs) that account for correlations among the mediators. We extend the use of SEMsfor the analysis of multiple mediators by creating a sparse group lasso penalized model such that the penalty considers the natural groupings of parameters that determine mediation, as well as encourages sparseness of the model parameters. This provides a way to simultaneously evaluate many mediators and select those that have the most impact, a feature of modern penalized models. Simulations are used to illustrate the benefits and limitations of our approach, and application to a study of DNA methylation and reactive cortisol stress following childhood trauma discovered two novel methylation loci that mediate the association of childhood trauma scores with reactive cortisol stress levels. Our new methods are incorporated into R software called regmed.

Genetic Variants Detection Based on Weighted Sparse Group Lasso

Identification of genetic variants associated with complex traits is a critical step for improving plant resistance and breeding. Although the majority of existing methods for variants detection have good predictive performance in the average case, they can not precisely identify the variants present in a small number of target genes. In this paper, we propose a weighted sparse group lasso (WSGL) method to select both common and low-frequency variants in groups. Under the biologically realistic assumption that complex traits are influenced by a few single loci in a small number of genes, our method involves a sparse group lasso approach to simultaneously select associated groups along with the loci within each group. To increase the probability of selecting out low-frequency variants, biological prior information is introduced in the model by re-weighting lasso regularization based on weights calculated from input data. Experimental results from both simulation and real data of single nucleotide polymorphisms (SNPs) associated with Arabidopsis flowering traits demonstrate the superiority of WSGL over other competitive approaches for genetic variants detection.

Hypernetwork Construction and Feature Fusion Analysis Based on Sparse Group Lasso Method on fMRI Dataset

Recent works have shown that the resting-state brain functional connectivity hypernetwork, where multiple nodes can be connected, are an effective technique for brain disease diagnosis and classification research. The lasso method was used to construct hypernetworks by solving sparse linear regression models in previous research. But, constructing a hypernetwork based on the lasso method simply selects a single variable, in that it lacks the ability to interpret the grouping effect. Considering the group structure problem, the previous study proposed to create a hypernetwork based on the elastic net and the group lasso methods, and the results showed that the former method had the best classification performance. However, the highly correlated variables selected by the elastic net method were not necessarily in the active set in the group. Therefore, we extended our research to address this issue. Herein, we propose a new method that introduces the sparse group lasso method to improve the construction of the hypernetwork by solving the group structure problem of the brain regions. We used the traditional lasso, group lasso method, and sparse group lasso method to construct a hypernetwork in patients with depression and normal subjects. Meanwhile, other clustering coefficients (clustering coefficients based on pairs of nodes) were also introduced to extract features with traditional clustering coefficients. Two types of features with significant differences obtained after feature selection were subjected to multi-kernel learning for feature fusion and classification using each method, respectively. The network topology results revealed differences among the three networks, where hypernetwork using the lasso method was the strictest; the group lasso, most lenient; and the sgLasso method, moderate. The network topology of the sparse group lasso method was similar to that of the group lasso method but different from the lasso method. The classification results show that the sparse group lasso method achieves the best classification accuracy by using multi-kernel learning, which indicates that better classification performance can be achieved when the group structure exists and is properly extended.

Group Guided Fused Laplacian Sparse Group Lasso for Modeling Alzheimer’s Disease Progression

As the largest cause of dementia, Alzheimer's disease (AD) has brought serious burdens to patients and their families, mostly in the financial, psychological, and emotional aspects. In order to assess the progression of AD and develop new treatment methods for the disease, it is essential to infer the trajectories of patients' cognitive performance over time to identify biomarkers that connect the patterns of brain atrophy and AD progression. In this article, a structured regularized regression approach termed group guided fused Laplacian sparse group Lasso (GFL-SGL) is proposed to infer disease progression by considering multiple prediction of the same cognitive scores at different time points (longitudinal analysis). The proposed GFL-SGL simultaneously exploits the interrelated structures within the MRI features and among the tasks with sparse group Lasso (SGL) norm and presents a novel group guided fused Laplacian (GFL) regularization. This combination effectively incorporates both the relatedness among multiple longitudinal time points with a general weighted (undirected) dependency graphs and useful inherent group structure in features. Furthermore, an alternating direction method of multipliers- (ADMM-) based algorithm is also derived to optimize the nonsmooth objective function of the proposed approach. Experiments on the dataset from Alzheimer's Disease Neuroimaging Initiative (ADNI) show that the proposed GFL-SGL outperformed some other state-of-the-art algorithms and effectively fused the multimodality data. The compact sets of cognition-relevant imaging biomarkers identified by our approach are consistent with the results of clinical studies.

Schizophrenia Identification Using Multi-View Graph Measures of Functional Brain Networks.

Schizophrenia (SZ) is a functional mental disorder that seriously affects the social life of patients. Therefore, accurate diagnosis of SZ has raised extensive attention of researchers. At present, study of brain network based on resting-state functional magnetic resonance imaging (rs-fMRI) has provided promising results for SZ identification by studying functional network alteration. However, previous studies based on brain network analysis are not very effective for SZ identification. Therefore, we propose an improved SZ identification method using multi-view graph measures of functional brain networks. Firstly, we construct an individual functional connectivity network based on Brainnetome atlas for each subject. Then, multi-view graph measures are calculated by the brain network analysis method as feature representations. Next, in order to consider the relationships between measures within the same brain region in feature selection, multi-view measures are grouped according to the corresponding regions and Sparse Group Lasso is applied to identify discriminative features based on this feature grouping structure. Finally, a support vector machine (SVM) classifier is employed to perform SZ identification task. To evaluate our proposed method, computational experiments are conducted on 145 subjects (71 schizophrenic patients and 74 healthy controls) using a leave-one-out cross-validation (LOOCV) scheme. The results show that our proposed method can obtain an accuracy of 93.10% for SZ identification. By comparison, our method is more effective for SZ identification than some existing methods.

A group lasso based sparse KNN classifier

Sparse features have been shown effective in applications of computer vision, machine learning, signal processing, etc. Group sparsity was proposed by considering that there exist natural group structures in many problems. Previous research mainly focuses on improving the way to extract sparse features, such as lasso, group lasso, overlapped group lasso, sparse group lasso. In existing work, sparse features are usually taken as input for classifiers, such as SVM, KNN, or SRC (Sparse Representation based Classification). In this paper, we find that, instead of using sparse group features as input for classifiers, sparse group features are good candidates for selection of most relevant classes/groups. We design a new classifier to improve classification accuracy: (1) we use sparse group lasso to select K most relevant classes/groups, which makes this approach robust, because it filters out unrelated classes/groups in group level, instead of individual sample level; (2) KSVD is used to get exact desired sparsity (k nonzero entries) and thus eliminates the difficulty of hyperparameter tuning; (3) simple summation of regression weights within each class/group contains sufficient class discriminant information, and the chance of a sample belonging to a specific class is denoted simply by the summation of corresponding regression weights within each class, which is in line with the need of Explainable AI (XAI). The K most relevant groups/classes can be considered as K neighbors of the correct class. Thus, we call this classifier Group Lasso based Sparse KNN (GLSKNN). Compared to 8 other approaches, GLSKNN classifier outperforms other methods in term of classification accuracy for two public image datasets and images with different occlusion/noise levels.