Group Lasso Penalty Research Articles

Jewel: A Novel Method for Joint Estimation of Gaussian Graphical Models

In this paper, we consider the problem of estimating multiple Gaussian Graphical Models from high-dimensional datasets. We assume that these datasets are sampled from different distributions with the same conditional independence structure, but not the same precision matrix. We propose jewel, a joint data estimation method that uses a node-wise penalized regression approach. In particular, jewel uses a group Lasso penalty to simultaneously guarantee the resulting adjacency matrix’s symmetry and the graphs’ joint learning. We solve the minimization problem using the group descend algorithm and propose two procedures for estimating the regularization parameter. Furthermore, we establish the estimator’s consistency property. Finally, we illustrate our estimator’s performance through simulated and real data examples on gene regulatory networks.

A Bayesian group lasso classification for ADNI volumetrics data.

The primary objective of this paper is to develop a statistically valid classification procedure for analyzing brain image volumetrics data obtained from the Alzheimer's Disease Neuroimaging Initiative (ADNI) in elderly subjects with cognitive impairments. The Bayesian group lasso method thereby proposed for logistic regression efficiently selects an optimal model with the use of a spike and slab type prior. This method selects groups of attributes of a brain subregion encouraged by the group lasso penalty. We conduct simulation studies for high- and low-dimensional scenarios where our method is always able to select the true parameters that are truly predictive among a large number of parameters. The method is then applied on dichotomous response ADNI data which selects predictive atrophied brain regions and classifies Alzheimer's disease patients from healthy controls. Our analysis is able to give an accuracy rate of 80% for classifying Alzheimer's disease. The suggested method selects 29 brain subregions. The medical literature indicates that all these regions are associated with Alzheimer's patients. The Bayesian method of model selection further helps selecting only the subregions that are statistically significant, thus obtaining an optimal model.

Group-sparsity-enforcing fault discrimination and estimation with dynamic process data

Fisher discriminative analysis (FDA) has been recognized a prototypical approach to fault classification and diagnosis. To enhance model performance with time-series data used, it is customary to encompass lag measurements into the model. This not only increases model complexity prohibitively but also reduces the interpretability of fault diagnosis strategies. To address this issue, we propose a novel group-sparsity-enforcing FDA model, which utilizes reweighted group Lasso penalty to prune out irrelevant variables at the variable level so as to improve interpretability of discriminant directions. A tailored algorithm based on alternating direction method of multipliers is developed to solve the non-smooth and non-convex optimization problem. In addition, to identify root cause variables and unveil the fault evolution over time, a sparse fault estimation approach based on reweighted group Lasso is developed. This eventually allows to develop a holistic online scheme yielding informative diagnostic verdicts with faulty variable information used. Experimental results demonstrate that, the proposed model significantly improves the discriminant capability between normal and faulty data, and yields more interpretable discriminative information than conventional methods using dynamic process data.

Reasons for being unsure or unwilling regarding intention to take COVID-19 vaccine among Japanese people: A large cross-sectional national survey.

SummaryBackgroundIdentifying and understanding reasons for being unsure or unwilling regarding intention to be vaccinated against coronavirus disease (COVID-19) may help to inform future public health messages aimed at increasing vaccination coverage. We analyzed a broad array of individual's psychological dispositions with regard to decision-making about COVID-19 vaccination in Japan.MethodsA nationally representative cross-sectional web survey was conducted with 30053 Japanese adults aged 20 years or older at the end of February 2021. In addition to the question on the individual's intention to be vaccinated against COVID-19, respondents were asked about their sociodemographic, health-related, and psychological characteristics as well as information sources about COVID-19 and their levels of trust. Also, those who responded ‘not sure’ or ‘no’ regarding intention to take COVID-19 vaccine were asked why. Multinomial logistic regression with sparse group Lasso (Least Absolute Shrinkage and Selection Operator) penalty was used to compute adjusted odds ratios for factors associated with the intention (not sure/no versus yes).FindingsThe percentages of respondents who answered ‘not sure’ or ‘no’ regarding intention to be vaccinated against COVID-19 vaccine were 32.9% and 11.0%, respectively. After adjusting for covariates, the perceived risks of COVID-19, perceived risk of a COVID-19 vaccine, perceived benefits of a COVID-19 vaccine, trust in scientists and public authorities, and the belief that healthcare workers should be vaccinated were significantly associated with vaccination intention. Several sources of information about COVID-19 were also significantly associated with vaccination intention, including physicians, nurses, and television, medical information sites with lower odds of being unsure or unwilling, and internet news sites, YouTube, family members, and scientists and researchers with higher odds. The higher the level of trust in television as a source of COVID-19 information, the higher the odds of responding ‘not sure’ (odds ratio 1.11, 95% confidence interval 1.01–1.21). We also demonstrated that many respondents presented concerns about the side effects and safety of a COVID-19 vaccine as a major reason for being unsure or unwilling. To decide whether or not to get the vaccine, many respondents requested more information about the compatibilities between the vaccine and their personal health conditions, whether other people had been vaccinated, the effectiveness of vaccines against variants, and doctors’ recommendations.InterpretationOur findings suggest that public health messaging based on the sociodemographic and psychological characteristics of those who are unsure or unwilling regarding intention to be vaccinated against COVID-19 vaccine may help to increase vaccine uptake amongst this population.FundingThe present work was supported in part by a grant from the Ministry of Health, Labour and Welfare of Japan (H29-Gantaisaku-ippan-009).

RAMRSGL: A Robust Adaptive Multinomial Regression Model for Multicancer Classification.

In view of the challenges of the group Lasso penalty methods for multicancer microarray data analysis, e.g., dividing genes into groups in advance and biological interpretability, we propose a robust adaptive multinomial regression with sparse group Lasso penalty (RAMRSGL) model. By adopting the overlapping clustering strategy, affinity propagation clustering is employed to obtain each cancer gene subtype, which explores the group structure of each cancer subtype and merges the groups of all subtypes. In addition, the data-driven weights based on noise are added to the sparse group Lasso penalty, combining with the multinomial log-likelihood function to perform multiclassification and adaptive group gene selection simultaneously. The experimental results on acute leukemia data verify the effectiveness of the proposed method.

Sparse PLS-Based Method for Overlapping Metabolite Set Enrichment Analysis.

Metabolite set enrichment analysis (MSEA) has gained increasing research interest for identification of perturbed metabolic pathways in metabolomics. The method incorporates predefined metabolic pathways information in the analysis where metabolite sets are typically assumed to be mutually exclusive to each other. However, metabolic pathways are known to contain common metabolites and intermediates. This situation, along with limitations in metabolite detection or coverage leads to overlapping, incomplete metabolite sets in pathway analysis. For overlapping metabolite sets, MSEA tends to result in high false positives due to improper weights allocated to the overlapping metabolites. Here, we proposed an extended partial least squares (PLS) model with a new sparse scheme for overlapping metabolite set enrichment analysis, named overlapping group PLS (ogPLS) analysis. The weight vector of the ogPLS model was decomposed into pathway-specific subvectors, and then a group lasso penalty was imposed on these subvectors to achieve a proper weight allocation for the overlapping metabolites. Two strategies were adopted in the proposed ogPLS model to identify the perturbed metabolic pathways. The first strategy involves debiasing regularization, which was used to reduce inequalities amongst the predefined metabolic pathways. The second strategy is stable selection, which was used to rank pathways while avoiding the nuisance problems of model parameter optimization. Both simulated and real-world metabolomic datasets were used to evaluate the proposed method and compare with two other MSEA methods including Global-test and the multiblock PLS (MB-PLS)-based pathway importance in projection (PIP) methods. Using a simulated dataset with known perturbed pathways, the average true discovery rate for the ogPLS method was found to be higher than the Global-test and the MB-PLS-based PIP methods. Analysis with a real-world metabolomics dataset also indicated that the developed method was less prone to select pathways with highly overlapped detected metabolite sets. Compared with the two other methods, the proposed method features higher accuracy, lower false-positive rate, and is more robust when applied to overlapping metabolite set analysis. The developed ogPLS method may serve as an alternative MSEA method to facilitate biological interpretation of metabolomics data for overlapping metabolite sets.

Extracting information from textual descriptions for actuarial applications

AbstractInitial insurance losses are often reported with a textual description of the claim. The claims manager must determine the adequate case reserve for each known claim. In this paper, we present a framework for predicting the amount of loss given a textual description of the claim using a large number of words found in the descriptions. Prior work has focused on classifying insurance claims based on keywords selected by a human expert, whereas in this paper the focus is on loss amount prediction with automatic word selection. In order to transform words into numeric vectors, we use word cosine similarities and word embedding matrices. When we consider all unique words found in the training dataset and impose a generalised additive model to the resulting explanatory variables, the resulting design matrix is high dimensional. For this reason, we use a group lasso penalty to reduce the number of coefficients in the model. The scalable, analytical framework proposed provides for a parsimonious and interpretable model. Finally, we discuss the implications of the analysis, including how the framework may be used by an insurance company and how the interpretation of the covariates can lead to significant policy change. The code can be found in the TAGAM R package (github.com/scottmanski/TAGAM).

Feature Selection Using a Neural Network With Group Lasso Regularization and Controlled Redundancy.

We propose a neural network-based feature selection (FS) scheme that can control the level of redundancy in the selected features by integrating two penalties into a single objective function. The Group Lasso penalty aims to produce sparsity in features in a grouped manner. The redundancy-control penalty, which is defined based on a measure of dependence among features, is utilized to control the level of redundancy among the selected features. Both the penalty terms involve the L2,1 -norm of weight matrix between the input and hidden layers. These penalty terms are nonsmooth at the origin, and hence, one simple but efficient smoothing technique is employed to overcome this issue. The monotonicity and convergence of the proposed algorithm are specified and proved under suitable assumptions. Then, extensive experiments are conducted on both artificial and real data sets. Empirical results explicitly demonstrate the ability of the proposed FS scheme and its effectiveness in controlling redundancy. The empirical simulations are observed to be consistent with the theoretical results.

Stable Anatomy Detection in Multimodal Imaging Through Sparse Group Regularization: A Comparative Study of Iron Accumulation in the Aging Brain.

Multimodal neuroimaging provides a rich source of data for identifying brain regions associated with disease progression and aging. However, present studies still typically analyze modalities separately or aggregate voxel-wise measurements and analyses to the structural level, thus reducing statistical power. As a central example, previous works have used two quantitative MRI parameters—R2* and quantitative susceptibility (QS)—to study changes in iron associated with aging in healthy and multiple sclerosis subjects, but failed to simultaneously account for both. In this article, we propose a unified framework that combines information from multiple imaging modalities and regularizes estimates for increased interpretability, generalizability, and stability. Our work focuses on joint region detection problems where overlap between effect supports across modalities is encouraged but not strictly enforced. To achieve this, we combine L1 (lasso), total variation (TV), and L2 group lasso penalties. While the TV penalty encourages geometric regularization by controlling estimate variability and support boundary geometry, the group lasso penalty accounts for similarities in the support between imaging modalities. We address the computational difficulty in this regularization scheme with an alternating direction method of multipliers (ADMM) optimizer. In a neuroimaging application, we compare our method against independent sparse and joint sparse models using a dataset of R2* and QS maps derived from MRI scans of 113 healthy controls: our method produces clinically-interpretable regions where specific iron changes are associated with healthy aging. Together with results across multiple simulation studies, we conclude that our approach identifies regions that are more strongly associated with the variable of interest (e.g., age), more accurate, and more stable with respect to training data variability. This work makes progress toward a stable and interpretable multimodal imaging analysis framework for studying disease-related changes in brain structure and can be extended for classification and disease prediction tasks.

Integrative analysis of multiple types of genomic data using an accelerated failure time frailty model

As the high throughput technologies rapidly develop, multiple types of genomic data become available within and across different studies. It has become a challenging task in modern statistical research to use all types of genomic data to infer some disease-prone genetic information. In this work, we propose an integrative analysis of multiple and different types of genomic data, clinical covariates and survival data under a framework of an accelerated failure time with frailty model. The proposed integrative approach aims to answer some aspects of the complex problem in genomic data analysis by finding relevant genomic features and inferring patients’ survival time using identified features. The proposed integrative approach is developed using a weighted least-squares with a sparse group LASSO penalty as the objective function to simultaneously estimate and select the relevant features. Extensive simulation studies are conducted to assess the performance of the proposed method with two types of genomic data, DNA methylation data and copy number variation data, on 600 genes and three clinical covariates. The simulation results show promises of the proposed method. The proposed method is applied to the analysis of the Cancer Genome Atlas data on Glioblastoma, a lethal brain cancer, and biologically interpretable results are obtained.

A Joint Graphical Model for Inferring Gene Networks Across Multiple Subpopulations and Data Types.

Reconstructing gene networks from gene expression data is a long-standing challenge. In most applications, the observations can be divided into several distinct but related subpopulations and the gene expression measurements can be collected from multiple data types. Most existing methods are designed to estimate a single gene network from a single dataset. These methods may be suboptimal since they do not exploit the similarities and differences among different subpopulations and data types. In this article, we propose a joint graphical model to estimate the multiple gene networks simultaneously. Our model decomposes each subpopulation-specific gene network as a sum of common and unique components and imposes a group lasso penalty on gene networks corresponding to different data types. The gene network variations across subpopulations can be learned automatically by the decompositions of networks, and the similarities and differences among data types can be captured by the group lasso penalty. The simulation studies demonstrate that our method outperforms the state-of-the-art methods. We also apply our method to the cancer genome atlas breast cancer datasets to reconstruct subtype-specific gene networks. Hub nodes in the estimated subnetworks unique to individual cancer subtypes rediscover well-known genes associated with breast cancer subtypes and provide interesting predictions.

Sparse-Group Lasso for Graph Learning From Multi-Attribute Data

We consider the problem of inferring the conditional independence graph (CIG) of high-dimensional Gaussian vectors from multi-attribute data. Most existing methods for graph estimation are based on single-attribute models where one associates a scalar random variable with each node. In multi-attribute graphical models, each node represents a random vector. In this paper, we present a sparse-group lasso based penalized log-likelihood approach for graph learning from multi-attribute data. Existing works on multi-attribute graphical modeling have considered only group lasso penalty. The main objective of this paper is to explore the use of sparse-group lasso for multi-attribute graph estimation. An alternating direction method of multipliers (ADMM) algorithm is presented to optimize the objective function to estimate the inverse covariance matrix. Sufficient conditions for consistency and sparsistency of the estimator are provided. Numerical results based on synthetic as well as real data are presented.

Neural Granger Causality.

While most classical approaches to Granger causality detection assume linear dynamics, many interactions in real-world applications, like neuroscience and genomics, are inherently nonlinear. In these cases, using linear models may lead to inconsistent estimation of Granger causal interactions. We propose a class of nonlinear methods by applying structured multilayer perceptrons (MLPs) or recurrent neural networks (RNNs) combined with sparsity-inducing penalties on the weights. By encouraging specific sets of weights to be zero-in particular, through the use of convex group-lasso penalties-we can extract the Granger causal structure. To further contrast with traditional approaches, our framework naturally enables us to efficiently capture long-range dependencies between series either via our RNNs or through an automatic lag selection in the MLP. We show that our neural Granger causality methods outperform state-of-the-art nonlinear Granger causality methods on the DREAM3 challenge data. This data consists of nonlinear gene expression and regulation time courses with only a limited number of time points. The successes we show in this challenging dataset provide a powerful example of how deep learning can be useful in cases that go beyond prediction on large datasets. We likewise illustrate our methods in detecting nonlinear interactions in a human motion capture dataset.

Logistic Regression Based on Statistical Learning Model with Linearized Kernel for Classification

In this paper, we propose a logistic regression classification method based on the integration of a statistical learning model with linearized kernel pre-processing. The single Gaussian kernel and fusion of Gaussian and cosine kernels are adopted for linearized kernel pre-processing respectively. The adopted statistical learning models are the generalized linear model and the generalized additive model. Using a generalized linear model, the elastic net regularization is adopted to explore the grouping effect of the linearized kernel feature space. Using a generalized additive model, an overlap group-lasso penalty is used to fit the sparse generalized additive functions within the linearized kernel feature space. Experiment results on the Extended Yale-B face database and AR face database demonstrate the effectiveness of the proposed method. The improved solution is also efficiently obtained using our method on the classification of spectra data.

Convergence of a Gradient-Based Learning Algorithm With Penalty for Ridge Polynomial Neural Networks

Recently there have been renewed interests in high order neural networks (HONNs) for its powerful mapping capability. Ridge polynomial neural network (RPNN) is an important kind of HONNs, which always occupies a key position as an efficient instrument in the tasks of classification or regression. In order to make the convergence speed faster and the network generalization ability stronger, we introduce a regularization model for RPNN with Group Lasso penalty, which deals with the structural sparse problem at the group level in this paper. Nevertheless, there are two main obstacles for introducing the Group Lasso penalty, one is numerical oscillation and the other is convergence analysis challenge. In doing so, we adopt smoothing function to approximate the Group Lasso penalty to overcome these drawbacks. Meanwhile, strong and weak convergence theorems, and monotonicity theorems are provided for this novel algorithm. We also demonstrate the efficiency of our proposed algorithm by numerical experiments, and compare it to the no regularizer, $L_{2}$ regularizer, $L_{1/2}$ regularizer, smoothing $L_{1/2}$ regularizer, and the Group Lasso regularizer, and also the relevant theoretical analysis has been verified.

Convex clustering method for compositional data via sparse group lasso

High-dimensional sparse clustering with compositional data is of great practical importance, as exemplified by applications in high-throughput gene expression profiles analysis. In this paper, we develop a compositional clustering framework based on convex clustering, which is a convex relaxation of hierarchical clustering that incorporates a fused penalty term on the cluster prototypes. To explicitly deal with the issue of high dimensionality and sparsity, we propose the Compositional Convex Clustering with Sparse Group Lasso (CCC-SGL). The isometric logratio (ilr) transformation is first applied to transform the composition in the simplex space to the standard Euclidean geometry. Then, a group lasso penalty and a lasso penalty are imposed on the cluster centers, which effectively selects informative features and promotes within-feature sparsity. The proposed convex clustering formulation is numerically and efficiently solved with the proximal gradient descent algorithm within the Alternating Direction Method of Multipliers (ADMM) framework. Simulation studies are carried out to evaluate the performance of the proposed methodology and also a real data set in microbiome sequencing is analyzed.

Granger causality detection in high-dimensional systems using feedforward neural networks

This paper proposes a novel methodology to detect Granger causality on average in vector autoregressive settings using feedforward neural networks. The approach accommodates unknown dependence structures between elements of high-dimensional multivariate time series with weak and strong persistence. To do this, we propose a two-stage procedure: first, we maximize the transfer of information between input and output variables in the network in order to obtain an optimal number of nodes in the intermediate hidden layers. Second, we apply a novel sparse double group lasso penalty function in order to identify the variables that have the predictive ability and, hence, indicate that Granger causality is present in the others. The penalty function inducing sparsity is applied to the weights that characterize the nodes of the neural network. We show the correct identification of these weights so as to increase sample sizes. We apply this method to the recently created Tobalaba network of renewable energy companies and show the increase in connectivity between companies after the creation of the network using Granger causality measures to map the connections.

Sparse Multicategory Generalized Distance Weighted Discrimination in Ultra-High Dimensions.

Distance weighted discrimination (DWD) is an appealing classification method that is capable of overcoming data piling problems in high-dimensional settings. Especially when various sparsity structures are assumed in these settings, variable selection in multicategory classification poses great challenges. In this paper, we propose a multicategory generalized DWD (MgDWD) method that maintains intrinsic variable group structures during selection using a sparse group lasso penalty. Theoretically, we derive minimizer uniqueness for the penalized MgDWD loss function and consistency properties for the proposed classifier. We further develop an efficient algorithm based on the proximal operator to solve the optimization problem. The performance of MgDWD is evaluated using finite sample simulations and miRNA data from an HIV study.

A Novel Convex Clustering Method for High-Dimensional Data Using Semiproximal ADMM

Clustering is an important ingredient of unsupervised learning; classical clustering methods include K-means clustering and hierarchical clustering. These methods may suffer from instability because of their tendency prone to sink into the local optimal solutions of the nonconvex optimization model. In this paper, we propose a new convex clustering method for high-dimensional data based on the sparse group lasso penalty, which can simultaneously group observations and eliminate noninformative features. In this method, the number of clusters can be learned from the data instead of being given in advance as a parameter. We theoretically prove that the proposed method has desirable statistical properties, including a finite sample error bound and feature screening consistency. Furthermore, the semiproximal alternating direction method of multipliers is designed to solve the sparse group lasso convex clustering model, and its convergence analysis is established without any conditions. Finally, the effectiveness of the proposed method is thoroughly demonstrated through simulated experiments and real applications.

A dual reformulation and solution framework for regularized convex clustering problems

Clustering techniques are powerful tools commonly used in statistical learning and data analytics. Most of the past research formulates clustering tasks as a non-convex problem, where a global optimum often cannot be found. Recent studies show that hierarchical clustering and k-means clustering can be relaxed and analyzed as a convex problem. Moreover, sparse convex clustering algorithms are proposed to extend the convex clustering framework to high-dimensional space by introducing an adaptive group-Lasso penalty term. Due to the non-smoothness nature of the associated objective functions, there are still no efficient fast-convergent algorithms for clustering problems even with convexity. In this paper, we first review the structure of convex clustering problems and prove the differentiability of their dual problems. We then show that such reformulated dual problems can be efficiently solved by the accelerated first-order methods with the feasibility projection. Furthermore, we present a general framework for convex clustering with regularization terms and discuss a specific implementation of this framework using L1,1-norm. We also derive the dual form for the regularized convex clustering problems and show that it can be efficiently solved by embedding a projection operator and a proximal operator in the accelerated gradient method. Finally, we compare our approach with several other co-clustering algorithms using a number of example clustering problems. Numerical results show that our models and solution methods outperform all the compared algorithms for both convex clustering and convex co-clustering.