Overlapping Group Lasso Research Articles

Dual-modal Image Reconstruction for Electrical Impedance Tomography with Overlapping Group Lasso and Laplacian Regularization.

Electrical Impedance Tomography (EIT) is a promising biomedical imaging modality, yet EIT image reconstruction remains an open challenge due to its severe ill-posedness. High-quality EIT image reconstruction algorithms are desired. This paper reports a segmentation-free dual-modal EIT image reconstruction algorithm that uses Overlapping Group Lasso and Laplacian (OGLL) regularization. An overlapping group lasso penalty is constructed based on conductivity change properties and encodes the imaging targets' structural information obtained from an auxiliary imaging modality that provides structural images of the sensing region. We introduce Laplacian regularization to alleviate the artifacts caused by group overlapping. The performance of OGLL is evaluated and compared with single-modal and dual-modal image reconstruction algorithms using simulation and real-world data. Quantitative metrics and visualized images confirm the superiority of the proposed method in terms of structure preservation, background artifact (BA) suppression, and conductivity contrast differentiation. This work proves the effectiveness of OGLL in improving EIT image quality. This study demonstrates that EIT has the potential to be adopted in quantitative tissue analysis by using such dual-modal imaging approaches.

Development and validation of a prediction model for the recurrence of stage 1 EGFR mutation positive NSCLC in patients using machine learning with WES-based gene sets.

8563 Background: Predicting the risk of postoperative recurrence is becoming increasingly important in EGFR mutation positive ( EGFR-m) non-small cell lung cancer (NSCLC). Only a few reports conducted whole exome sequencing (WES) for genomic profiling in large scale EGFR-m NSCLC cases. Methods: We conducted WES and analyzed the clinical course of patients (pts) with NSCLC who underwent surgery between 1985 and 2019 in the PRISM project of our institute. We evaluated the EGFR-m patients’ characteristics, recurrence-free survival (RFS), and developed a prediction model using machine learning, Overlapping Group LASSO to predict whether the EGFR-m patients will recur within 5 years or not as "high-risk" and "low-risk". To develop and validate the prediction model, we divided the data from 2006 as the training cohort data set and the data before 2006 as the validation cohort data set. After the development of the prediction model, we performed a stratified Cox proportional hazards model to compare the RFS between the predicted groups. Results: A total of 585/1351(43.3%) pts were EGFR-m included in the PRISM project. Of all pts, stage I, EGFR-m were 205 pts. The median RFS of the stage I, EGFR-m 123.1 months (m). In the training cohort, the model detected 43 pts for high-risk and 88 pts for low-risk. In the validation cohort, the model predicted 29 pts for high-risk and 45 pts for low-risk. Median RFS of high-risk vs. low-risk were 52.2 m vs. 105 m (HR 2.43, p=0.01). The 1-year RFS rate, 2-year RFS rate, and 5-year RFS rates for high-risk and low-risk pts were 100% vs. 98.5%, 66.7% vs. 89.2%, and 41.7% vs. 66.1%, respectively. Twenty-eight gene set coefficients were non-zero in the prediction model. The gene sets with large positive coefficients that were considered important for prediction as "high risk" were the gene sets affected by KRAS gene overexpression and p53 gene knockdown. In contrast, gene sets repressed by mTOR inhibition and genes repressed by TBK1 gene knockdown and KRAS gene overexpression had large negative coefficients. Conclusions: We developed and validated the prediction model whether the EGFR-m patients will recur within 5 years or not. EGFR-m NSCLC recurrence appears to be high risk with pathways associated with KRAS and p53 genes, and low risk with mutations in the gene sets suppressed by the mTOR pathway and TBK1 gene. [Table: see text]

Brain space image reconstruction of functional near-infrared spectroscopy using a Bayesian adaptive fused sparse overlapping group lasso model.

Functional near-infrared spectroscopy (fNIRS) is a noninvasive technology that uses low levels of nonionizing light in the range of red and near-infrared to record changes in the optical absorption and scattering of the underlying tissue that can be used to infer blood flow and oxygen changes during brain activity. The challenges and difficulties of reconstructing spatial images of hemoglobin changes from fNIRS data are mainly caused by the illposed nature of the optical inverse model. We describe a Bayesian approach combining several lasso-based regularizations to apply anatomy-prior information to solving the inverse model. We built a Bayesian hierarchical model to solve the Bayesian adaptive fused sparse overlapping group lasso (Ba-FSOGL) model. The method is evaluated and validated using simulation and experimental datasets. We apply this approach to the simulation and experimental datasets to reconstruct a known brain activity. The reconstructed images and statistical plots are shown. We discuss the adaptation of this method to fNIRS data and demonstrate that this approach provides accurate image reconstruction with a low false-positive rate, through numerical simulations and application to experimental data collected during motor and sensory tasks.

Stochastic relaxed inertial forward-backward-forward splitting for monotone inclusions in Hilbert spaces

We consider monotone inclusions defined on a Hilbert space where the operator is given by the sum of a maximal monotone operator T and a single-valued monotone, Lipschitz continuous, and expectation-valued operator V. We draw motivation from the seminal work by Attouch and Cabot (Attouch in AMO 80:547–598, 2019, Attouch in MP 184: 243–287) on relaxed inertial methods for monotone inclusions and present a stochastic extension of the relaxed inertial forward–backward-forward method. Facilitated by an online variance reduction strategy via a mini-batch approach, we show that our method produces a sequence that weakly converges to the solution set. Moreover, it is possible to estimate the rate at which the discrete velocity of the stochastic process vanishes. Under strong monotonicity, we demonstrate strong convergence, and give a detailed assessment of the iteration and oracle complexity of the scheme. When the mini-batch is raised at a geometric (polynomial) rate, the rate statement can be strengthened to a linear (suitable polynomial) rate while the oracle complexity of computing an epsilon -solution improves to {mathcal {O}}(1/epsilon ). Importantly, the latter claim allows for possibly biased oracles, a key theoretical advancement allowing for far broader applicability. By defining a restricted gap function based on the Fitzpatrick function, we prove that the expected gap of an averaged sequence diminishes at a sublinear rate of {mathcal {O}}(1/k) while the oracle complexity of computing a suitably defined epsilon -solution is {mathcal {O}}(1/epsilon ^{1+a}) where a > 1. Numerical results on two-stage games and an overlapping group Lasso problem illustrate the advantages of our method compared to competitors.

FUNCTION GROUP SELECTION OF SEMBUNG LEAVES (BLUMEA BALSAMIFERA) SIGNIFICANT TO ANTIOXIDANTS USING OVERLAPPING GROUP LASSO

Functional groups of sembung leaf metabolites can be detected using FTIR spectrometry by looking at the spectrum's shape from specific peaks that indicate the type of functional group of a compound. There were 35 observations and 1866 explanatory variables (wavelength) in this study. The number of explanatory variables more than the number of observations is high-dimensional data. One method that can be used to analyze high-dimensional data is penalized regression. The overlapping group lasso method is a development of the group-based penalized regression method that can solve the problem of selecting variable groups and members of overlapping groups of variables. The results of selecting the variable groups using the overlapping group lasso method found that the functional groups that were significant for the antioxidants of sembung leaves were C=C Unstructured, CN amide, Polyphenol, Sio2.

Robust structured heterogeneity analysis approach for high-dimensional data.

Revealing relationships between genes and disease phenotypes is a critical problem in biomedical studies. This problem has been challenged by the heterogeneity of diseases. Patients of a perceived same disease may form multiple subgroups, and different subgroups have distinct sets of important genes. It is hence imperative to discover the latent subgroups and reveal the subgroup-specific important genes. Some heterogeneity analysis methods have been proposed in the recent literature. Despite considerable successes, most of the existing studies are still limited as they cannot accommodate data contamination and ignore the interconnections among genes. Aiming at these shortages, we develop a robust structured heterogeneity analysis approach to identify subgroups, select important genes as well as estimate their effects on the phenotype of interest. Possible data contamination is accommodated by employing the Huber loss function. A sparse overlapping group lasso penalty is imposed to conduct regularization estimation and gene identification, while taking into account the possibly overlapping cluster structure of genes. This approach takes an iterative strategy in the similar spirit of K-means clustering. Simulations demonstrate that the proposed approach outperforms alternatives in revealing the heterogeneity and selecting important genes for each subgroup. The analysis of Cancer Cell Line Encyclopedia data leads to biologically meaningful findings with improved prediction and grouping stability.

A First-Order Optimization Algorithm for Statistical Learning with Hierarchical Sparsity Structure

In many statistical learning problems, it is desired that the optimal solution conform to an a priori known sparsity structure represented by a directed acyclic graph. Inducing such structures by means of convex regularizers requires nonsmooth penalty functions that exploit group overlapping. Our study focuses on evaluating the proximal operator of the latent overlapping group lasso developed by Jacob et al. in 2009 . We implemented an alternating direction method of multiplier with a sharing scheme to solve large-scale instances of the underlying optimization problem efficiently. In the absence of strong convexity, global linear convergence of the algorithm is established using the error bound theory. More specifically, the paper contributes to establishing primal and dual error bounds when the nonsmooth component in the objective function does not have a polyhedral epigraph. We also investigate the effect of the graph structure on the speed of convergence of the algorithm. Detailed numerical simulation studies over different graph structures supporting the proposed algorithm and two applications in learning are provided. Summary of Contribution: The paper proposes a computationally efficient optimization algorithm to evaluate the proximal operator of a nonsmooth hierarchical sparsity-inducing regularizer and establishes its convergence properties. The computationally intensive subproblem of the proposed algorithm can be fully parallelized, which allows solving large-scale instances of the underlying problem. Comprehensive numerical simulation studies benchmarking the proposed algorithm against five other methods on the speed of convergence to optimality are provided. Furthermore, performance of the algorithm is demonstrated on two statistical learning applications related to topic modeling and breast cancer classification. The code along with the simulation studies and benchmarks are available on the corresponding author’s GitHub website for evaluation and future use.

Kurdyka–Łojasiewicz Exponent via Inf-projection

Kurdyka–Łojasiewicz (KL) exponent plays an important role in estimating the convergence rate of many contemporary first-order methods. In particular, a KL exponent of $$\frac{1}{2}$$ for a suitable potential function is related to local linear convergence. Nevertheless, KL exponent is in general extremely hard to estimate. In this paper, we show under mild assumptions that KL exponent is preserved via inf-projection. Inf-projection is a fundamental operation that is ubiquitous when reformulating optimization problems via the lift-and-project approach. By studying its operation on KL exponent, we show that the KL exponent is $$\frac{1}{2}$$ for several important convex optimization models, including some semidefinite-programming-representable functions and some functions that involve $$C^2$$ -cone reducible structures, under conditions such as strict complementarity. Our results are applicable to concrete optimization models such as group-fused Lasso and overlapping group Lasso. In addition, for nonconvex models, we show that the KL exponent of many difference-of-convex functions can be derived from that of their natural majorant functions, and the KL exponent of the Bregman envelope of a function is the same as that of the function itself. Finally, we estimate the KL exponent of the sum of the least squares function and the indicator function of the set of matrices of rank at most k.

Selective linearization for multi-block statistical learning

We consider the problem of minimizing a sum of several convex non-smooth functions and discuss the selective linearization method (SLIN), which iteratively linearizes all but one of the functions and employs simple proximal steps. The algorithm is a form of multiple operator splitting in which the order of processing partial functions is not fixed, but rather determined in the course of calculations. SLIN is globally convergent for an arbitrary number of component functions without artificial duplication of variables. We report results from extensive numerical experiments in two statistical learning settings such as large-scale overlapping group Lasso and doubly regularized support vector machine. In each setting, we introduce novel and efficient solutions for solving sub-problems. The numerical results demonstrate the efficacy and accuracy of SLIN.

An Efficient Algorithm for Minimizing Multi Non-Smooth Component Functions

ABSTRACT Many problems in statistics and machine learning can be formulated as an optimization problem of a finite sum of nonsmooth convex functions. We propose an algorithm to minimize this type of objective functions based on the idea of alternating linearization. Our algorithm retains the simplicity of contemporary methods without any restrictive assumptions on the smoothness of the loss function. We apply our proposed method to solve two challenging problems: overlapping group lasso and convex regression with sharp partitions. Numerical experiments show that our method is superior to the state-of-the-art algorithms, many of which are based on the accelerated proximal gradient method. Supplementary materials for this article are available online.

Convergence analysis of accelerated proximal extra-gradient method with applications

Proximal algorithms are popular class of methods for handling sparsity structure in the datasets due to their low iteration costs and faster convergence. In this paper, we consider the framework of the sum of two convex functions, one of which is a smooth function with a Lipschitz gradient, while the other may be a non-smooth function. The usages of such non-smooth functions for identifying complex sparsity-structures in datasets in form of non-smooth regularizers has been an active research direction in the recent past. In this paper, we present the convergence analysis for the extragradient-based fixed-point method with an inertial component, based on which recently a new accelerated proximal extragradient algorithm is designed. In addition, extending the application areas of this algorithm, we applied it to solve (i) the logistic regression problem with complex ℓ1-based penalties, namely, overlapping group lasso and fused lasso frameworks, and (ii) a recently proposed structurally-regularized learning problem for representation selection where the objective function consists of a reconstruction error and structured regularizers as combination of group sparsity regularizer, diversity regularizer, and locality-sensitivity regularizer. With the help of extensive experiments on several publicly available real-world datasets, the efficacy of the inertial-based extragradient methods has been demonstrated for solving the extended lasso and representation selection problems of machine learning.

Multiresolution Functional ANOVA for Large-Scale, Many-Input Computer Experiments

The Gaussian process is a standard tool for building emulators for both deterministic and stochastic computer experiments. However, application of Gaussian process models is greatly limited in practice, particularly for large-scale and many-input computer experiments that have become typical. We propose a multiresolution functional ANOVA (MRFA) model as a computationally feasible emulation alternative. More generally, this model can be used for large-scale and many-input nonlinear regression problems. An overlapping group lasso approach is used for estimation, ensuring computational feasibility in a large-scale and many-input setting. New results on consistency and inference for the (potentially overlapping) group lasso in a high-dimensional setting are developed and applied to the proposed MRFA model. Importantly, these results allow us to quantify the uncertainty in our predictions. Numerical examples demonstrate that the proposed model enjoys marked computational advantages. Data capabilities, in terms of both sample size and dimension, meet or exceed best available emulation tools while meeting or exceeding emulation accuracy. Supplementary materials for this article are available online.

Use of Overlapping Group LASSO Sparse Deep Belief Network to Discriminate Parkinson's Disease and Normal Control.

As a medical imaging technology which can show the metabolism of the brain, 18F-fluorodeoxyglucose (FDG)-positron emission tomography (PET) is of great value for the diagnosis of Parkinson's Disease (PD). With the development of pattern recognition technology, analysis of brain images using deep learning are becoming more and more popular. However, existing computer-aided-diagnosis technologies often over fit and have poor generalizability. Therefore, we aimed to improve a framework based on Group Lasso Sparse Deep Belief Network (GLS-DBN) for discriminating PD and normal control (NC) subjects based on FDG-PET imaging. In this study, 225 NC and 125 PD cohorts from Huashan and Wuxi 904 hospitals were selected. They were divided into the training & validation dataset and 2 test datasets. First, in the training & validation set, subjects were randomly partitioned 80:20, with multiple training iterations for the deep learning model. Next, Locally Linear Embedding was used as a dimension reduction algorithm. Then, GLS-DBN was used for feature learning and classification. Different sparse DBN models were used to compare datasets to evaluate the effectiveness of our framework. Accuracy, sensitivity, and specificity were examined to validate the results. Output variables of the network were also correlated with longitudinal changes of rating scales about movement disorders (UPDRS, H&Y). As a result, accuracy of prediction (90% in Test 1, 86% in Test 2) for classification of PD and NC patients outperformed conventional approaches. Output scores of the network were strongly correlated with UPDRS and H&Y (R = 0.705, p < 0.001; R = 0.697, p < 0.001 in Test 1; R = 0.592, p = 0.0018, R = 0.528, p = 0.0067 in Test 2). These results show the GLS-DBN is feasible method for early diagnosis of PD.

Overlapping group lasso for high-dimensional generalized linear models

Structured sparsity has recently been a very popular technique to deal with the high-dimensional data. In this paper, we mainly focus on the theoretical problems for the overlapping group structure of generalized linear models (GLMs). Although the overlapping group lasso method for GLMs has been widely applied in some applications, the theoretical properties about it are still unknown. Under some general conditions, we presents the oracle inequalities for the estimation and prediction error of overlapping group Lasso method in the generalized linear model setting. Then, we apply these results to the so-called Logistic and Poisson regression models. It is shown that the results of the Lasso and group Lasso procedures for GLMs can be recovered by specifying the group structures in our proposed method. The effect of overlap and the performance of variable selection of our proposed method are both studied by numerical simulations. Finally, we apply our proposed method to two gene expression data sets: the p53 data and the lung cancer data.

Group-sparse SVD Models via $L_1$- and $L_0$-norm Penalties and Their Applications in Biological Data

Sparse Singular Value Decomposition (SVD) models have been proposed for biclustering high dimensional gene expression data to identify block patterns with similar expressions. However, these models do not take into account prior group effects upon variable selection. To this end, we first propose group-sparse SVD models with group Lasso ( $GL_1$ G L 1 -SVD) and group $L_0$ L 0 -norm penalty ( $GL_0$ G L 0 -SVD) for non-overlapping group structure of variables. However, such group-sparse SVD models limit their applicability in some problems with overlapping structure. Thus, we also propose two group-sparse SVD models with overlapping group Lasso ( $OGL_1$ O G L 1 -SVD) and overlapping group $L_0$ L 0 -norm penalty ( $OGL_0$ O G L 0 -SVD). We first adopt an alternating iterative strategy to solve $GL_1$ G L 1 -SVD based on a block coordinate descent method, and $GL_0$ G L 0 -SVD based on a projection method. The key of solving $OGL_1$ O G L 1 -SVD is a proximal operator with overlapping group Lasso penalty. We employ an alternating direction method of multipliers (ADMM) to solve the proximal operator. Similarly, we develop an approximate method to solve $OGL_0$ O G L 0 -SVD. Applications of these methods and comparison with competing ones using simulated data demonstrate their effectiveness. Extensive applications of them onto several real gene expression data with gene prior group knowledge identify some biologically interpretable gene modules.

Learning with optimal interpolation norms

We analyze a class of norms defined via an optimal interpolation problem involving the composition of norms and a linear operator. This construction, known as infimal postcomposition in convex analysis, is shown to encompass various norms which have been used as regularizers in machine learning, signal processing, and statistics. In particular, these include the latent group lasso, the overlapping group lasso, and certain norms used for learning tensors. We establish basic properties of this class of norms and we provide dual norms. The extension to more general classes of convex functions is also discussed. A stochastic block-coordinate version of the Douglas-Rachford algorithm is devised to solve minimization problems involving these regularizers. A prominent feature of the algorithm is that it yields iterates that converge to a solution in the case of nonsmooth losses and random block updates. Finally, we present numerical experiments with problems employing the latent group lasso penalty.

Adaptive multinomial regression with overlapping groups for multi-class classification of lung cancer

Multi-class classification has attracted much attention in cancer diagnosis and treatment and many machine learning methods have emerged for addressing this issue recently. However, class imbalance and gene selection problems occur in classifying lung cancer data. In this paper, an adaptive multinomial regression with a sparse overlapping group lasso penalty is proposed to perform classification and grouped gene selection for lung cancer gene expression data. An overlapped grouping strategy with biological interpretability is proposed, which highlights the importance of gene groups from the minority classes. By using the conditional mutual information, the gene significance within each group is evaluated and the data-driven weights are constructed. Based on the grouping strategy and constructed weights, a regularized adaptive multinomial regression is presented and the solving algorithm is developed, which can not only select the important gene groups for each class in performing multi-class classification, but also adaptively select important genes within each group. The experiment results show that the proposed method significantly outperforms the other 6 methods on classification accuracy, and the selected genes are disease-causing genes for lung cancer.

Grouped gene selection and multi-classification of acute leukemia via new regularized multinomial regression

Diagnosing acute leukemia is the necessary prerequisite to treating it. Multi-classification on the gene expression data of acute leukemia is help for diagnosing it which contains B-cell acute lymphoblastic leukemia (BALL), T-cell acute lymphoblastic leukemia (TALL) and acute myeloid leukemia (AML). However, selecting cancer-causing genes is a challenging problem in performing multi-classification. In this paper, weighted gene co-expression networks are employed to divide the genes into groups. Based on the dividing groups, a new regularized multinomial regression with overlapping group lasso penalty (MROGL) has been presented to simultaneously perform multi-classification and select gene groups. By implementing this method on three-class acute leukemia data, the grouped genes which work synergistically are identified, and the overlapped genes shared by different groups are also highlighted. Moreover, MROGL outperforms other five methods on multi-classification accuracy.

Continuation of Nesterov's Smoothing for Regression With Structured Sparsity in High-Dimensional Neuroimaging.

Predictive models can be used on high-dimensional brain images to decode cognitive states or diagnosis/prognosis of a clinical condition/evolution. Spatial regularization through structured sparsity offers new perspectives in this context and reduces the risk of overfitting the model while providing interpretable neuroimaging signatures by forcing the solution to adhere to domain-specific constraints. Total variation (TV) is a promising candidate for structured penalization: it enforces spatial smoothness of the solution while segmenting predictive regions from the background. We consider the problem of minimizing the sum of a smooth convex loss, a non-smooth convex penalty (whose proximal operator is known) and a wide range of possible complex, non-smooth convex structured penalties such as TV or overlapping group Lasso. Existing solvers are either limited in the functions they can minimize or in their practical capacity to scale to high-dimensional imaging data. Nesterov's smoothing technique can be used to minimize a large number of non-smooth convex structured penalties. However, reasonable precision requires a small smoothing parameter, which slows down the convergence speed to unacceptable levels. To benefit from the versatility of Nesterov's smoothing technique, we propose a first order continuation algorithm, CONESTA, which automatically generates a sequence of decreasing smoothing parameters. The generated sequence maintains the optimal convergence speed toward any globally desired precision. Our main contributions are: gap to probe the current distance to the global optimum in order to adapt the smoothing parameter and the To propose an expression of the duality convergence speed. This expression is applicable to many penalties and can be used with other solvers than CONESTA. We also propose an expression for the particular smoothing parameter that minimizes the number of iterations required to reach a given precision. Furthermore, we provide a convergence proof and its rate, which is an improvement over classical proximal gradient smoothing methods. We demonstrate on both simulated and high-dimensional structural neuroimaging data that CONESTA significantly outperforms many state-of-the-art solvers in regard to convergence speed and precision.

Speech bottleneck feature extraction method based on overlapping group lasso sparse deep neural network

In this paper, a novel speech bottleneck feature extraction method based on overlapping group lasso sparse deep neural network is proposed. This method extracts the speech bottleneck features which contain supervised class information and adjacent voice frames related information by changing the architecture of deep neural network (DNN), to improve of the performance speech recognition. Firstly, in order to construct the sparse deep neural network, the sparse regularization of the overlapping group lasso algorithm is added to the single DNN object function which is regarded as the penalty item. Second, the speech bottleneck features can be extracted from the bottleneck layer of the sparse bottleneck DNN (BN-DNN). Finally, the speech bottleneck features are used as the input features of the deep neural network-hidden Markov model (DNN-HMM) speech recognition system. The large vocabulary continuous speech recognition experiment results on the Switchboard database indicate that the algorithm proposed in this paper can extract the speech bottleneck features with priori information, and reduce the word error rate in continuous speech recognition.