L1 Penalty Research Articles

Sparse simulation-based estimator built on quantiles

The method of simulated quantiles is extended to a general multivariate framework and to provide sparse estimation of the scaling matrix. The method is based on the minimisation of a distance between appropriate statistics evaluated on the true and synthetic data simulated from the postulated model. Those statistics are functions of the quantiles providing an effective way to deal with distributions that do not admit moments of any order like the α–Stable or the Tukey lambda distribution. The lack of a natural ordering represents the major challenge for the extension of the method to the multivariate framework, which is addressed by considering the notion of projectional quantile. The SCAD ℓ1–penalty is then introduced in order to achieve sparse estimation of the scaling matrix which is mostly responsible for the curse of dimensionality. The asymptotic properties of the proposed estimator have been discussed and the method is illustrated and tested on several synthetic datasets simulated from the Elliptical Stable distribution for which alternative methods are recognised to perform poorly.

Age-Coherent Mortality Modeling and Forecasting Using a Constrained Sparse Vector-Autoregressive Model

Accurate forecasts and analyses of mortality rates are essential to many practical issues, such as population projections and the design of pension schemes. Recent studies have considered a spatial–temporal autoregressive (STAR) model, in which the mortality rates of each age depend on their own historical values (temporality) and the neighboring cohort ages (spatiality). Despite the realization of age coherence and improved forecasting accuracy over the famous Lee-Carter (LC) model, the assumption of STAR that only the effects of the same and the neighboring cohorts exist can be too restrictive. In this study, we adopt a data-driven principle, as in a sparse vector autoregressive (SVAR) model, to improve the flexibility of the parametric structure of STAR and develop a constrained SVAR (CSVAR) model. To solve its objective function consisting of non-standard L2 and L1 penalties subject to constraints, we develop a new algorithm and prove the existence of the desirable age-coherence in CSVAR. Using empirical data from the United Kingdom, France, Italy, Spain, and Australia, we show that CSVAR consistently outperforms the LC, SVAR, and STAR models with respect to forecasting accuracy. The estimates and forecasts of the CSVAR model also demonstrate important demographic differences between these five countries.

Spectral decoupling for training transferable neural networks in medical imaging

SummaryMany neural networks for medical imaging generalize poorly to data unseen during training. Such behavior can be caused by overfitting easy-to-learn features while disregarding other potentially informative features. A recent implicit bias mitigation technique called spectral decoupling provably encourages neural networks to learn more features by regularizing the networks' unnormalized prediction scores with an L2 penalty. We show that spectral decoupling increases the networks′ robustness for data distribution shifts and prevents overfitting on easy-to-learn features in medical images. To validate our findings, we train networks with and without spectral decoupling to detect prostate cancer on tissue slides and COVID-19 in chest radiographs. Networks trained with spectral decoupling achieve up to 9.5 percent point higher performance on external datasets. Spectral decoupling alleviates generalization issues associated with neural networks and can be used to complement or replace computationally expensive explicit bias mitigation methods, such as stain normalization in histological images.

Feature selection for kernel methods in systems biology.

The substantial development of high-throughput biotechnologies has rendered large-scale multi-omics datasets increasingly available. New challenges have emerged to process and integrate this large volume of information, often obtained from widely heterogeneous sources. Kernel methods have proven successful to handle the analysis of different types of datasets obtained on the same individuals. However, they usually suffer from a lack of interpretability since the original description of the individuals is lost due to the kernel embedding. We propose novel feature selection methods that are adapted to the kernel framework and go beyond the well-established work in supervised learning by addressing the more difficult tasks of unsupervised learning and kernel output learning. The method is expressed under the form of a non-convex optimization problem with a ℓ1 penalty, which is solved with a proximal gradient descent approach. It is tested on several systems biology datasets and shows good performances in selecting relevant and less redundant features compared to existing alternatives. It also proved relevant for identifying important governmental measures best explaining the time series of Covid-19 reproducing number evolution during the first months of 2020. The proposed feature selection method is embedded in the R package mixKernel version 0.8, published on CRAN. Installation instructions are available at http://mixkernel.clementine.wf/.

Space-log: a novel approach to inferring gene-gene net-works using SPACE model with log penalty.

Gene expression data have been used to infer gene-gene networks (GGN) where an edge between two genes implies the conditional dependence of these two genes given all the other genes. Such gene-gene networks are of-ten referred to as gene regulatory networks since it may reveal expression regulation. Most of existing methods for identifying GGN employ penalized regression with L1(lasso), L2(ridge), or elastic net penalty, which spans the range of L1to L2penalty. However, for high dimensional gene expression data, a penalty that spans the range of L0and L1penalty, such as the log penalty, is often needed for variable selection consistency. Thus, we develop a novel method that em-ploys log penalty within the framework of an earlier network identification method space (Sparse PArtial Correlation Estimation), and implement it into a R package space-log. We show that the space-log is computationally efficient (source code implemented in C), and has good performance comparing with other methods, particularly for networks with hubs. Space-log is open source and available at GitHub,https://github.com/wuqian77/SpaceLog.

Variable Selection for Robust Mixture Regression Model with Skew Scale Mixtures of Normal Distributions

In this paper, we propose a robust mixture regression model based on the skew scale mixtures of normal distributions (RMR-SSMN) which can accommodate asymmetric, heavy-tailed and contaminated data better. For the variable selection problem, the penalized likelihood approach with a new combined penalty function which balances the SCAD and l2 penalty is proposed. The adjusted EM algorithm is presented to get parameter estimates of RMR-SSMN models at a faster convergence rate. As simulations show, our mixture models are more robust than general FMR models and the new combined penalty function outperforms SCAD for variable selection. Finally, the proposed methodology and algorithm are applied to a real data set and achieve reasonable results.

High-dimensional sufficient dimension reduction through principal projections

We develop in this work a new dimension reduction method for high-dimensional settings. The proposed procedure is based on a principal support vector machine framework where principal projections are used in order to overcome the non-invertibility of the covariance matrix. Using a series of equivalences we show that one can accurately recover the central subspace using a projection on a lower dimensional subspace and then applying an ℓ1 penalization strategy to obtain sparse estimators of the sufficient directions. Based next on a desparsified estimator, we provide an inferential procedure for high-dimensional models that allows testing for the importance of variables in determining the sufficient direction. Theoretical properties of the methodology are illustrated and computational advantages are demonstrated with simulated and real data experiments.

IPGM: Inertial Proximal Gradient Method for Convolutional Dictionary Learning

Inspired by the recent success of the proximal gradient method (PGM) and recent efforts to develop an inertial algorithm, we propose an inertial PGM (IPGM) for convolutional dictionary learning (CDL) by jointly optimizing both an ℓ2-norm data fidelity term and a sparsity term that enforces an ℓ1 penalty. Contrary to other CDL methods, in the proposed approach, the dictionary and needles are updated with an inertial force by the PGM. We obtain a novel derivative formula for the needles and dictionary with respect to the data fidelity term. At the same time, a gradient descent step is designed to add an inertial term. The proximal operation uses the thresholding operation for needles and projects the dictionary to a unit-norm sphere. We prove the convergence property of the proposed IPGM algorithm in a backtracking case. Simulation results show that the proposed IPGM achieves better performance than the PGM and slice-based methods that possess the same structure and are optimized using the alternating-direction method of multipliers (ADMM).

A Combination of Generalized Linear Mixed Model and LASSO Methods for Estimating Number of Patients Covid 19 in the Intensive Care Units

Generalized linear mixed models (GLMM) combined with the L1 penalty (Least Absolute Shrinkage and Selection Operator/LASSO) is called LASSO GLMM. LASSO GLMM reduces overfitting and selects predictor variables in modeling. The aim of this study is to evaluate the model's performance for predicting Covid-19 patients with certain congenital disease that require ICU based on the results of blood tests laboratory and patient’s vital signs. This study used binary response variables, 1 if the patient was admitted to the ICU and 0 if the patient was not admitted to the ICU. The fixed effect predictor variables are the results of blood tests laboratory and patient’s vital signs. The random effect predictor variable is patient's congenital disease. The result showed that the average of accuracy and AUC from LASSO GLMM is more than the average of accuracy and AUC from LASSO GLM by using 5% level of significance. Respiratory rate and Lactate show a significance effect to predict the ICU needs of Covid-19 patients. The random effects patient's congenital disease has significance effect at 5% level of significance. It means that the ICU needs for Covid-19 patients varies among patient's congenital disease. We can conclude that GLMM LASSO with the random effect of patient’s congenital diseases has better modeling performance to predict the ICU needs of Covid-19 patients based on the results of blood tests laboratory and patient’s vital signs. The results of this modeling can quickly detect Covid-19 patients who need the ICU and can help medical staff use ICU resources optimally

Canonical correlation analysis in high dimensions with structured regularization.

Canonical correlation analysis (CCA) is a technique for measuring the association between two multivariate data matrices. A regularized modification of canonical correlation analysis (RCCA) which imposes an ℓ2 penalty on the CCA coefficients is widely used in applications with high-dimensional data. One limitation of such regularization is that it ignores any data structure, treating all the features equally, which can be ill-suited for some applications. In this article we introduce several approaches to regularizing CCA that take the underlying data structure into account. In particular, the proposed group regularized canonical correlation analysis (GRCCA) is useful when the variables are correlated in groups. We illustrate some computational strategies to avoid excessive computations with regularized CCA in high dimensions. We demonstrate the application of these methods in our motivating application from neuroscience, as well as in a small simulation example.

Transient stability assessment in large-scale power systems using sparse logistic classifiers

In this paper, the problem of transient stability assessment is formulated as a pattern recognition problem. The transient stability boundary (TSB) separates the region between the secure and unsecure operation conditions. In large-scale power networks, the TSB is a very high dimensional hyperplane. A modern machine learning method called the “sparse logistic classifier” is applied for finding the TSB. This approach combines the classical logistic classifier with a L1 penalty, and it inherently possesses the automatic feature reduction property desired for high-dimensional modeling. This methodology is demonstrated by a 470-bus power network, and compared with several competing methods recently applied in this field. These competing methods include the support vector machine (SVM) and the k-nearest neighbor (kNN) classifier, as well as the classical logistic classifier which is not equipped with the L1 design. Fit for high dimensional problems, our approach demonstrates superior predictive classification accuracy.

Machine learning approaches improve risk stratification for secondary cardiovascular disease prevention in multiethnic patients

ObjectivesIdentifying high-risk patients is crucial for effective cardiovascular disease (CVD) prevention. It is not known whether electronic health record (EHR)-based machine-learning (ML) models can improve CVD risk stratification compared with...

Shrinkage quantile regression for panel data with multiple structural breaks

We consider a shrinkage quantile regression model for high‐dimensional panel data with multiple structural breaks. The structural breaks are assumed to be common across all individuals, but may vary across different quantile levels while sharing an identical location shift effect. We impose an L1 penalty on the individual effects and an L1‐type fusion penalty to estimate both the slope coefficients and the structural breaks by combining information at multiple quantile levels. The proposed method can detect “partial” changes of the regression coefficients and consistently estimate both the number and dates of the breaks with probability tending to 1. We establish the asymptotic properties of the proposed regression coefficient estimators as well as their post‐selection counterparts, where the dimensionality of the covariates is allowed to diverge. Simulation results demonstrate that the proposed method works well in finite‐sample cases. Using the proposed method, we obtain many interesting results by analyzing a dataset concerning environmental Kuznets curves.

Optimizing for Interpretability in Deep Neural Networks with Tree Regularization

Deep models have advanced prediction in many domains, but their lack of interpretability remains a key barrier to the adoption in many real world applications. There exists a large body of work aiming to help humans understand these black box functions to varying levels of granularity – for example, through distillation, gradients, or adversarial examples. These methods however, all tackle interpretability as a separate process after training. In this work, we take a different approach and explicitly regularize deep models so that they are well-approximated by processes that humans can step through in little time. Specifically, we train several families of deep neural networks to resemble compact, axis-aligned decision trees without significant compromises in accuracy. The resulting axis-aligned decision functions uniquely make tree regularized models easy for humans to interpret. Moreover, for situations in which a single, global tree is a poor estimator, we introduce a regional tree regularizer that encourages the deep model to resemble a compact, axis-aligned decision tree in predefined, human-interpretable contexts. Using intuitive toy examples, benchmark image datasets, and medical tasks for patients in critical care and with HIV, we demonstrate that this new family of tree regularizers yield models that are easier for humans to simulate than L1 or L2 penalties without sacrificing predictive power.

Coordinating distributed MPC efficiently on a plantwide scale: The Lyapunov envelope algorithm

The model predictive control (MPC) of large-scale systems should adopt a distributed optimization approach, where controllers for the constituent subsystems optimize their control actions and iterations are used to coordinate their decisions. The real-time implementation of MPC, however, usually allows very limited time for computation and inevitably needs to be terminated early. In this work, we propose a splitting algorithm for distributed optimization analogous to forward-backward splitting (FBS), where ℓ1 and quadratic penalties are imposed on the violation of interconnecting relations among the subsystems. By designing the involved parameters based on dissipative analysis, the iterations result in the monotonic decrease of a plant-wide Lyapunov function, which we call Lyapunov envelope, thus maintaining closed-loop stability under distributed MPC despite early termination and yielding improving control performance as the allowed computational time or number of iterations increases. The proposed Lyapunov envelope algorithm is tested on an industrial-scale vinyl acetate monomer process.

Autophagy-related circRNA evaluation reveals hsa_circ_0001747 as a potential favorable prognostic factor for biochemical recurrence in patients with prostate cancer

Prostate cancer (PCa) is a common high-incidence malignancy in men, some of whom develop biochemical recurrence (BCR) in the advanced stage. However, there are currently no accurate prognostic indicators of BCR in PCa. The aim of our study was to identify an autophagy-related circular RNA prognostic factor of BCR for patients with PCa. In this study, immunochemistry revealed that the classic autophagy marker MAP1LC3B was positively correlated with Gleason score. Least absolute shrinkage and selector operator regression were conducted to develop a novel prognostic model with tenfold cross-validation and an L1 penalty. Five autophagy-related circRNA signatures were included in the prognostic model. Patients with PCa were ultimately divided into high- and low-risk groups, based on the median risk score. Patients with PCa, who had a high risk score, were more likely to develop BCR in a shorter period of time. Univariate and multivariate Cox regression analyses demonstrated that the risk score was an independent variable for predicting BCR in PCa. In addition, a prognostic nomogram integrated with the risk score and numerous clinicopathological parameters was developed to accurately predict 3- and 5-year BCR of patients with PCa. Finally, the hsa_circ_0001747 signature was selected for further experimental verification in vitro and in vivo, which showed that downregulated hsa_circ_0001747 might facilitate PCa via augmenting autophagy. Our findings indicate that the autophagy-related circRNA signature hsa_circ_0001747 may serve as a promising indicator for BCR prediction in patients with PCa.

A Novel Computational Framework to Predict Disease-Related Copy Number Variations by Integrating Multiple Data Sources.

Copy number variation (CNV) may contribute to the development of complex diseases. However, due to the complex mechanism of path association and the lack of sufficient samples, understanding the relationship between CNV and cancer remains a major challenge. The unprecedented abundance of CNV, gene, and disease label data provides us with an opportunity to design a new machine learning framework to predict potential disease-related CNVs. In this paper, we developed a novel machine learning approach, namely, IHI-BMLLR (Integrating Heterogeneous Information sources with Biweight Mid-correlation and L1-regularized Logistic Regression under stability selection), to predict the CNV-disease path associations by using a data set containing CNV, disease state labels, and gene data. CNVs, genes, and diseases are connected through edges and then constitute a biological association network. To construct a biological network, we first used a self-adaptive biweight mid-correlation (BM) formula to calculate correlation coefficients between CNVs and genes. Then, we used logistic regression with L1 penalty (LLR) function to detect genes related to disease. We added stability selection strategy, which can effectively reduce false positives, when using self-adaptive BM and LLR. Finally, a weighted path search algorithm was applied to find top D path associations and important CNVs. The experimental results on both simulation and prostate cancer data show that IHI-BMLLR is significantly better than two state-of-the-art CNV detection methods (i.e., CCRET and DPtest) under false-positive control. Furthermore, we applied IHI-BMLLR to prostate cancer data and found significant path associations. Three new cancer-related genes were discovered in the paths, and these genes need to be verified by biological research in the future.

Sample-Efficient L0-L2 Constrained Structure Learning of Sparse Ising Models

We consider the problem of learning the underlying graph of a sparse Ising model with p nodes from n i.i.d. samples. The most recent and best performing approaches combine an empirical loss (the logistic regression loss or the interaction screening loss) with a regularizer (an L1 penalty or an L1 constraint). This results in a convex problem that can be solved separately for each node of the graph. In this work, we leverage the cardinality constraint L0 norm, which is known to properly induce sparsity, and further combine it with an L2 norm to better model the non-zero coefficients. We show that our proposed estimators achieve an improved sample complexity, both (a) theoretically, by reaching new state-of-the-art upper bounds for recovery guarantees, and (b) empirically, by showing sharper phase transitions between poor and full recovery for graph topologies studied in the literature, when compared to their L1-based state-of-the-art methods.

Understanding Decoupled and Early Weight Decay

Weight decay (WD) is a traditional regularization technique in deep learning, but despite its ubiquity, its behavior is still an area of active research. Golatkar et al. have recently shown that WD only matters at the start of the training in computer vision, upending traditional wisdom. Loshchilov et al. show that for adaptive optimizers, manually decaying weights can outperform adding an l2 penalty to the loss. This technique has become increasingly popular and is referred to as decoupled WD. The goal of this paper is to investigate these two recent empirical observations. We demonstrate that by applying WD only at the start, the network norm stays small throughout training. This has a regularizing effect as the effective gradient updates become larger. However, traditional generalizations metrics fail to capture this effect of WD, and we show how a simple scale-invariant metric can. We also show how the growth of network weights is heavily influenced by the dataset and its generalization properties. For decoupled WD, we perform experiments in NLP and RL where adaptive optimizers are the norm. We demonstrate that the primary issue that decoupled WD alleviates is the mixing of gradients from the objective function and the l2 penalty in the buffers of Adam (which stores the estimates of the first-order moment). Adaptivity itself is not problematic and decoupled WD ensures that the gradients from the l2 term cannot "drown out" the true objective, facilitating easier hyperparameter tuning.

A Unified Primal Dual Active Set Algorithm for Nonconvex Sparse Recovery

In this paper, we consider the problem of recovering a sparse signal based on penalized least squares formulations. We develop a novel algorithm of primal-dual active set type for a class of nonconvex sparsity-promoting penalties, including ℓ0, bridge, smoothly clipped absolute deviation, capped ℓ1 and minimax concavity penalty. First, we establish the existence of a global minimizer for the related optimization problems. Then we derive a novel necessary optimality condition for the global minimizer using the associated thresholding operator. The solutions to the optimality system are coordinatewise minimizers, and under minor conditions, they are also local minimizers. Upon introducing the dual variable, the active set can be determined using the primal and dual variables together. Further, this relation lends itself to an iterative algorithm of active set type which at each step involves first updating the primal variable only on the active set and then updating the dual variable explicitly. When combined with a continuation strategy on the regularization parameter, the primal dual active set method is shown to converge globally to the underlying regression target under certain regularity conditions. Extensive numerical experiments with both simulated and real data demonstrate its superior performance in terms of computational efficiency and recovery accuracy compared with the existing sparse recovery methods.