Sparse Linear Model Research Articles

Genome-Wide Association Analysis and Genomic Prediction of Thyroglobulin Plasma Levels.

Thyroglobulin (Tg) is an iodoglycoprotein produced by thyroid follicular cells which acts as an essential substrate for thyroid hormone synthesis. To date, only one genome-wide association study (GWAS) of plasma Tg levels has been performed by our research group. Utilizing recent advancements in computation and modeling, we apply a Bayesian approach to the probabilistic inference of the genetic architecture of Tg. We fitted a Bayesian sparse linear mixed model (BSLMM) and a frequentist linear mixed model (LMM) of 7,289,083 variants in 1096 healthy European-ancestry participants of the Croatian Biobank. Meta-analysis with two independent cohorts (total n = 2109) identified 83 genome-wide significant single nucleotide polymorphisms (SNPs) within the ST6GAL1 gene (). BSLMM revealed additional association signals on chromosomes 1, 8, 10, and 14. For ST6GAL1 and the newly uncovered genes, we provide physiological and pathophysiological explanations of how their expression could be associated with variations in plasma Tg levels. We found that the SNP-heritability of Tg is 17% and that 52% of this variation is due to a small number of 16 variants that have a major effect on Tg levels. Our results suggest that the genetic architecture of plasma Tg is not polygenic, but influenced by a few genes with major effects.

Sparse latent factor regression models for genome-wide and epigenome-wide association studies.

Association of phenotypes or exposures with genomic and epigenomic data faces important statistical challenges. One of these challenges is to account for variation due to unobserved confounding factors, such as individual ancestry or cell-type composition in tissues. This issue can be addressed with penalized latent factor regression models, where penalties are introduced to cope with high dimension in the data. If a relatively small proportion of genomic or epigenomic markers correlate with the variable of interest, sparsity penalties may help to capture the relevant associations, but the improvement over non-sparse approaches has not been fully evaluated yet. Here, we present least-squares algorithms that jointly estimate effect sizes and confounding factors in sparse latent factor regression models. In simulated data, sparse latent factor regression models generally achieved higher statistical performance than other sparse methods, including the least absolute shrinkage and selection operator and a Bayesian sparse linear mixed model. In generative model simulations, statistical performance was slightly lower (while being comparable) to non-sparse methods, but in simulations based on empirical data, sparse latent factor regression models were more robust to departure from the model than the non-sparse approaches. We applied sparse latent factor regression models to a genome-wide association study of a flowering trait for the plant Arabidopsis thaliana and to an epigenome-wide association study of smoking status in pregnant women. For both applications, sparse latent factor regression models facilitated the estimation of non-null effect sizes while overcoming multiple testing issues. The results were not only consistent with previous discoveries, but they also pinpointed new genes with functional annotations relevant to each application.

Robust regression via error tolerance

Real-world datasets are often characterised by outliers; data items that do not follow the same structure as the rest of the data. These outliers might negatively influence modelling of the data. In data analysis it is, therefore, important to consider methods that are robust to outliers. In this paper we develop a robust regression method that finds the largest subset of data items that can be approximated using a sparse linear model to a given precision. We show that this can yield the best possible robustness to outliers. However, this problem is NP-hard and to solve it we present an efficient approximation algorithm, termed SLISE. Our method extends existing state-of-the-art robust regression methods, especially in terms of speed on high-dimensional datasets. We demonstrate our method by applying it to both synthetic and real-world regression problems.

Distributed adaptive Huber regression

Distributed data naturally arise in scenarios involving multiple sources of observations, each stored at a different location. Directly pooling all the data together is often prohibited due to limited bandwidth and storage, or due to privacy protocols. A new robust distributed algorithm is introduced for fitting linear regressions when data are subject to heavy-tailed and/or asymmetric errors with finite second moments. The algorithm only communicates gradient information at each iteration, and therefore is communication-efficient. To achieve the bias-robustness tradeoff, the key is a novel double-robustification approach that applies on both the local and global objective functions. Statistically, the resulting estimator achieves the centralized nonasymptotic error bound as if all the data were pooled together and came from a distribution with sub-Gaussian tails. Under a finite (2+δ)-th moment condition, a Berry-Esseen bound for the distributed estimator is established, based on which robust confidence intervals are constructed. In high dimensions, the proposed doubly-robustified loss function is complemented with ℓ1-penalization for fitting sparse linear models with distributed data. Numerical studies further confirm that compared with extant distributed methods, the proposed methods achieve near-optimal accuracy with low variability and better coverage with tighter confidence width.

Sparse linear mixed model selection via streamlined variational Bayes

Linear mixed models are a versatile statistical tool to study data by accounting for fixed effects and random effects from multiple sources of variability. In many situations, a large number of candidate fixed effects is available and it is of interest to select a parsimonious subset of those being effectively relevant for predicting the response variable. Variational approximations facilitate fast approximate Bayesian inference for the parameters of a variety of statistical models, including linear mixed models. However, for models having a high number of fixed or random effects, simple application of standard variational inference principles does not lead to fast approximate inference algorithms, due to the size of model design matrices and inefficient treatment of sparse matrix problems arising from the required approximating density parameters updates. We illustrate how recently developed streamlined variational inference procedures can be generalized to make fast and accurate inference for the parameters of linear mixed models with nested random effects and global-local priors for Bayesian fixed effects selection. Our variational inference algorithms achieve convergence to the same optima of their standard implementations, although with significantly lower computational effort, memory usage and time, especially for large numbers of random effects. Using simulated and real data examples, we assess the quality of automated procedures for fixed effects selection that are free from hyperparameters tuning and only rely upon variational posterior approximations. Moreover, we show high accuracy of variational approximations against model fitting via Markov Chain Monte Carlo sampling.

Estimation and group variable selection for additive partial linear models withwavelets and splines

In this paper we study sparse high dimensional additive partial linear models with nonparametric additive components of heterogeneous smoothness. We review several existing algorithms that have been developed for this problem in the recent literature, highlighting the connections between them, and present some computationally efficient algorithms for fitting such models. To achieve optimal rates in large sample situations we use hybrid P-splines and block wavelet penalisation techniques combined with adaptive (group) LASSO-like procedures for selecting the additive components in the nonparametric part of the models. Hence, the component selection and estimation in the nonparametric part may be viewed as a functional version of estimation and grouped variable selection. This allows to take advantage of several oracle results which yield asymptotic optimality of estimators in high-dimensional but sparse additive models. Numerical implementations of our procedures for proximal like algorithms are discussed. Large sample properties of the estimates and of the model selection are presented and the results are illustrated with simulated examples and a real data analysis. Keywords: Additive models, Backfitting, Penalisation, Proximal algorithms, Squared group-LASSO, Splines, Wavelets

Neural generators of sparse local linear models for achieving both accuracy and interpretability

For reliability, it is important for the predictions made by machine learning methods to be interpretable by humans. In general, deep neural networks (DNNs) can provide accurate predictions, although it is difficult to interpret why such predictions are obtained by the DNNs. On the other hand, interpretation of linear models is easy, although their predictive performance is low because real-world data are often intrinsically non-linear. To combine both the benefits of the high predictive performance of DNNs and the high interpretability of linear models into a single model, we propose neural generators of sparse local linear models (NGSLL). Sparse local linear models have high flexibility because they can approximate non-linear functions. NGSLL generates sparse linear weights for each sample using DNNs that take the original representations of each sample (e.g., word sequence) and their simplified representations (e.g., bag-of-words) as input. By extracting features from the original representations, the weights can contain rich information and achieve a high predictive performance. In addition, the prediction is interpretable because it is obtained through the inner product between the simplified representations and the sparse weights, where only a small number of weights are selected by our gate module in NGSLL. In experiments on image, text and tabular datasets, we demonstrate the effectiveness of NGSLL quantitatively and qualitatively by evaluating the prediction performance and visualizing generated weights.

An adaptive shortest-solution guided decimation approach to sparse high-dimensional linear regression

High-dimensional linear regression model is the most popular statistical model for high-dimensional data, but it is quite a challenging task to achieve a sparse set of regression coefficients. In this paper, we propose a simple heuristic algorithm to construct sparse high-dimensional linear regression models, which is adapted from the shortest-solution guided decimation algorithm and is referred to as ASSD. This algorithm constructs the support of regression coefficients under the guidance of the shortest least-squares solution of the recursively decimated linear models, and it applies an early-stopping criterion and a second-stage thresholding procedure to refine this support. Our extensive numerical results demonstrate that ASSD outperforms LASSO, adaptive LASSO, vector approximate message passing, and two other representative greedy algorithms in solution accuracy and robustness. ASSD is especially suitable for linear regression problems with highly correlated measurement matrices encountered in real-world applications.

Estimating dynamic cutting forces of machine tools from measured vibrations using sparse regression with nonlinear function basis

Estimating relationships between system inputs and outputs can provide insight to system characteristics. Furthermore, with an established input-output relationship and measured output, one can estimate the corresponding input to the system. Traditionally, the relationship between input and output can be represented with transfer functions or frequency response functions. However, those functions need to be built on physical parameters, which are hard to obtain in practical systems. Also, the reverse problem of solving for the input with a known/measured output is often more difficult to solve than the forward problem. This paper aims to explore the data-driven input-output relationship between system inputs and outputs for system diagnostics, prognostics, performance prediction, and control. A data-driven relationship can provide a new way for system input estimation or output prediction. In this paper, a sparse linear regression model with nonlinear function basis is proposed for input estimation with measured outputs. The proposed method explicitly creates a nonlinear function basis for the regression relationship. A threshold-based sparse linear regression is designed to ensure sparsity. The method is tested with experimental data from a spindle testbed that simulates cutting forces within machine tools. The results show that the proposed approach can predict the input force based on the measured vibration response with high accuracy. The current model is also compared with neural networks, which is another nonlinear regression method.

Abstract 13151: Shared Genetic Architecture Between Adiposity and Right Ventricular Structure

Introduction: Right ventricular (RV) enlargement has been associated with increased mortality, regardless of the etiology. Pathophysiologic determinants of RV structure, however, are under characterized. Examining genetic architecture underlying variation in RV size may identify etiologic risk factors contributing to RV enlargement. Therefore, we interrogated a polygenic predictor for RV size to identify etiologic drivers of RV structure. Methods: We studied 10,522 adult subjects of European ancestry in Vanderbilt University Medical Center’s de-identified DNA biobank (BioVU) who had undergone an echocardiogram including measurement of RV diameter. Bayesian Sparse Linear Mixed Modeling (GEMMA v0.95) was used to create a polygenic predictor of RV diameter. A PheWAS using logistic regression was then conducted to determine clinical diagnoses significantly associated with genetically-determined RV diameter. Further associations between identified clinical factors and RV diameter were examined using 2-sample bivariate and multivariable Mendelian randomization (MR). Results: PheWAS identified obesity-related clinical diagnoses as the most significantly associated with RV diameter (Figure 1). Univariate MR for body mass index (BMI), waist circumference, fat mass, and body impedance all demonstrated a causal and positive association with RV diameter. Multivariable MR demonstrated the association between measures of adiposity and RV diameter persisted even after accounting for the effect of pulmonary pressure. Conclusions: Genetic architecture is shared between adiposity and RV structure. Measures of adiposity are causally associated with genetically predicted larger RV diameter. Common pathophysiologic mechanisms may regulate adiposity and right heart structure, suggesting adiposity may represent an intervenable therapeutic target for RV failure.

Does SLOPE outperform bridge regression?

Abstract A recently proposed SLOPE estimator [6] has been shown to adaptively achieve the minimax $\ell _2$ estimation rate under high-dimensional sparse linear regression models [25]. Such minimax optimality holds in the regime where the sparsity level $k$, sample size $n$ and dimension $p$ satisfy $k/p\rightarrow 0, k\log p/n\rightarrow 0$. In this paper, we characterize the estimation error of SLOPE under the complementary regime where both $k$ and $n$ scale linearly with $p$, and provide new insights into the performance of SLOPE estimators. We first derive a concentration inequality for the finite sample mean square error (MSE) of SLOPE. The quantity that MSE concentrates around takes a complicated and implicit form. With delicate analysis of the quantity, we prove that among all SLOPE estimators, LASSO is optimal for estimating $k$-sparse parameter vectors that do not have tied nonzero components in the low noise scenario. On the other hand, in the large noise scenario, the family of SLOPE estimators are sub-optimal compared with bridge regression such as the Ridge estimator.

A nonlinear mixed–integer programming approach for variable selection in linear regression model

Modern statistical studies often encounter regression models with high dimensions in which the number of features p is greater than the sample size n. Although the theory of linear models is well–established for the traditional assumption p < n, making valid statistical inference in high dimensional cases is a considerable challenge. With recent advances in technologies, the problem appears in many biological, medical, social, industrial, and economic studies. As known, the LASSO method is a popular technique for variable selection/estimation in high dimensional sparse linear models. Here, we show that the prediction performance of the LASSO method can be improved by eliminating the structured noises through a mixed–integer programming approach. As a result of our analysis, a modified variable selection/estimation scheme is proposed for a high dimensional regression model which can be considered as an alternative of the LASSO method. Some numerical experiments are made on the classical riboflavin production and some simulated data sets to shed light on the practical performance of the suggested method.

Optimization of the Hybrid Movie Recommendation System Based on Weighted Classification and User Collaborative Filtering Algorithm

Aiming at the problem that the single model of the traditional recommendation system cannot accurately capture user preferences, this paper proposes a hybrid movie recommendation system and optimization method based on weighted classification and user collaborative filtering algorithm. The sparse linear model is used as the basic recommendation model, and the local recommendation model is trained based on user clustering, and the top-N personalized recommendation of movies is realized by fusion with the weighted classification model. According to the item category preference, the scoring matrix is converted into a low-dimensional, dense item category preference matrix, multiple cluster centers are obtained, the distance between the target user and each cluster center is calculated, and the target user is classified into the closest cluster. Finally, the collaborative filtering algorithm is used to predict the scores for the unrated items of the target user to form a recommendation list. The items are clustered through the item category preference, and the high-dimensional rating matrix is converted into a low-dimensional item category preference matrix, which further reduces the sparsity of the data. Experiments based on the Douban movie dataset verify that the recommendation algorithm proposed in this article solves the shortcomings of a single algorithm model to a certain extent and improves the recommendation effect.

A SHRINKAGE PRINCIPLE FOR HEAVY-TAILED DATA: HIGH-DIMENSIONAL ROBUST LOW-RANK MATRIX RECOVERY.

This paper introduces a simple principle for robust statistical inference via appropriate shrinkage on the data. This widens the scope of high-dimensional techniques, reducing the distributional conditions from sub-exponential or sub-Gaussian to more relaxed bounded second or fourth moment. As an illustration of this principle, we focus on robust estimation of the low-rank matrix Θ* from the trace regression model Y = Tr(Θ*⊤ X) + ϵ. It encompasses four popular problems: sparse linear model, compressed sensing, matrix completion and multi-task learning. We propose to apply the penalized least-squares approach to the appropriately truncated or shrunk data. Under only bounded 2+δ moment condition on the response, the proposed robust methodology yields an estimator that possesses the same statistical error rates as previous literature with sub-Gaussian errors. For sparse linear model and multi-task regression, we further allow the design to have only bounded fourth moment and obtain the same statistical rates. As a byproduct, we give a robust covariance estimator with concentration inequality and optimal rate of convergence in terms of the spectral norm, when the samples only bear bounded fourth moment. This result is of its own interest and importance. We reveal that under high dimensions, the sample covariance matrix is not optimal whereas our proposed robust covariance can achieve optimality. Extensive simulations are carried out to support the theories.

Strong selection consistency of Bayesian vector autoregressive models based on a pseudo-likelihood approach

Vector autoregressive (VAR) models aim to capture linear temporal interdependencies among multiple time series. They have been widely used in macroeconomics and financial econometrics and more recently have found novel applications in functional genomics and neuroscience. These applications have also accentuated the need to investigate the behavior of the VAR model in a high-dimensional regime, which will provide novel insights into the role of temporal dependence for regularized estimates of the models parameters. However, hardly anything is known regarding posterior model selection consistency for Bayesian VAR models in such regimes. In this work, we develop a pseudo-likelihood based Bayesian approach for consistent variable selection in high-dimensional VAR models by considering hierarchical normal priors on the autoregressive coefficients, as well as on the model space. We establish strong selection consistency of the proposed method, namely that the posterior probability assigned to the true underlying VAR model converges to one under high-dimensional scaling where the dimension p of the VAR system grows nearly exponentially with the sample size n. Further, the result is established under mild regularity conditions on the problem parameters. Finally, as a by-product of these results, we also establish strong selection consistency for the sparse high-dimensional linear regression model with serially correlated regressors and errors.

CpG content-dependent associations between transcription factors and histone modifications.

Understanding the factors that underlie the epigenetic regulation of genes is crucial to understand the gene regulatory machinery as a whole. Several experimental and computational studies examined the relationship between different factors involved. Here we investigate the relationship between transcription factors (TFs) and histone modifications (HMs), based on ChIP-seq data in cell lines. As it was shown that gene regulation by TFs differs depending on the CpG class of a promoter, we study the impact of the CpG content in promoters on the associations between TFs and HMs. We suggest an approach based on sparse linear regression models to infer associations between TFs and HMs with respect to CpG content. A study of the partial correlation of HMs for the two classes of high and low CpG content reveals possible CpG dependence and potential candidates for confounding factors in our models. We show that the models are accurate, inferred associations reflect known biological relationships, and we give new insight into associations with respect to CpG content. Moreover, analysis of a ChIP-seq dataset in HepG2 cells of the HM H3K122ac, an HM about little is known, reveals novel TF associations and supports a previously established link to active transcription.

MuSP: A multistep screening procedure for sparse recovery

We propose a multistep screening procedure (MuSP) for the recovery of sparse linear models in high‐dimensional data. This method is based on a repeated small penalty strategy that quickly converges to an estimate within a few iterations. Specifically, in each iteration, an adaptive lasso regression with a small penalty is fit within the reduced feature space obtained from the previous step, rendering its computational complexity roughly comparable with the Lasso. MuSP is shown to select the true model under complex correlation structures among the predictors and response, even when the irrepresentable condition fails. Further, under suitable regularity conditions, MuSP achieves the optimal minimax rate for the upper bound of l2‐norm error. Numerical comparisons show that the method works effectively both in model selection and estimation, and the MuSP fitted model is stable over a range of small tuning parameter values, eliminating the need to choose the tuning parameter by cross‐validation. We also apply MuSP to financial data and show that MuSP is successful in asset allocation selection.

Wetland Mapping Using HJ-1A/B Hyperspectral Images and an Adaptive Sparse Constrained Least Squares Linear Spectral Mixture Model

In this study, we proposed an adaptive sparse constrained least squares linear spectral mixture model (SCLS-LSMM) to map wetlands in a sophisticated scene. It includes three procedures: (1) estimating the abundance based on sparse constrained least squares method with all endmembers in the spectral library, (2) selecting “active” endmember combinations for each pixel based on the estimated abundances and (3) estimating abundances based on the linear spectral unmixing algorithm only with the adaptively selected endmember combinations. The performances of the proposed SCLS-LSMM on wetland vegetation communities mapping were compared with the traditional full constrained least squares linear spectral mixture model (FCLS-LSMM) using HJ-1A/B hyperspectral images. The accuracy assessment results showed that the proposed SCLS-LSMM obtained a significantly better performance with a systematic error (SE) of –0.014 and a root-mean-square error (RMSE) of 0.087 for Reed marsh, and a SE of 0.004 and a RMSE of 0.059 for Weedy meadow, compared with the traditional FCLS-LSMM. The proposed methods improved the unmixing accuracies of wetlands’ vegetation communities and have the potential to understand the process of wetlands’ degradation under the impacts of climate changes and permafrost degradation.

Downward-Looking Linear Array Three-Dimensional SAR Imaging Based on the Two-Dimensional Mismatch Compensation

For downward-looking linear array (DLLA) three-dimensional (3-D) synthetic aperture radar (SAR), it is necessary to realize the super-resolution in both azimuth and cross-track direction due to the limited lengths of the synthetic aperture and the linear array. As all the scatterers are assumed on the uniform grids, the cross-track super-resolution can be achieved by 1-D compressed sensing. In the real imaging system, however, the gridding error should be considered because the biased scatterers lead to the mismatch of the measurement matrix and affect the imaging performance. The 1-D mismatch in cross-track direction has been solved by atomic norm minimization and off-grid sparse Bayesian inference. With the development of the super-resolution methods, the 2-D super-resolution in both azimuth and cross-track direction is realized by the 2-D compressed sensing (CS) algorithms. To solve the 2-D mismatch problem, a novel 2-D mismatch compensation method for DLLA 3-D SAR is proposed. Instead of converting the 2-D matrix signals to the 1-D vectors, the proposed method directly processes the 2-D mismatch with 2-D joint model. Furthermore, the 2-D joint model with 2-D mismatch is simplified as a normal sparse linear model, which is suitable for most of the CS reconstruction algorithms. It can not only provide better reconstruction performance but also reduce the memory cost and computation load. Finally, the simulation experiments are shown to demonstrate the validity of the proposed method.

A Deep Neural Network for Accurate and Robust Prediction of the Glass Transition Temperature of Polyhydroxyalkanoate Homo- and Copolymers.

The purpose of this study was to develop a data-driven machine learning model to predict the performance properties of polyhydroxyalkanoates (PHAs), a group of biosourced polyesters featuring excellent performance, to guide future design and synthesis experiments. A deep neural network (DNN) machine learning model was built for predicting the glass transition temperature, Tg, of PHA homo- and copolymers. Molecular fingerprints were used to capture the structural and atomic information of PHA monomers. The other input variables included the molecular weight, the polydispersity index, and the percentage of each monomer in the homo- and copolymers. The results indicate that the DNN model achieves high accuracy in estimation of the glass transition temperature of PHAs. In addition, the symmetry of the DNN model is ensured by incorporating symmetry data in the training process. The DNN model achieved better performance than the support vector machine (SVD), a nonlinear ML model and least absolute shrinkage and selection operator (LASSO), a sparse linear regression model. The relative importance of factors affecting the DNN model prediction were analyzed. Sensitivity of the DNN model, including strategies to deal with missing data, were also investigated. Compared with commonly used machine learning models incorporating quantitative structure–property (QSPR) relationships, it does not require an explicit descriptor selection step but shows a comparable performance. The machine learning model framework can be readily extended to predict other properties.