High-dimensional Variable Selection Research Articles

Instrumental Variable Method for Regularized Estimation in Generalized Linear Measurement Error Models

Regularized regression methods have attracted much attention in the literature, mainly due to its application in high-dimensional variable selection problems. Most existing regularization methods assume that the predictors are directly observed and precisely measured. It is well known that in a low-dimensional regression model if some covariates are measured with error, then the naive estimators that ignore the measurement error are biased and inconsistent. However, the impact of measurement error in regularized estimation procedures is not clear. For example, it is known that the ordinary least squares estimate of the regression coefficient in a linear model is attenuated towards zero and, on the other hand, the variance of the observed surrogate predictor is inflated. Therefore, it is unclear how the interaction of these two factors affects the selection outcome. To correct for the measurement error effects, some researchers assume that the measurement error covariance matrix is known or can be estimated using external data. In this paper, we propose the regularized instrumental variable method for generalized linear measurement error models. We show that the proposed approach yields a consistent variable selection procedure and root-n consistent parameter estimators. Extensive finite sample simulation studies show that the proposed method performs satisfactorily in both linear and generalized linear models. A real data example is provided to further demonstrate the usage of the method.

Distributed non-convex regularization for generalized linear regression

Distributed penalized generalized linear regression algorithms have been widely studied in recent years. However, they all assume that the data should be randomly distributed. In real applications, this assumption is not necessarily true, since the whole data are often stored in a non-random manner. To tackle this issue, a non-convex penalized distributed pilot sample surrogate negative log-likelihood learning procedure is developed, which can realize distributed high-dimensional variable selection for generalized linear models, and be adaptive to the non-random situations. The established theoretical results and numerical studies all validate the proposed method.

High dimensional variable selection through group Lasso for multiple function‐on‐function linear regression: A case study in PM10 monitoring

SummaryAnalyzing the effect of chemical and local meteorological variables over the behaviour in concentrations in the Abruzzo region (Italy), with the objective of forecasting and controlling air quality, motivates the current work. Given that the available data are curves that represent the day‐to‐day variations, a multiple function‐on‐function linear regression (MFFLR) model is considered. By assuming the Karhunen‐Loève expansion, MFFLR model can be reduced to a classical linear regression model for each principal component of the functional response in terms of all principal components (PCs) of the functional predictors. In this sense, a regularization approach for functional principal component regression based on the merge of functional data analysis with group Lasso is proposed. This novel methodology allows to estimate the model and, simultaneously, select those relevant functional predictors with the functional response, where each functional independent variable is represented by a group of input variables derived by the PCs.

Variable selection and subgroup analysis for high-dimensional censored data

This paper proposes a penalized method for high-dimensional variable selection and subgroup identification in the Tobit model. Based on Olsen's [(1978). Note on the uniqueness of the maximum likelihood estimator for the Tobit model. Econometrica: Journal of the Econometric Society, 46(5), 1211–1215. https://doi.org/10.2307/1911445] convex reparameterization of the Tobit negative log-likelihood, we develop an efficient algorithm for minimizing the objective function by combining the alternating direction method of multipliers (ADMM) and generalised coordinate descent (GCD). We also establish the oracle properties of our proposed estimator under some mild regularity conditions. Furthermore, extensive simulations and an empirical data study are conducted to demonstrate the performance of the proposed approach.

High dimensional controlled variable selection with model-X knockoffs in the AFT model

High dimensional controlled variable selection with model-X knockoffs in the AFT model

A neutral comparison of algorithms to minimize L0 penalties for high-dimensional variable selection.

Variable selection methods based on L0 penalties have excellent theoretical properties to select sparse models in a high-dimensional setting. There exist modifications of the Bayesian Information Criterion (BIC) which either control the familywise error rate (mBIC) or the false discovery rate (mBIC2) in terms of which regressors are selected to enter a model. However, the minimization of L0 penalties comprises a mixed-integer problem which is known to be NP-hard and therefore becomes computationally challenging with increasing numbers of regressor variables. This is one reason why alternatives like the LASSO have become so popular, which involve convex optimization problems that are easier to solve. The last few years have seen some real progress in developing new algorithms to minimize L0 penalties. The aim of this article is to compare the performance of these algorithms in terms of minimizing L0 -based selection criteria. Simulation studies covering a wide range of scenarios that are inspired by genetic association studies are used to compare the values of selection criteria obtained with different algorithms. In addition, some statistical characteristics of the selected models and the runtime of algorithms are compared. Finally, the performance of the algorithms is illustrated in a real data example concerned with expression quantitative trait loci (eQTL) mapping.

Feature Screening for High-Dimensional Variable Selection in Generalized Linear Models.

The two-stage feature screening method for linear models applies dimension reduction at first stage to screen out nuisance features and dramatically reduce the dimension to a moderate size; at the second stage, penalized methods such as LASSO and SCAD could be applied for feature selection. A majority of subsequent works on the sure independent screening methods have focused mainly on the linear model. This motivates us to extend the independence screening method to generalized linear models, and particularly with binary response by using the point-biserial correlation. We develop a two-stage feature screening method called point-biserial sure independence screening (PB-SIS) for high-dimensional generalized linear models, aiming for high selection accuracy and low computational cost. We demonstrate that PB-SIS is a feature screening method with high efficiency. The PB-SIS method possesses the sure independence property under certain regularity conditions. A set of simulation studies are conducted and confirm the sure independence property and the accuracy and efficiency of PB-SIS. Finally we apply PB-SIS to one real data example to show its effectiveness.

High-Dimensional Variable Selection for Quantile Regression Based on Variational Bayesian Method

The quantile regression model is widely used in variable relationship research of moderate sized data, due to its strong robustness and more comprehensive description of response variable characteristics. With the increase of data size and data dimensions, there have been some studies on high-dimensional quantile regression under the classical statistical framework, including a high-efficiency frequency perspective; however, this comes at the cost of randomness quantification, or the use of a lower efficiency Bayesian method based on MCMC sampling. To overcome these problems, we propose high-dimensional quantile regression with a spike-and-slab lasso penalty based on variational Bayesian (VBSSLQR), which can, not only improve the computational efficiency, but also measure the randomness via variational distributions. Simulation studies and real data analysis illustrated that the proposed VBSSLQR method was superior or equivalent to other quantile and nonquantile regression methods (including Bayesian and non-Bayesian methods), and its efficiency was higher than any other method.

Identification of significant gene expression changes in multiple perturbation experiments using knockoffs.

Large-scale multiple perturbation experiments have the potential to reveal a more detailed understanding of the molecular pathways that respond to genetic and environmental changes. A key question in these studies is which gene expression changes are important for the response to the perturbation. This problem is challenging because (i) the functional form of the nonlinear relationship between gene expression and the perturbation is unknown and (ii) identification of the most important genes is a high-dimensional variable selection problem. To deal with these challenges, we present here a method based on the model-X knockoffs framework and Deep Neural Networks to identify significant gene expression changes in multiple perturbation experiments. This approach makes no assumptions on the functional form of the dependence between the responses and the perturbations and it enjoys finite sample false discovery rate control for the selected set of important gene expression responses. We apply this approach to the Library of Integrated Network-Based Cellular Signature data sets which is a National Institutes of Health Common Fund program that catalogs how human cells globally respond to chemical, genetic and disease perturbations. We identified important genes whose expression is directly modulated in response to perturbation with anthracycline, vorinostat, trichostatin-a, geldanamycin and sirolimus. We compare the set of important genes that respond to these small molecules to identify co-responsive pathways. Identification of which genes respond to specific perturbation stressors can provide better understanding of the underlying mechanisms of disease and advance the identification of new drug targets.

Bayesian Variable Selection in Generalized Extreme Value Regression: Modeling Annual Maximum Temperature

In many applications, interest focuses on assessing relationships between covariates and the extremes of the distribution of a continuous response. For example, in climate studies, a usual approach to assess climate change has been based on the analysis of annual maximum data. Using the generalized extreme value (GEV) distribution, we can model trends in the annual maximum temperature using the high number of available atmospheric covariates. However, there is typically uncertainty in which of the many candidate covariates should be included. Bayesian methods for variable selection are very useful to identify important covariates. However, such methods are currently very limited for moderately high dimensional variable selection in GEV regression. We propose a Bayesian method for variable selection based on a stochastic search variable selection (SSVS) algorithm proposed for posterior computation. The method is applied to the selection of atmospheric covariates in annual maximum temperature series in three Spanish stations.

Adaptive signal recovery with Subbotin noise

In this article, we study the problem of high-dimensional variable selection. Specifically, we consider a single vector X from a sparse sequence model with Subbotin noise and derive the regions of variable selection with respect to a Hamming loss function.

Potential application of elastic nets for shared polygenicity detection with adapted threshold selection.

Current research suggests that hundreds to thousands of single nucleotide polymorphisms (SNPs) with small to modest effect sizes contribute to the genetic basis of many disorders, a phenomenon labeled as polygenicity. Additionally, many such disorders demonstrate polygenic overlap, in which risk alleles are shared at associated genetic loci. A simple strategy to detect polygenic overlap between two phenotypes is based on rank-ordering the univariate p-values from two genome-wide association studies (GWASs). Although high-dimensional variable selection strategies such as Lasso and elastic nets have been utilized in other GWAS analysis settings, they are yet to be utilized for detecting shared polygenicity. In this paper, we illustrate how elastic nets, with polygenic scores as the dependent variable and with appropriate adaptation in selecting the penalty parameter, may be utilized for detecting a subset of SNPs involved in shared polygenicity. We provide theory to better understand our approaches, and illustrate their utility using synthetic datasets. Results from extensive simulations are presented comparing the elastic net approaches with the rank ordering approach, in various scenarios. Results from simulations studies exhibit one of the elastic net approaches to be superior when the correlations among the SNPs are high. Finally, we apply the methods on two real datasets to illustrate further the capabilities, limitations and differences among the methods.

High-Dimensional Propensity Score-Adjusted Case-Crossover for Discovering Adverse Drug Reactions from Computerized Administrative Healthcare Databases.

Increasing availability of medico-administrative databases has prompted the development of automated pharmacovigilance signal detection methodologies. Self-controlled approaches have recently been proposed. They account for time-independent confounding factors that may not be recorded. So far, large numbers of drugs have been screened either univariately or with LASSO penalized regressions. We propose and assess a new method that combines the case-crossover self-controlled design with propensity scores (propensity score-adjusted case-crossover) built from high-dimensional data-driven variable selection, to account for co-medications or possibly other measured confounders. Comparison with the univariate and LASSO case-crossover was performed from simulations and a real-data study. Multiple regressions (LASSO, propensity score-adjusted case-crossover) accounted for co-medications and no other covariates. For the univariate and propensity score-adjusted case-crossover methods, the detection threshold was based on a false discovery rate procedure, while for LASSO, it relied on the Akaike Information Criterion. For the real-data study, two drug safety experts evaluated the signals generated from the analysis of 4099 patients with acute myocardial infarction from the French national health database. On simulations, our approach ranked the signals similarly to the LASSO and better than the univariate method while controlling the false discovery rate at the prespecified level, contrary to the univariate method. The LASSO provided the best sensitivity at the cost of larger false discovery rate estimates. On the application, our approach showed similar performances to the LASSO and better performances than the univariate method. It highlighted 43 signals out of 609 drug candidates: 22 (51%) were considered as potentially pharmacologically relevant, including seven (16%) regarded as highly relevant. Our findings show the interest of a propensity score combined with a case-crossover for pharmacovigilance. They also confirm that indication bias remains a challenge when mining medico-administrative databases.

Actual versus ‘natural’ rates of suicide: Evidence from the USA

This paper employs a high dimensional variable selection technique to select a subset of suicide determinants from 167 potential factors, which are then used to estimate ‘natural’ suicide rates for US states by least squares dummy variables. Over the period 2005–2017, all states are found to have a non-zero and positive natural suicide rate, below their respective actual average rate. Higher actual rates suggest deterioration in socioeconomic conditions, inaccessible and unaffordable mental health care for certain sections of the population and inadequate implementation of measures to identify and reduce suicidal mortality. Evidence-informed policies aiming for zero suicide target could draw inspiration from exemplar states to direct resources towards states with greater relative differences between actual average and natural suicide rates.

Regional heterogeneity and U.S. presidential elections: Real-time 2020 forecasts and evaluation

This paper exploits cross-sectional variation at the level of U.S. counties to generate real-time forecasts for the 2020 U.S. presidential election. The forecasting models are trained on data covering the period 2000–2016, using high-dimensional variable selection techniques. Our county-based approach contrasts the literature that focuses on national and state level data but uses longer time periods to train their models. The paper reports forecasts of popular and electoral college vote outcomes and provides a detailed ex-post evaluation of the forecasts released in real time before the election. It is shown that all of these forecasts outperform autoregressive benchmarks. A pooled national model using One-Covariate-at-a-time-Multiple-Testing (OCMT) variable selection significantly outperformed all models in forecasting the U.S. mainland national vote share and electoral college outcomes (forecasting 236 electoral votes for the Republican party compared to 232 realized). This paper also shows that key determinants of voting outcomes at the county level include incumbency effects, unemployment, poverty, educational attainment, house price changes, and international competitiveness. The results are also supportive of myopic voting: economic fluctuations realized a few months before the election tend to be more powerful predictors of voting outcomes than their long-horizon analogs.

Randomized boosting with multivariable base-learners for high-dimensional variable selection and prediction

BackgroundStatistical boosting is a computational approach to select and estimate interpretable prediction models for high-dimensional biomedical data, leading to implicit regularization and variable selection when combined with early stopping. Traditionally, the set of base-learners is fixed for all iterations and consists of simple regression learners including only one predictor variable at a time. Furthermore, the number of iterations is typically tuned by optimizing the predictive performance, leading to models which often include unnecessarily large numbers of noise variables.ResultsWe propose three consecutive extensions of classical component-wise gradient boosting. In the first extension, called Subspace Boosting (SubBoost), base-learners can consist of several variables, allowing for multivariable updates in a single iteration. To compensate for the larger flexibility, the ultimate selection of base-learners is based on information criteria leading to an automatic stopping of the algorithm. As the second extension, Random Subspace Boosting (RSubBoost) additionally includes a random preselection of base-learners in each iteration, enabling the scalability to high-dimensional data. In a third extension, called Adaptive Subspace Boosting (AdaSubBoost), an adaptive random preselection of base-learners is considered, focusing on base-learners which have proven to be predictive in previous iterations. Simulation results show that the multivariable updates in the three subspace algorithms are particularly beneficial in cases of high correlations among signal covariates. In several biomedical applications the proposed algorithms tend to yield sparser models than classical statistical boosting, while showing a very competitive predictive performance also compared to penalized regression approaches like the (relaxed) lasso and the elastic net.ConclusionsThe proposed randomized boosting approaches with multivariable base-learners are promising extensions of statistical boosting, particularly suited for highly-correlated and sparse high-dimensional settings. The incorporated selection of base-learners via information criteria induces automatic stopping of the algorithms, promoting sparser and more interpretable prediction models.

Data Analysis for Gansoline Octane Problem with High Dimensional Variable Selection

With the progress and development of science and technology, high dimensional data has become a hot topic in scientific research. Lasso is one of the most commonly used methods for solving high dimensional variable selection so far. In the process of performing Lasso solution, the least angle regression algorithm and coordinate descent algorithm are often used. In this paper, Lasso is applied to reduce the dimensionality of the gasoline octane problem, and three algorithms are used to compare, and it is found that the mean square error of the model obtained by using the generalized path search algorithm is the smallest.

Comparative Effectiveness of Sodium-Glucose Cotransporter 2 Inhibitors vs Sulfonylureas in Patients With Type 2 Diabetes

In the treatment of type 2 diabetes, evidence of the comparative effectiveness of sodium-glucose cotransporter 2 (SGLT2) inhibitors vs sulfonylureas-the second most widely used antihyperglycemic class after metformin-is lacking. To evaluate the comparative effectiveness of SGLT2 inhibitors and sulfonylureas associated with the risk of all-cause mortality among patients with type 2 diabetes using metformin. A cohort study used data from the US Department of Veterans Affairs compared the use of SGLT2 inhibitors vs sulfonylureas in individuals receiving metformin for treatment of type 2 diabetes. A total of 23 870 individuals with new use of SGLT2 inhibitors and 104 423 individuals with new use of sulfonylureas were enrolled between October 1, 2016, and February 29, 2020, and followed up until January 31, 2021. New use of SGLT2 inhibitors or sulfonylureas. This study examined the outcome of all-cause mortality. Predefined variables and covariates identified by a high-dimensional variable selection algorithm were used to build propensity scores. The overlap weighting method based on the propensity scores was used to estimate the intention-to-treat effect sizes of SGLT2 inhibitor compared with sulfonylurea therapy. The inverse probability of the treatment adherence weighting method was used to estimate the per-protocol effect sizes. Among the 128 293 participants (mean [SD] age, 64.60 [9.84] years; 122 096 [95.17%] men), 23 870 received an SGLT2 inhibitor and 104 423 received a sulfonylurea. Compared with sulfonylureas, SGLT2 inhibitors were associated with reduced risk of all-cause mortality (hazard ratio [HR], 0.81; 95% CI, 0.75-0.87), yielding an event rate difference of -5.15 (95% CI, -7.16 to -3.02) deaths per 1000 person-years. Compared with sulfonylureas, SGLT2 inhibitors were associated with a reduced risk of death, regardless of cardiovascular disease status, in several categories of estimated glomerular filtration rate (including rates from >90 to ≤30 mL/min/1.73 m2) and in participants with no albuminuria (albumin to creatinine ratio [ACR] ≤30 mg/g), microalbuminuria (ACR >30 to ≤300 mg/g), and macroalbuminuria (ACR >300 mg/g). In per-protocol analyses, continued use of SGLT2 inhibitors was associated with a reduced risk of death compared with continued use of sulfonylureas (HR, 0.66; 95% CI, 0.60-0.74; event rate difference, -10.10; 95% CI, -12.97 to -7.24 deaths per 1000 person-years). In additional per-protocol analyses, continued use of SGLT2 inhibitors with metformin was associated with a reduced risk of death compared with SGLT2 inhibitor treatment without metformin (HR, 0.70; 95% CI, 0.50-0.97; event rate difference, -7.62; 95% CI, -17.12 to -0.48 deaths per 1000 person-years). In this comparative effectiveness study analyzing data from the US Department of Veterans Affairs, among patients with type 2 diabetes receiving metformin therapy, SGLT2 inhibitor treatment was associated with a reduced risk of all-cause mortality compared with sulfonylureas. The results provide data from a real-world setting that might help guide the choice of antihyperglycemic therapy.

Error Bound of Mode-Based Additive Models.

Due to their flexibility and interpretability, additive models are powerful tools for high-dimensional mean regression and variable selection. However, the least-squares loss-based mean regression models suffer from sensitivity to non-Gaussian noises, and there is also a need to improve the model’s robustness. This paper considers the estimation and variable selection via modal regression in reproducing kernel Hilbert spaces (RKHSs). Based on the mode-induced metric and two-fold Lasso-type regularizer, we proposed a sparse modal regression algorithm and gave the excess generalization error. The experimental results demonstrated the effectiveness of the proposed model.

Improved two-stage model averaging for high-dimensional linear regression, with application to Riboflavin data analysis

BackgroundModel averaging has attracted increasing attention in recent years for the analysis of high-dimensional data. By weighting several competing statistical models suitably, model averaging attempts to achieve stable and improved prediction. In this paper, we develop a two-stage model averaging procedure to enhance accuracy and stability in prediction for high-dimensional linear regression. First we employ a high-dimensional variable selection method such as LASSO to screen redundant predictors and construct a class of candidate models, then we apply the jackknife cross-validation to optimize model weights for averaging.ResultsIn simulation studies, the proposed technique outperforms commonly used alternative methods under high-dimensional regression setting, in terms of minimizing the mean of the squared prediction error. We apply the proposed method to a riboflavin data, the result show that such method is quite efficient in forecasting the riboflavin production rate, when there are thousands of genes and only tens of subjects.ConclusionsCompared with a recent high-dimensional model averaging procedure (Ando and Li in J Am Stat Assoc 109:254–65, 2014), the proposed approach enjoys three appealing features thus has better predictive performance: (1) More suitable methods are applied for model constructing and weighting. (2) Computational flexibility is retained since each candidate model and its corresponding weight are determined in the low-dimensional setting and the quadratic programming is utilized in the cross-validation. (3) Model selection and averaging are combined in the procedure thus it makes full use of the strengths of both techniques. As a consequence, the proposed method can achieve stable and accurate predictions in high-dimensional linear models, and can greatly help practical researchers analyze genetic data in medical research.