Independence Screening Research Articles

Ultra-high dimensional variable screening via Gram–Schmidt orthogonalization

Independence screening procedure plays a vital role in variable selection when the number of variables is massive. However, high dimensionality of the data may bring in many challenges, such as multicollinearity or high correlation (possibly spurious) between the covariates, which results in marginal correlation being unreliable as a measure of association between the covariates and the response. We propose a novel and simple screening procedure called Gram–Schmidt screening (GSS) by integrating the classical Gram–Schmidt orthogonalization and the sure independence screening technique, which takes into account high correlations between the covariates in a data-driven way. GSS could successfully discriminate between the relevant and the irrelevant variables to achieve a high true positive rate without including many irrelevant and redundant variables, which offers a new perspective for screening method when the covariates are highly correlated. The practical performance of GSS was shown by comparative simulation studies and analysis of two real datasets.

A consistent variable screening procedure with family-wise error control

ABSTRACTVariable selection in ultra-high-dimensional data analysis is of critical importance in diverse scientific fields. There are many methods proposed for this problem in the past few decades. However, no attention has been paid on the model size produced by screening procedures and the difficulty of determining the screening threshold for practical use, which is key to real applications. Specifically, the pre-defined model size in a screening procedure lacks theoretical support, and users have no idea about the true positives and false positives in the final results. To solve this problem, we developed a new consistent independence screening procedure called HHG-CIS. The screening threshold of HHG-CIS can be set to explicitly control the family-wise error rate, thus it is easy to use in practice. Furthermore, HHG-CIS is model-free, robust to outliers, and outperforms other methods in models with interactions. We showed these advantages of HHG-CIS through extensive simulations and a real data application.

Combination of Ensembles of Regularized Regression Models with Resampling-Based Lasso Feature Selection in High Dimensional Data

In high-dimensional data, the performances of various classifiers are largely dependent on the selection of important features. Most of the individual classifiers with the existing feature selection (FS) methods do not perform well for highly correlated data. Obtaining important features using the FS method and selecting the best performing classifier is a challenging task in high throughput data. In this article, we propose a combination of resampling-based least absolute shrinkage and selection operator (LASSO) feature selection (RLFS) and ensembles of regularized regression (ERRM) capable of dealing data with the high correlation structures. The ERRM boosts the prediction accuracy with the top-ranked features obtained from RLFS. The RLFS utilizes the lasso penalty with sure independence screening (SIS) condition to select the top k ranked features. The ERRM includes five individual penalty based classifiers: LASSO, adaptive LASSO (ALASSO), elastic net (ENET), smoothly clipped absolute deviations (SCAD), and minimax concave penalty (MCP). It was built on the idea of bagging and rank aggregation. Upon performing simulation studies and applying to smokers’ cancer gene expression data, we demonstrated that the proposed combination of ERRM with RLFS achieved superior performance of accuracy and geometric mean.

High‐dimensional variable screening under multicollinearity

Variable screening is of fundamental importance in linear regression models when the number of predictors far exceeds the number of observations. Multicollinearity is a common phenomenon in high‐dimensional settings, in which two or more predictor variables are highly correlated, leading to the notorious difficulty for high‐dimensional variable screening. Sure independence screening (SIS) procedure can greatly reduce the dimensionality, but it may break down when the predictors are highly correlated. By combing the factor modelling with SIS, the profiled independence screening (PIS) approach was proposed. However, under a spiked population model, the profiled predictors could not be guaranteed to be uncorrelated and PIS may therefore be misleading. Instead of assuming either the predictors are uncorrelated as in SIS or the profiled predictors are uncorrelated as in PIS, a more general and challenging scenario is considered in which the predictors can be highly correlated. A so‐called preconditioned PIS (PPIS) method is proposed that produces asymptotically uncorrelated profiled predictors and thus leads to consistent model selection results under a spiked population model. Compared with PIS, the proposed method could handle the complex multicollinearity case, such as a spiked population model with a slow spectrum decay of population covariance matrix, while keeping the calculation simple. The promising performance of the proposed PPIS method will be illustrated via extensive simulation studies and two real examples.

Group feature screening via the F statistic

Feature screening is crucial in the analysis of ultrahigh dimensional data, where the number of variables (features) is in an exponential order of the number of observations. In various ultrahigh dimensional data, variables are naturally grouped, giving us a good rationale to develop a screening method using joint effect of multiple variables. In this article, we propose a group screening procedure via the F-test statistic. The proposed method is a direct extension of the original sure independence screening procedure, when the group information is known, for example, from prior knowledge. Under certain regularity conditions, we prove that the proposed group screening procedure possesses the sure screening property that selects all effective groups with a probability approaching one at an exponential rate. We use simulations to demonstrate the advantages of the proposed method and show its application in a genome-wide association study. We conclude that the grouping method is very useful in the analysis of ultrahigh dimensional data, as the optimal F-test can detect true signals with desired properties.

Testing Mediation Effects in High-Dimensional Epigenetic Studies.

Mediation analysis has been a powerful tool to identify factors mediating the association between exposure variables and outcomes. It has been applied to various genomic applications with the hope to gain novel insights into the underlying mechanism of various diseases. Given the high-dimensional nature of epigenetic data, recent effort on epigenetic mediation analysis is to first reduce the data dimension by applying high-dimensional variable selection techniques, then conducting testing in a low dimensional setup. In this paper, we propose to assess the mediation effect by adopting a high-dimensional testing procedure which can produce unbiased estimates of the regression coefficients and can properly handle correlations between variables. When the data dimension is ultra-high, we first reduce the data dimension from ultra-high to high by adopting a sure independence screening (SIS) method. We apply the method to two high-dimensional epigenetic studies: one is to assess how DNA methylations mediate the association between alcohol consumption and epithelial ovarian cancer (EOC) status; the other one is to assess how methylation signatures mediate the association between childhood maltreatment and post-traumatic stress disorder (PTSD) in adulthood. We compare the performance of the method with its counterpart via simulation studies. Our method can be applied to other high-dimensional mediation studies where high-dimensional mediation variables are collected.

Feature screening for ultrahigh-dimensional additive logistic models

This paper introduces a sure screening method for ultrahigh-dimensional additive logistic models. With binary response variable, additive logistic model is very useful in social and biological research such as disease detection. The proposed feature screening procedure, ALNIS (nonparametric independence screening for additive logistic models), employs B-spline approximation to model the marginal effect, transforming nonparametric problems into parametric ones. This screening process ranks the nonparametric components according to their norms of the marginal likelihood estimate. Under appropriate conditions, the proposed method is shown to possess sure screening property with a vanishing false selection rate. In numerical studies, we use simulated data to compare the performance of the proposed approach with other seven methods that allow the existence of binary response. We further illustrate the proposed procedure by a real data analysis. Numerical comparison indicates that the proposed approach enjoys robustness and effectiveness under ultrahigh-dimensional additive logistic models.

Feature Screening for Ultrahigh-dimensional Censored Data with Varying Coefficient Single-index Model

In this paper, we study the sure independence screening of ultrahigh-dimensional censored data with varying coefficient single-index model. This general model framework covers a large number of commonly used survival models. The property that the proposed method is not derived for a specific model is appealing in ultrahigh dimensional regressions, as it is difficult to specify a correct model for ultrahigh dimensional predictors. Once the assuming data generating process does not meet the actual one, the screening method based on the model will be problematic. We establish the sure screening property and consistency in ranking property of the proposed method. Simulations are conducted to study the finite sample performances, and the results demonstrate that the proposed method is competitive compared with the existing methods. We also illustrate the results via the analysis of data from The National Alzheimers Coordinating Center (NACC).

Conditional distance correlation sure independence screening for ultra-high dimensional survival data

In this article, we propose a new conditional feature screening procedure for ultra-high dimensional survival data via the conditional distance correlation. Compared with the existing methods, the proposed conditional feature screening approach has two key advantages. First, it is model-free and thus robust to model misspecification. Second, it is robust to heavy tails or extreme values in both of the covaraites and response. The sure screening property of suggested means is well established under rather mild assumptions. Simulation studies are carried out to examine and compare the performance of the advised procedure with its competitors, while a real data example is analyzed to illustrate the proposed approach.

Variable screening with multiple studies

Advancement in technology has generated abundant high-dimensional data that allows integration of multiple relevant studies. Due to their huge computational advantage, variable screening methods based on marginal correlation have become promising alternatives to the popular regularization methods for variable selection. However, all these screening methods are limited to single study so far. In this paper, we consider a general framework for variable screening with multiple related studies, and further propose a novel two-step screening procedure using a self-normalized estimator for high-dimensional regression analysis in this framework. Compared to the one-step procedure and rank-based sure independence screening (SIS) procedure, our procedure greatly reduces false negative errors while keeping a low false positive rate. Theoretically, we show that our procedure possesses the sure screening property with weaker assumptions on signal strengths and allows the number of features to grow at an exponential rate of the sample size. In addition, we relax the commonly used normality assumption and allow sub-Gaussian distributions. Simulations and a real transcriptomic application illustrate the advantage of our method as compared to the rank-based SIS method.

Single-Locus and Multi-Locus Genome-Wide Association Studies for Intramuscular Fat in Duroc Pigs.

Intramuscular fat (IMF) is an important quantitative trait of meat, which affects the associated sensory properties and nutritional value of pork. To gain a better understanding of the genetic determinants of IMF, we used a composite strategy, including single-locus and multi-locus association analyses to perform genome-wide association studies (GWAS) for IMF in 1,490 Duroc boars. We estimated the genomic heritability of IMF to be 0.23 ± 0.04. A total of 30 single nucleotide polymorphisms (SNPs) were found to be significantly associated with IMF. The single-locus mixed linear model (MLM) and multiple-locus methods multi-locus random-SNP-effect mixed linear model (mrMLM), fast multi-locus random-SNP-effect efficient mixed model association (FASTmrEMMA), and integrative sure independence screening expectation maximization Bayesian least absolute shrinkage and selection operator model (ISIS EM-BLASSO) analyses identified 5, 9, 8, and 21 significant SNPs, respectively. Interestingly, a novel quantitative trait locus (QTL) on SSC 7 was found to affect IMF. In addition, 10 candidate genes (BDKRB2, GTF2IRD1, UTRN, TMEM138, DPYD, CASQ2, ZNF518B, S1PR1, GPC6, and GLI1) were found to be associated with IMF based on their potential functional roles in IMF. GO analysis showed that most of the genes were involved in muscle and organ development. A significantly enriched KEGG pathway, the sphingolipid signaling pathway, was reported to be associated with fat deposition and obesity. Identification of novel variants and functional genes will advance our understanding of the genetic mechanisms of IMF and provide specific opportunities for marker-assisted or genomic selection in pigs. In general, such a composite single-locus and multi-locus strategy for GWAS may be useful for understanding the genetic architecture of economic traits in livestock.

Sure independence screening in the presence of missing data

Variable selection in ultra-high dimensional data sets is an increasingly prevalent issue with the readily available data arising from, for example, genome-wide associations studies or gene expression data. When the dimension of the feature space is exponentially larger than the sample size, it is desirable to screen out unimportant predictors in order to bring the dimension down to a moderate scale. In this paper we consider the case when observations of the predictors are missing at random. We propose performing screening using the marginal linear correlation coefficient between each predictor and the response variable accounting for the missing data using maximum likelihood estimation. This method is shown to have the sure screening property. Moreover, a novel method of screening that uses additional predictors when estimating the correlation coefficient is proposed. Simulations show that simply performing screening using pairwise complete observations is out-performed by both the proposed methods and is not recommended. Finally, the proposed methods are applied to a gene expression study on prostate cancer.

Two-Stage Classification with SIS Using a New Filter Ranking Method in High Throughput Data

Over the last decade, high dimensional data have been popularly paid attention to in bioinformatics. These data increase the likelihood of detecting the most promising novel information. However, there are limitations of high-performance computing and overfitting issues. To overcome the issues, alternative strategies need to be explored for the detection of true important features. A two-stage approach, filtering and variable selection steps, has been receiving attention. Filtering methods are divided into two categories of individual ranking and feature subset selection methods. Both have issues with the lack of consideration for joint correlation among features and computing time of an NP-hard problem. Therefore, we proposed a new filter ranking method (PF) using the elastic net penalty with sure independence screening (SIS) based on resampling technique to overcome these issues. We demonstrated that SIS-LASSO, SIS-MCP, and SIS-SCAD with the proposed filtering method achieved superior performance of not only accuracy, AUROC, and geometric mean but also true positive detection compared to those with the marginal maximum likelihood ranking method (MMLR) through extensive simulation studies. In addition, we applied it in a real application of colon and lung cancer gene expression data to investigate the classification performance and power of detecting true genes associated with colon and lung cancer.

L2RM: Low-Rank Linear Regression Models for High-Dimensional Matrix Responses

The aim of this article is to develop a low-rank linear regression model to correlate a high-dimensional response matrix with a high-dimensional vector of covariates when coefficient matrices have low-rank structures. We propose a fast and efficient screening procedure based on the spectral norm of each coefficient matrix to deal with the case when the number of covariates is extremely large. We develop an efficient estimation procedure based on the trace norm regularization, which explicitly imposes the low rank structure of coefficient matrices. When both the dimension of response matrix and that of covariate vector diverge at the exponential order of the sample size, we investigate the sure independence screening property under some mild conditions. We also systematically investigate some theoretical properties of our estimation procedure including estimation consistency, rank consistency, and nonasymptotic error bound under some mild conditions. We further establish a theoretical guarantee for the overall solution of our two-step screening and estimation procedure. We examine the finite-sample performance of our screening and estimation methods using simulations and a large-scale imaging genetic dataset collected by the Philadelphia Neurodevelopmental Cohort study. Supplementary materials for this article are available online.

PREDICTION OF TREATMENT OUTCOME FOR AUTISM FROM STRUCTURE OF THE BRAIN BASED ON SURE INDEPENDENCE SCREENING.

Autism spectrum disorder (ASD) is a complex neurodevelopmental disorder, and behavioral treatment interventions have shown promise for young children with ASD. However, there is limited progress in understanding the effect of each type of treatment. In this project, we aim to detect structural changes in the brain after treatment and select structural features associated with treatment outcomes. The difficulty in building large databases of patients who have received specific treatments and the high dimensionality of medical image analysis problems are the challenges in this work. To select predictive features and build accurate models, we use the sure independence screening (SIS) method. SIS is a theoretically and empirically validated method for ultra-high dimensional general linear models, and it achieves both predictive accuracy and correct feature selection by iterative feature selection. Compared with step-wise feature selection methods, SIS removes multiple features in each iteration and is computationally efficient. Compared with other linear models such as elastic-net regression, support vector regression (SVR) and partial least squares regression (PSLR), SIS achieves higher accuracy. We validated the superior performance of SIS in various experiments: First, we extract brain structural features from FreeSurfer, including cortical thickness, surface area, mean curvature and cortical volume. Next, we predict different measures of treatment outcomes based on structural features. We show that SIS achieves the highest correlation between prediction and measurements in all tasks. Furthermore, we report regions selected by SIS as biomarkers for ASD.

A modified mean-variance feature-screening procedure for ultrahigh-dimensional discriminant analysis

Cui et al. (2015) proposed a mean–variance feature-screening method based on the index MV(X|Y). By modifying MV(X|Y) with a weight function, a new index AD(X,Y) is introduced to measure the dependence between X and Y, and a corresponding feature-screening procedure called Anderson–Darling sure independence screening (AD-SIS) is proposed for ultrahigh-dimensional discriminant analysis. The sure screening and ranking consistency properties are established under mild conditions. It is shown that AD-SIS is model free with no specification of model structure and can be applied to multi-classification. Furthermore, AD-SIS is robust against heavy-tailed distributions. As such, it can be used to identify the tail difference for the covariate’s distribution. The finite-sample performance of AD-SIS is assessed by simulation and real data analysis. The results show that, compared with existing methods, AD-SIS can be more competitive for feature screening for ultrahigh-dimensional discriminant analysis.

ExSIS: Extended sure independence screening for ultrahigh-dimensional linear models

Statistical inference can be computationally prohibitive in ultrahigh-dimensional linear models. Correlation-based variable screening, in which one leverages marginal correlations for removal of irrelevant variables from the model prior to statistical inference, can be used to overcome this challenge. Prior works on correlation-based variable screening either impose statistical priors on the linear model or assume specific post-screening inference methods. This paper first extends the analysis of correlation-based variable screening to arbitrary linear models and post-screening inference techniques. In particular, (i) it shows that a condition---termed the screening condition---is sufficient for successful correlation-based screening of linear models, and (ii) it provides insights into the dependence of marginal correlation-based screening on different problem parameters. Numerical experiments confirm that these insights are not mere artifacts of analysis; rather, they are reflective of the challenges associated with marginal correlation-based variable screening. Second, the paper explicitly derives the screening condition for arbitrary (random or deterministic) linear models and, in the process, it establishes that---under appropriate conditions---it is possible to reduce the dimension of an ultrahigh-dimensional, arbitrary linear model to almost the sample size even when the number of active variables scales almost linearly with the sample size. Third, it specializes the screening condition to sub-Gaussian linear models and contrasts the final results to those existing in the literature. This specialization formally validates the claim that the main result of this paper generalizes existing ones on correlation-based screening.

Ensemble Learning for Power Systems TTC Prediction With Wind Farms

Being aware of the reliable margin of vital tie-lines, acting on the connection of power exporting area and power importing area, is significant to power systems. However, the high penetration of wind power causes fast variation of boundary limit parameters such as the available amount of power that can be transferred on the tie-lines, namely, total transfer capability (TTC), which may result in the inaccurate security assessment. Unfortunately, the traditional optimal power flow-based TTC model has computation burden for online applications. To address this problem, computational efficiency is improved via a data-driven TTC predictor based on an ensemble learning architecture in this paper. In the first stage, a daily profiles-based method including probabilistic sampling is proposed to simulate plenty of operation scenarios as data samples for ensemble training. Then, a hybrid feature selection approach, which is composed of the maximal information coefficient and nonparametric independence screening, is applied to determine the most correlative features to the objective variable. To enable the TTC predictor with high accuracy and generalization ability, a novel ensemble learning scheme for TTC predictor is constituted through clustering few adaptive hierarchical GA-based neural networks (AHGA-NNs predictor). At last, a modified New England test system is used to validate the proposed methodology. The results illustrate that combining with the appropriate feature selection, the presented ensemble learning has high performance on creating the accurate TTC predictor, which enables online secure margin monitoring for the vital tie-lines.

Online TTC Estimation Using Nonparametric Analytics Considering Wind Power Integration

Total transfer capability (TTC) is an effective indicator to evaluate the transmission limit of the interconnected systems. However, due to the large-scale wind power integration, operation conditions of a power system may change rapidly, yielding time-varying characteristics of the TTC. As a result, the traditional time-consuming transient stability constrained TTC model is unable to assess the online transmission margin. In this paper, we propose an online measurement-based TTC estimator using the nonparametric analytics. It consists of three major components: the probabilistic data generation, the composite feature selection, and the group Lasso regression-based training scheme. Specifically, we present a probabilistic data generation approach to take into account the uncertainties of the day-ahead generation scheduling and to reduce the number of redundant or infeasible data. Then, the composite feature selection is used to reduce the dimension of the generated data and identify the features which are highly correlated with TTC. The features are determined by the maximal information coefficients and nonparametric independence screening approach. Finally, these selected features are trained by the group Lasso regression to learn the correlation between the TTC and the online measurements. Once real-time measurements are available, the TTC can be assessed immediately through the learned correlation relationship. Extensive numerical results carried out on the modified New England 39-bus test system demonstrate the feasibility of the proposed TTC estimator for online applications.

Robust variable screening for regression using factor profiling

Sure independence screening is a fast procedure for variable selection in ultra‐high dimensional regression analysis. Unfortunately, its performance greatly deteriorates with increasing dependence among the predictors. To solve this issue, Factor Profiled Sure Independence Screening (FPSIS) models the correlation structure of the predictor variables, assuming that it can be represented by a few latent factors. The correlations can then be profiled out by projecting the data onto the orthogonal complement of the subspace spanned by these factors. However, neither of these methods can handle the presence of outliers in the data. Therefore, we propose a robust screening method which uses a least trimmed squares method to estimate the latent factors and the factor‐profiled variables. Variable screening is then performed on factor profiled variables by using regression MM‐estimators. Different types of outliers in this model and their roles in variable screening are studied. Both simulation studies and a real data analysis show that the proposed robust procedure has good performance on clean data and outperforms the two nonrobust methods on contaminated data.