Ultrahigh Dimensional Feature Research Articles

Variation of conditional mean and its application in ultrahigh dimensional feature screening

. A new metric, called variation of conditional mean (VCM), is proposed to measure the dependence of conditional mean of a response variable on a predictor variable. The VCM has several appealing merits. It equals zero if and only if the conditional mean of the response is independent of the predictor; it can be used for both real vector-valued variables and functional data. An estimator of the VCM is given through kernel smoothing, and a test for the conditional mean independence based on the estimated VCM is constructed. The limit distributions of the test statistic under the null hypothesis and alternative hypothesis are deduced, respectively. We further use VCM as a marginal utility to do high-dimensional feature screening to screen out variables that do not contribute to the conditional mean of the response given the predictors and prove the validity of the sure screening property. Furthermore, we find the cross variation of conditional mean (CVCM), a variant of the VCM, has a faster convergence rate than the VCM under conditional mean independence. Numerical comparison shows that the VCM and CVCM performs well in both conditional independence testing and feature screening. We also illustrate their applications to real data sets.

Comparison of leaf chlorophyll content retrieval performance of citrus using FOD and CWT methods with field-based full-spectrum hyperspectral reflectance data

Citrus is one of the most economically valuable fruit trees in the world, for which leaf chlorophyll content (LCC) serves as a crucial indicator for evaluating its growth and health status. However, the quantitative estimation of LCC using remote-sensing techniques is still challenging owing to unclear sensitive spectral ranges, baseline drift and overlapping spectrum peaks. To resolve these issues, we clarified the spectral response characteristics of citrus LCC using fractional-order derivatives (FOD) and continuous wavelet transform (CWT) methods to determine its sensitive spectral range with in situ full-spectrum leaf hyperspectral data. We proposed a novel method for estimating the LCC of citrus by combining an ensemble learning regression model based on Hyperopt optimization (H-ELR) with partial least squares regression (PLSR) using the ultra-high dimensional feature variables produced by dual- and tri-band combination strategies. We evaluated the retrieval performance of LCC between the FOD- and CWT-based optimal spectral feature variables. Besides, we further examined the feasibility of improving the estimation accuracy of LCC by the combination of their optimal feature variables. Finally, we evaluated the effect of the spectral curve trend changes and different dimensional spectral features on LCC estimation. The results showed that: (1) The FOD- and CWT-based methods improved the correlation between original spectral reflectance and LCC, with the correlation coefficient increasing by 0.046 and 0.054, respectively. We confirmed that 425–740 nm is the optimal spectral range for LCC estimation. (2) We found that the 0.9 order derivative Tri-band index (TBI2 (R571, R1697, R740)) constructed by combining the sensitive spectral bands of leaf water content achieved a high-precision LCC estimation (R2 = 0.876). In addition, the MERIS terrestrial chlorophyll index (MTCI) constructed based on the scale6 of CWT also gained good retrieval accuracy (R2 = 0.806). (3) We demonstrated that a combination of FOD-based and CWT-based sensitive spectral features improves estimation accuracies (R2 = 0.891) of LCC and revealed that the reflectance peaks and slope peaks at 550 and 750 nm are essential variables for predicting the citrus LCC. (4) The combination of PLSR and H-ELR model provided a good retrieval performance of citrus LCC with the kurtosis (γ = 3.2) and skewness (Sk = 0.066) of the residual prediction values. Our proposed method can provide a scientific basis for estimating LCC and other physiological parameters of citrus and other crop types, which is also important to optimize agricultural management practices and improve crop yield.

Support vector machine in ultrahigh-dimensional feature space

Classification and feature selection play an important role in knowledge discovery in high-dimensional data. Although penalized Support Vector Machine (SVM) is among the most powerful methods for classification and automatic feature selection in high-dimensional feature space, it is not directly applicable to ultrahigh-dimensional cases, wherein the number of features far exceeds the sample size. In this paper, we suggest an efficient two-step method for simultaneous classification and identifying important features in the setting of ultrahigh-dimensional models. Specifically, we first develop an independence screening procedure to reduce the dimensionality of the feature space to a moderate scale, and then penalized support vector machine is applied to the dimension-reduced feature space to select important features further and estimate the coefficients, via a (penalized) model fit. Implementation of the suggested two-step method is not limited by the dimensionality of the models and entails much less computational cost. Numerical examples and a real data analysis are used to demonstrate the finite sample performance of our proposal.

A fast asynchronous Markov chain Monte Carlo sampler for sparse Bayesian inference

Abstract We propose a very fast approximate Markov chain Monte Carlo sampling framework that is applicable to a large class of sparse Bayesian inference problems. The computational cost per iteration in several regression models is of order O(n(s+J)), where n is the sample size, s is the underlying sparsity of the model, and J is the size of a randomly selected subset of regressors. This cost can be further reduced by data sub-sampling when stochastic gradient Langevin dynamics are employed. The algorithm is an extension of the asynchronous Gibbs sampler of Johnson et al. [(2013). Analyzing Hogwild parallel Gaussian Gibbs sampling. In Proceedings of the 26th International Conference on Neural Information Processing Systems (NIPS’13) (Vol. 2, pp. 2715–2723)], but can be viewed from a statistical perspective as a form of Bayesian iterated sure independent screening [Fan, J., Samworth, R., & Wu, Y. (2009). Ultrahigh dimensional feature selection: Beyond the linear model. Journal of Machine Learning Research, 10, 2013–2038]. We show that in high-dimensional linear regression problems, the Markov chain generated by the proposed algorithm admits an invariant distribution that recovers correctly the main signal with high probability under some statistical assumptions. Furthermore, we show that its mixing time is at most linear in the number of regressors. We illustrate the algorithm with several models.

Ultra-High Dimensional Feature Selection and Mean Estimation under Missing at Random

Ultra-High Dimensional Feature Selection and Mean Estimation under Missing at Random

A screening method for ultra-high dimensional features with overlapped partition structures.

Ultra-high dimensional data, such as gene and neuroimaging data, are becoming increasingly important in biomedical science. Identifying important biomarkers from the huge number of features can help us gain better insights into further researches. Variable screening is an efficient tool to achieve this goal under the large scale cases, which reduces the dimension of features into a moderate size by removing the major part of inactive ones. Developing novel variable screening methods for high-dimensional features with group structures is challenging, especially under the overlapped cases. For example, the huge-scaled genes usually can be partitioned into hundreds of pathways according to background knowledge. One primary characteristic for this type of data is that many genes may appear across more than one pathway, which means that different pathways are overlapped. However, existing variable screening methods only could deal with disjoint group structure cases. To fill this gap, we propose a novel variable screening method for the generalized linear model by incorporating overlapped partition structures with theoretical guarantee. Besides the sure screening property, we also test the performance of the proposed method through a series of numerical studies and apply it to statistical analysis of a breast cancer data.

LARGE-SCALE MULTIVARIATE SPARSE REGRESSION WITH APPLICATIONS TO UK BIOBANK.

In high-dimensional regression problems, often a relatively small subset of the features are relevant for predicting the outcome, and methods that impose sparsity on the solution are popular. When multiple correlated outcomes are available (multitask), reduced rank regression is an effective way to borrow strength and capture latent structures that underlie the data. Our proposal is motivated by the UK Biobank population-based cohort study, where we are faced with large-scale, ultrahigh-dimensional features, and have access to a large number of outcomes (phenotypes)-lifestyle measures, biomarkers, and disease outcomes. We are hence led to fit sparse reduced-rank regression models, using computational strategies that allow us to scale to problems of this size. We use a scheme that alternates between solving the sparse regression problem and solving the reduced rank decomposition. For the sparse regression component we propose a scalable iterative algorithm based on adaptive screening that leverages the sparsity assumption and enables us to focus on solving much smaller subproblems. The full solution is reconstructed and tested via an optimality condition to make sure it is a valid solution for the original problem. We further extend the method to cope with practical issues, such as the inclusion of confounding variables and imputation of missing values among the phenotypes. Experiments on both synthetic data and the UK Biobank data demonstrate the effectiveness of the method and the algorithm. We present multiSnpnet package, available at http://github.com/junyangq/multiSnpnet that works on top of PLINK2 files, which we anticipate to be a valuable tool for generating polygenic risk scores from human genetic studies.

Overlapping group screening for detection of gene-environment interactions with application to TCGA high-dimensional survival genomic data

BackgroundIn the context of biomedical and epidemiological research, gene-environment (G-E) interaction is of great significance to the etiology and progression of many complex diseases. In high-dimensional genetic data, two general models, marginal and joint models, are proposed to identify important interaction factors. Most existing approaches for identifying G-E interactions are limited owing to the lack of robustness to outliers/contamination in response and predictor data. In particular, right-censored survival outcomes make the associated feature screening even challenging. In this article, we utilize the overlapping group screening (OGS) approach to select important G-E interactions related to clinical survival outcomes by incorporating the gene pathway information under a joint modeling framework.ResultsSimulation studies under various scenarios are carried out to compare the performances of our proposed method with some commonly used methods. In the real data applications, we use our proposed method to identify G-E interactions related to the clinical survival outcomes of patients with head and neck squamous cell carcinoma, and esophageal carcinoma in The Cancer Genome Atlas clinical survival genetic data, and further establish corresponding survival prediction models. Both simulation and real data studies show that our method performs well and outperforms existing methods in the G-E interaction selection, effect estimation, and survival prediction accuracy.ConclusionsThe OGS approach is useful for selecting important environmental factors, genes and G-E interactions in the ultra-high dimensional feature space. The prediction ability of OGS with the Lasso penalty is better than existing methods. The same idea of the OGS approach can apply to other outcome models, such as the proportional odds survival time model, the logistic regression model for binary outcomes, and the multinomial logistic regression model for multi-class outcomes.

A Hyperparameter-Free, Fast and Efficient Framework to Detect Clusters From Limited Samples Based on Ultra High-Dimensional Features.

Clustering is a challenging problem in machine learning in which one attempts to group N objects into K0 groups based on P features measured on each object. In this article, we examine the case where N ≪ P and K0 is not known. Clustering in such high dimensional, small sample size settings has numerous applications in biology, medicine, the social sciences, clinical trials, and other scientific and experimental fields. Whereas most existing clustering algorithms either require the number of clusters to be known a priori or are sensitive to the choice of tuning parameters, our method does not require the prior specification of K0 or any tuning parameters. This represents an important advantage for our method because training data are not available in the applications we consider (i.e., in unsupervised learning problems). Without training data, estimating K0 and other hyperparameters-and thus applying alternative clustering algorithms-can be difficult and lead to inaccurate results. Our method is based on a simple transformation of the Gram matrix and application of the strong law of large numbers to the transformed matrix. If the correlation between features decays as the number of features grows, we show that the transformed feature vectors concentrate tightly around their respective cluster expectations in a low-dimensional space. This result simplifies the detection and visualization of the unknown cluster configuration. We illustrate the algorithm by applying it to 32 benchmarked microarray datasets, each containing thousands of genomic features measured on a relatively small number of tissue samples. Compared to 21 other commonly used clustering methods, we find that the proposed algorithm is faster and twice as accurate in determining the "best" cluster configuration.

Feature Screening for Massive Data Analysis by Subsampling

Modern statistical analysis often encounters massive datasets with ultrahigh-dimensional features. In this work, we develop a subsampling approach for feature screening with massive datasets. The approach is implemented by repeated subsampling of massive data and can be used for analyzing tasks with memory constraints. To conduct the procedure, we first calculate an R-squared screening measure (and related sample moments) based on subsamples. Second, we consider three methods to combine the local statistics. In addition to the simple average method, we design a jackknife debiased screening measure and an aggregated moment screening measure. Both approaches reduce the bias of the subsampling screening measure and therefore increase the accuracy of the feature screening. Last, we consider a novel sequential sampling method, that is more computationally efficient than the traditional random sampling method. The theoretical properties of the three screening measures under both sampling schemes are rigorously discussed. Finally, we illustrate the usefulness of the proposed method with an airline dataset containing 32.7 million records.

Interaction Identification and Clique Screening for Classification with Ultra-high Dimensional Discrete Features

Interaction Identification and Clique Screening for Classification with Ultra-high Dimensional Discrete Features

VIF-Regression Screening Ultrahigh Dimensional Feature Space

Iterative Sure Independent Screening (ISIS) was proposed for the problem of variable selection with ultrahigh dimensional feature space. Unfortunately, the ISIS method transforms the dimensionality of features from ultrahigh to ultra-low and may result in un-reliable inference when the number of important variables particularly is greater than the screening threshold. The proposed method has transformed the ultrahigh dimensionality of features to high dimension space in order to remedy of losing some information by ISIS method. The proposed method is compared with ISIS method by using real data and simulation. The results show this method is more efficient and more reliable than ISIS method.

Copula-based Partial Correlation Screening: a Joint and Robust Approach

Screening for ultrahigh dimensional features may encounter complicated issues such as outlying observations, heteroscedasticity or heavy-tailed distribution, multi-collinearity and confounding effects. Standard correlation-based marginal screening methods may be a weak solution to these issues. We contribute a novel robust joint screener to safeguard against outliers and distribution mis-specification for both the response variable and the covariates, and to account for external variables at the screening step. Specifically, we introduce a copula-based partial correlation (CPC) screener. We show that the empirical process of the estimated CPC converges weakly to a Gaussian process and establish the sure screening property for CPC screener under very mild technical conditions, where we need not require any moment condition, weaker than existing alternatives in the literature. Moreover, our approach allows for a diverging number of conditional variables from the theoretical point of view. Extensive simulation studies and two data applications are included to illustrate our proposal.

Sequential Feature Screening for Generalized Linear Models with Sparse Ultra-High Dimensional Data

This paper considers the iterative sequential lasso (ISLasso) variable selection for generalized linear model with ultrahigh dimensional feature space. The ISLasso selects features by estimated parameter sequentially iteratively for the second order approximation of likelihood function where the features selected depend on regulatory parameters. The procedure stops when extended BIC (EBIC) reaches a minimum. Simulation study demonstrates that the new method is a desirable approach over other methods.

Weighted Mean Squared Deviation Feature Screening for Binary Features.

In this study, we propose a novel model-free feature screening method for ultrahigh dimensional binary features of binary classification, called weighted mean squared deviation (WMSD). Compared to Chi-square statistic and mutual information, WMSD provides more opportunities to the binary features with probabilities near 0.5. In addition, the asymptotic properties of the proposed method are theoretically investigated under the assumption . The number of features is practically selected by a Pearson correlation coefficient method according to the property of power-law distribution. Lastly, an empirical study of Chinese text classification illustrates that the proposed method performs well when the dimension of selected features is relatively small.

Joint model-free feature screening for ultra-high dimensional semi-competing risks data

High-dimensional semi-competing risks data consisting of two probably correlated events, namely terminal event and non-terminal event, arise commonly in many biomedical studies. However, the corresponding statistical analysis is rarely investigated. A joint model-free feature screening procedure for both terminal and non-terminal events is proposed, which could allow the associated covariates to be in an ultra-high dimensional feature space. The joint screening utility is constructed from distance correlation between each predictor’s survival function and joint survival function of terminal and non-terminal events. Under rather mild technical assumptions, it is demonstrated that the proposed joint feature screening procedure enjoys sure screening and consistency in ranking properties. An adaptive threshold rule is further suggested to simultaneously identify important covariates and determine number of these covariates. Extensive numerical studies are conducted to examine the finite-sample performance of the proposed methods. Lastly, the suggested joint feature screening procedure is illustrated through a real example.

Fully-connected LSTM–CRF on medical concept extraction

Patient symptoms, test results, and treatment information have been taking down in extensive electronic records. Specifically, the named entity recognition of these medical concepts has high application value in the clinical field. However, due to issues like patient privacy, labeled data is expensive and difficult to find. In 2010, the i2b2/VA Natural Language Processing Challenge started a conceptual extraction task for electronic medical records. One of the task requirements is to classify natural language descriptions as corresponding concept types. In this paper, we proposed a new fully-connected LSTM network, while the LSTM + CRF model is used as the framework to test the effects of various LSTM structures. The real-data experiments demonstrate that the proposed fully-connected LSTM outperforms many of the mainstream LSTM structures in the quantitative evaluation. It is confirmed that the multi-layer bidirectional fully-connected LSTM cooperates with the character level word vector and the pre-trained word embedding, which achieves similar performance compared with the state-of-the-art methods, avoiding the using of prior knowledge data and ultra-high dimensional feature representation. Moreover, this end-to-end training method saves a lot of feature engineering work and storage spaces.

Ultrahigh dimensional feature screening for additive model with multivariate response

ABSTRACTWe consider feature screening for ultrahigh dimensional additive model with multivariate response in this paper. A new method named generalized correlation based projection screening is proposed by using generalized correlation between each predictor and multivariate response. The sure screening and ranking consistency properties are established under some regularized conditions for the proposed procedure. In addition, we construct an iterative version of the proposed screening procedure to enhance the finite sample screening performance. Both simulation studies and the real data analysis demonstrate that the proposed method works effectively.

Building generalized linear models with ultrahigh dimensional features: A sequentially conditional approach.

Conditional screening approaches have emerged as a powerful alternative to the commonly used marginal screening, as they can identify marginally weak but conditionally important variables. However, most existing conditional screening methods need to fix the initial conditioning set, which may determine the ultimately selected variables. If the conditioning set is not properly chosen, the methods may produce false negatives and positives. Moreover, screening approaches typically need to involve tuning parameters and extra modeling steps in order to reach a final model. We propose a sequential conditioning approach by dynamically updating the conditioning set with an iterative selection process. We provide its theoretical properties under the framework of generalized linear models. Powered by an extended Bayesian information criterion as the stopping rule, the method will lead to a final model without the need to choose tuning parameters or threshold parameters. The practical utility of the proposed method is examined via extensive simulations and analysis of a real clinical study on predicting multiple myeloma patients' response to treatment based on their genomic profiles.

Multiclass linear discriminant analysis with ultrahigh-dimensional features.

Within the framework of Fisher's discriminant analysis, we propose a multiclass classification method which embeds variable screening for ultrahigh-dimensional predictors. Leveraging interfeature correlations, we show that the proposed linear classifier recovers informative features with probability tending to one and can asymptotically achieve a zero misclassification rate. We evaluate the finite sample performance of the method via extensive simulations and use this method to classify posttransplantation rejection types based on patients' gene expressions.