High-dimensional Predictors Research Articles

Nonconcave penalized inverse regression in single-index models with high dimensional predictors

In this paper we aim to estimate the direction in general single-index models and to select important variables simultaneously when a diverging number of predictors are involved in regressions. Towards this end, we propose the nonconcave penalized inverse regression method. Specifically, the resulting estimation with the SCAD penalty enjoys an oracle property in semi-parametric models even when the dimension, p n , of predictors goes to infinity. Under regularity conditions we also achieve the asymptotic normality when the dimension of predictor vector goes to infinity at the rate of p n = o ( n 1 / 3 ) where n is sample size, which enables us to construct confidence interval/region for the estimated index. The asymptotic results are augmented by simulations, and illustrated by analysis of an air pollution dataset.

High-dimensional pseudo-logistic regression and classification with applications to gene expression data

High dimension low sample size data, like the microarray gene expression levels, pose numerous challenges to conventional statistical methods. In the particular case of binary classification, some classification methods, such as the support vector machine (SVM), can efficiently deal with high-dimensional predictors, but lacks the accuracy in estimating the probability of membership of a class. In contrast, the traditional logistic regression (TLR) effectively estimates the probability of class membership for data with low-dimensional inputs, but does not handle high-dimensional cases. The study bridges the gap between SVM and TLR by their loss functions. Based on the proposed new loss function, a pseudo-logistic regression and classification approach which simultaneously combines the strengths of both SVM and TLR is also proposed. Simulation evaluations and real data applications demonstrate that for low-dimensional data, the proposed method produces regression estimates comparable to those of TLR and penalized logistic regression, and that for high-dimensional data, the new method possesses higher classification accuracy than SVM and, in the meanwhile, enjoys enhanced computational convergence and stability.

Fourier Methods for Estimating the Central Subspace and the Central Mean Subspace in Regression

In regression with a high-dimensional predictor vector, it is important to estimate the central and central mean subspaces that preserve sufficient information about the response and the mean response. Using the Fourier transform, we have derived the candidate matrices whose column spaces recover the central and central mean subspaces exhaustively. Under the normality assumption of the predictors, explicit estimates of the central and central mean subspaces are derived. Bootstrap procedures are used for determining dimensionality and choosing tuning parameters. Simulation results and an application to a real data are reported. Our methods demonstrate competitive performance compared with SIR, SAVE, and other existing methods. The approach proposed in the article provides a novel view on sufficient dimension reduction and may lead to more powerful tools in the future.

Validation in Genomics: CpG Island Methylation Revisited

In a recent article in PLoS Genetics, Bock et al., (2006) undertake an extensive computational epigenetics analysis of the ability of DNA sequence-derived features, capturing attributes such as tetramer frequencies, repeats and predicted structure, to predict the methylation status of CpG islands. Their suite of analyses appears highly rigorous with regard to accompanying validation procedures, employing stringent Bonferroni corrections, stratified cross-validation, and follow-up experimental verification. Here, however, we showcase concerns with the validation steps, in part ascribable to the genome scale of the investigation, that serve as a cautionary note and indicate the heightened need for careful selection of analytic and companion validation methods. A series of new analyses of the same CpG island methylation data helps illustrate these issues, not just for this particular study, but also analogous investigations involving high-dimensional predictors with complex between-feature dependencies.

High-dimensional data: p >> n in mathematical statistics and bio-medical applications

High-dimensional data: p >> n in mathematical statistics and bio-medical applications

Partially Supervised Learning Using an EM‐Boosting Algorithm

Training data in a supervised learning problem consist of the class label and its potential predictors for a set of observations. Constructing effective classifiers from training data is the goal of supervised learning. In biomedical sciences and other scientific applications, class labels may be subject to errors. We consider a setting where there are two classes but observations with labels corresponding to one of the classes may in fact be mislabeled. The application concerns the use of protein mass-spectrometry data to discriminate between serum samples from cancer and noncancer patients. The patients in the training set are classified on the basis of tissue biopsy. Although biopsy is 100% specific in the sense that a tissue that shows itself to have malignant cells is certainly cancer, it is less than 100% sensitive. Reference gold standards that are subject to this special type of misclassification due to imperfect diagnosis certainty arise in many fields. We consider the development of a supervised learning algorithm under these conditions and refer to it as partially supervised learning. Boosting is a supervised learning algorithm geared toward high-dimensional predictor data, such as those generated in protein mass-spectrometry. We propose a modification of the boosting algorithm for partially supervised learning. The proposal is to view the true class membership of the samples that are labeled with the error-prone class label as missing data, and apply an algorithm related to the EM algorithm for minimization of a loss function. To assess the usefulness of the proposed method, we artificially mislabeled a subset of samples and applied the original and EM-modified boosting (EM-Boost) algorithms for comparison. Notable improvements in misclassification rates are observed with EM-Boost.

Boosting With the L2 Loss

This article investigates a computationally simple variant of boosting, L2Boost, which is constructed from a functional gradient descent algorithm with the L2-loss function. Like other boosting algorithms, L2Boost uses many times in an iterative fashion a prechosen fitting method, called the learner. Based on the explicit expression of refitting of residuals of L2Boost, the case with (symmetric) linear learners is studied in detail in both regression and classification. In particular, with the boosting iteration m working as the smoothing or regularization parameter, a new exponential bias-variance trade-off is found with the variance (complexity) term increasing very slowly as m tends to infinity. When the learner is a smoothing spline, an optimal rate of convergence result holds for both regression and classification and the boosted smoothing spline even adapts to higher-order, unknown smoothness. Moreover, a simple expansion of a (smoothed) 0–1 loss function is derived to reveal the importance of the decision boundary, bias reduction, and impossibility of an additive bias-variance decomposition in classification. Finally, simulation and real dataset results are obtained to demonstrate the attractiveness of L2Boost. In particular, we demonstrate that L2Boosting with a novel component-wise cubic smoothing spline is both practical and effective in the presence of high-dimensional predictors.

Pre-processing of high-dimensional categorical predictors in classification settings

Models in industrial applications can encounter categorical predictors with a large number of categories (hundreds or thousands). An example is the lot identifier of product in semiconductor manufacturing. Such variables represent a serious problem for practically all modern classification techniques. The goal is an efficient, computationally fast way to discover a small number of natural partitions of values for such variables that have similar statistical properties in terms of categorical response. Such partitions (interesting by itself) can be used then as an input to standard learning algorithms, such as decision trees, support vector machines, etc. The proposed approach introduces a data transformation on derived sparse frequency tables. Application of even simplest non-hierarchical metric clustering method to the transformed coordinates shows significant improvement both in speed and quality of partition in comparison to currently used methods.

Testing additivity by kernel-based methods - what is a reasonable test?

In the common nonparametric regression model with high dimensional predictor several tests for the hypothesis of an additive regression are investigated. The corresponding test statistics are either based on the diiferences between a fit under the assumption of additivity and a fit in the general model or based on residuals under the assumption of additivity. For all tests asymptotic normality is established under the null hypothesis of additivity and under fixed alternatives with different rates of convergence corresponding to both cases. These results are used for a comparison of the different methods. It is demonstrated that a statistic based on an empirical L1 - distance of the Nadaraya Watson and the marginal integration estimator yields the asymptotically most efficient procedure if these are compared with respect to the asymptotic behaviour under fixed alternatives.

Testing Additivity by Kernel-Based Methods: What Is a Reasonable Test?

In the common nonparametric regression model with high dimensional predictor several tests for the hypothesis of an additive regression are investigated. The corresponding test statistics are either based on the diiferences between a fit under the assumption of additivity and a fit in the general model or based on residuals under the assumption of additivity. For all tests asymptotic normality is established under the null hypothesis of additivity and under fixed alternatives with different rates of convergence corresponding to both cases. These results are used for a comparison of the different methods. It is demonstrated that a statistic based on an empirical L1 - distance of the Nadaraya Watson and the marginal integration estimator yields the asymptotically most efficient procedure if these are compared with respect to the asymptotic behaviour under fixed alternatives.

Additive nonparametric regression on principal components

Nonparametric regression smoothing in high dimensions faces the problem of data sparseness. Additive regression models alleviate this problem by fitting a sum of one-dimensional smooth functions. A common approach for dimension reduction in multivariate statistics is to replace the original high dimensional predictor variables by their dominant principal components. In this paper we consider an additive nonparametric regression model on principal components. A three-stage procedure is proposed to decide how many and which components should be included into such an additive model. In a first step the predictor variables are made orthogonal by the principal component transformation. After the second step, which determines the number and sequence of components, the additive regression model is fit by the kernel method. The asymptotic distribution of this regression estimate is given. The practical performance is investigated via a simulation study.

Doubly regularized Cox regression for high-dimensional survival data with group structures

The goal of this research is to integrate group structures to the Cox proportional hazards model with ultra highdimensional predictors. By doubly regularizing the partial likelihood based on the Cox model with convex penalties, this method is able to perform group selection and withingroup selection simultaneously. Compared with methods ignoring the structure information, our method yields better variable selection and more accurate prediction. The convexity of our regularized objective function makes the method numerically stable especially when the number of predictors far exceeds the number of the observations. A fast coordinate descent algorithm is exploited to avoid matrix operations and speed up the computation. Numerical experiments on simulated data demonstrate the good performance of our doubly regularized method. We analyze the TCGA ovarian cancer data with this new method.