High Dimension Low Sample Size Research Articles

High dimensional low sample size activity recognition using geometric classifiers

Research on high dimension, low sample size (HDLSS) data has revealed their neighborless nature. This paper addresses the classification of HDLSS image or video data for human activity recognition. Existing approaches often use off-the-shelf classifiers such as nearest neighbor techniques or support vector machines and tend to ignore the geometry of underlying feature distributions. Addressing this issue, we investigate different geometric classifiers and affirm the lack of neighborhoods within HDLSS data. As this undermines proximity based methods and may cause over-fitting for discriminant methods, we propose a QR factorization approach to Nearest Affine Hull (NAH) classification which remedies the HDLSS dilemma and noticeably reduces time and memory requirements of existing methods. We show that the resulting non-parametric models provide smooth decision surfaces and yield efficient and accurate solutions in multiclass HDLSS scenarios. On several action recognition benchmarks, the proposed NAH classifier outperforms other instance based methods and shows competitive or superior performance than SVMs. In addition, for online settings, the proposed NAH method is faster than online SVMs.

On some transformations of high dimension, low sample size data for nearest neighbor classification

For data with more variables than the sample size, phenomena like concentration of pairwise distances, violation of cluster assumptions and presence of hubness often have adverse effects on the performance of the classic nearest neighbor classifier. To cope with such problems, some dimension reduction techniques like those based on random linear projections and principal component directions have been proposed in the literature. In this article, we construct nonlinear transformations of the data based on inter-point distances, which also lead to reduction in data dimension. More importantly, for such high dimension low sample size data, they enhance separability among the competing classes in the transformed space. When the classic nearest neighbor classifier is used on the transformed data, it usually yields lower misclassification rates. Under appropriate regularity conditions, we derive asymptotic results on misclassification probabilities of nearest neighbor classifiers based on the $$l_2$$l2 norm and the $$l_p$$lp norms (with $$p \in (0,1]$$p?(0,1]) in the transformed space, when the training sample size remains fixed and the dimension of the data grows to infinity. Strength of the proposed transformations in the classification context is demonstrated by analyzing several simulated and benchmark data sets.

Multivariate multidistance tests for high-dimensional low sample size case-control studies.

A class of multivariate tests for case-control studies with high-dimensional low sample size data and with complex dependence structure, which are common in medical imaging and molecular biology, is proposed. The tests can be applied when the number of variables is much larger than the number of subjects and when the underlying population distributions are heavy-tailed or skewed. As a motivating application, we consider a case-control study where phase-contrast cinematic cardiovascular magnetic resonance imaging has been used to compare many cardiovascular characteristics of young healthy smokers and young healthy non-smokers. The tests are based on the combination of tests on interpoint distances. It is theoretically proved that the tests are exact, unbiased and consistent. It is shown that the tests are very powerful under normal, heavy-tailed and skewed distributions. The tests can also be applied to case-control studies with high-dimensional low sample size data from other medical imaging techniques (like computed tomography or X-ray radiography), chemometrics and microarray data (proteomics and transcriptomics).

On some exact distribution-free one-sample tests for high dimension low sample size data

Several rank-based tests for the multivariate one-sample problem are available in the literature. But, unlike univariate rank-based tests, most of these multivariate tests are not distribution-free. Moreover, many of them are not ap- plicable when the dimension of the data exceeds the sample size. We develop and investigate some distribution-free tests for the one-sample location problem, which can be conveniently used in high dimension low sample size (HDLSS) situations. Under some appropriate regularity conditions, we prove the consistency of these tests when the sample size remains fixed and the dimension grows to infinity. Some simulations and data sets are analyzed to compare their performance with popular one-sample tests.

A Variable Selection Method Considering Cluster Loading for Labeled High Dimension Low Sample Size Data

As the information society rapidly develops, there is an increased importance placed of dealing with high dimension low sample size (HDLSS) data, whose number of variables is much larger than the number of objects. Moreover, the selection of effective variables for HDLSS data is becoming more crucial. In this paper, a variable selection method considering cluster loading for labeled HDLSS data is proposed. Related to cluster loading, the conventional model considering principal component analysis has been proposed. However, the model can not be used for HDLSS data. Therefore, we propose a cluster loading model using a clustering result. By using the obtained cluster loading, we can select variables which belong to clusters unrelated with the given discrimination information represented by the labels of objects. Several numerical examples show a better performance of the proposed method.

Random Subspace Learning Approach to High-Dimensional Outliers Detection

We introduce and develop a novel approach to outlier detection based on adaptation of random subspace learning. Our proposed method handles both high-dimension low-sample size and traditional low-dimensional high-sample size datasets. Essentially, we avoid the computational bottleneck of techniques like Minimum Covariance Determinant (MCD) by computing the needed determinants and associated measures in much lower dimensional subspaces. Both theoretical and computational development of our approach reveal that it is computationally more efficient than the regularized methods in high-dimensional low-sample size, and often competes favorably with existing methods as far as the percentage of correct outlier detection are concerned.

Using visual statistical inference to better understand random class separations in high dimension, low sample size data

Statistical graphics play an important role in exploratory data analysis, model checking and diagnosis. With high dimensional data, this often means plotting low-dimensional projections, for example, in classification tasks projection pursuit is used to find low-dimensional projections that reveal differences between labelled groups. In many contemporary data sets the number of observations is relatively small compared to the number of variables, which is known as a high dimension low sample size (HDLSS) problem. This paper explores the use of visual inference on understanding low-dimensional pictures of HDLSS data. Visual inference helps to quantify the significance of findings made from graphics. This approach may be helpful to broaden the understanding of issues related to HDLSS data in the data analysis community. Methods are illustrated using data from a published paper, which erroneously found real separation in microarray data, and with a simulation study conducted using Amazon's Mechanical Turk.

Convergence of Sample Eigenvalues, Eigenvectors, and Principal Component Scores for Ultra-High Dimensional Data.

The development of high-throughput biomedical technologies has led to increased interest in the analysis of high-dimensional data where the number of features is much larger than the sample size. In this paper, we investigate principal component analysis under the ultra-high dimensional regime, where both the number of features and the sample size increase as the ratio of the two quantities also increases. We bridge the existing results from the finite and the high-dimension low sample size regimes, embedding the two regimes in a more general framework. We also numerically demonstrate the universal application of the results from the finite regime.

An algorithm for deciding the number of clusters and validation using simulated data with application to exploring crop population structure

A first step in exploring population structure in crop plants and other organisms is to define the number of subpopulations that exist for a given data set. The genetic marker data sets being generated have become increasingly large over time and commonly are of the high-dimension, low sample size (HDLSS) situation. An algorithm for deciding the number of clusters is proposed, and is validated on simulated data sets varying in both the level of structure and the number of clusters covering the range of variation observed empirically. The algorithm was then tested on six empirical data sets across three small grain species. The algorithm uses bootstrapping, three methods of clustering, and defines the optimum number of clusters based on a common criterion, the Hubert's gamma statistic. Validation on simulated sets coupled with testing on empirical sets suggests that the algorithm can be used for a wide variety of genetic data sets.

A nonparametric two-sample test applicable to high dimensional data

The multivariate two-sample testing problem has been well investigated in the literature, and several parametric and nonparametric methods are available for it. However, most of these two-sample tests perform poorly for high dimensional data, and many of them are not applicable when the dimension of the data exceeds the sample size. In this article, we propose a multivariate two-sample test that can be conveniently used in the high dimension low sample size setup. Asymptotic results on the power properties of our proposed test are derived when the sample size remains fixed, and the dimension of the data grows to infinity. We investigate the performance of this test on several high-dimensional simulated and real data sets, and demonstrate its superiority over several other existing two-sample tests. We also study some theoretical properties of the proposed test for situations when the dimension of the data remains fixed and the sample size tends to infinity. In such cases, it turns out to be asymptotically distribution-free and consistent under general alternatives.

Statistical Significance of Clustering Using Soft Thresholding

Clustering methods have led to a number of important discoveries in bioinformatics and beyond. A major challenge in their use is determining which clusters represent important underlying structure, as opposed to spurious sampling artifacts. This challenge is especially serious, and very few methods are available when the data are very high in dimension. Statistical Significance of Clustering (SigClust) is a recently developed cluster evaluation tool for high dimensional low sample size data. An important component of the SigClust approach is the very definition of a single cluster as a subset of data sampled from a multivariate Gaussian distribution. The implementation of SigClust requires the estimation of the eigenvalues of the covariance matrix for the null multivariate Gaussian distribution. We show that the original eigenvalue estimation can lead to a test that suffers from severe inflation of type-I error, in the important case where there are huge single spikes in the eigenvalues. This paper addresses this critical challenge using a novel likelihood based soft thresholding approach to estimate these eigenvalues which leads to a much improved SigClust. These major improvements in SigClust performance are shown by both theoretical work and an extensive simulation study. Applications to some cancer genomic data further demonstrate the usefulness of these improvements.

HDLSS Discrimination With Adaptive Data Piling

We propose new discrimination methods for classification of high dimension, low sample size (HDLSS) data that regularize the degree of data piling. The within-class scatter of the HDLSS data, when projected onto a low-dimensional discriminant subspace, can be selected to be arbitrarily small. Using this fact, we develop two different ways of tuning the amount of within-class scatter, or equivalently, the degree of data piling. In the first approach, we consider a linear path connecting the maximal data piling and the least data piling directions. We also formulate a problem of finding the optimal classifier under a constraint on data piling. The data piling regularization methods are extended to multicategory problems. Simulated and real data examples show competitive performances of the proposed classification methods. Supplementary materials for this article are available online on the journal web site.

Improved Sparse Multi-Class SVM and Its Application for Gene Selection in Cancer Classification

BackgroundMicroarray techniques provide promising tools for cancer diagnosis using gene expression profiles. However, molecular diagnosis based on high-throughput platforms presents great challenges due to the overwhelming number of variables versus the small sample size and the complex nature of multi-type tumors. Support vector machines (SVMs) have shown superior performance in cancer classification due to their ability to handle high dimensional low sample size data. The multi-class SVM algorithm of Crammer and Singer provides a natural framework for multi-class learning. Despite its effective performance, the procedure utilizes all variables without selection. In this paper, we propose to improve the procedure by imposing shrinkage penalties in learning to enforce solution sparsity.ResultsThe original multi-class SVM of Crammer and Singer is effective for multi-class classification but does not conduct variable selection. We improved the method by introducing soft-thresholding type penalties to incorporate variable selection into multi-class classification for high dimensional data. The new methods were applied to simulated data and two cancer gene expression data sets. The results demonstrate that the new methods can select a small number of genes for building accurate multi-class classification rules. Furthermore, the important genes selected by the methods overlap significantly, suggesting general agreement among different variable selection schemes.ConclusionsHigh accuracy and sparsity make the new methods attractive for cancer diagnostics with gene expression data and defining targets of therapeutic intervention.Availability: The source MATLAB code are available from http://math.arizona.edu/~hzhang/software.html.

A Comparative Study of Feature Selection and Classification Methods for Gene Expression Data of Glioma

Microarray gene expression data gained great importance in recent years due to its role in disease diagnoses and prognoses which help to choose the appropriate treatment plan for patients. This technology has shifted a new era in molecular classification. Interpreting gene expression data remains a difficult problem and an active research area due to their native nature of “high dimensional low sample size”. Such problems pose great challenges to existing classification methods. Thus, effective feature selection techniques are often needed in this case to aid to correctly classify different tumor types and consequently lead to a better understanding of genetic signatures as well as improve treatment strategies. This paper aims on a comparative study of state-of-the- art feature selection methods, classification methods, and the combination of them, based on gene expression data. We compared the efficiency of three different classification methods including: support vector machines, k-nearest neighbor and random forest, and eight different feature selection methods, including: information gain, twoing rule, sum minority, max minority, gini index, sum of variances, t-statistics, and one-dimension support vector machine. Five-fold cross validation was used to evaluate the classification performance. Two publicly available gene expression data sets of glioma were used in the experiments. Results revealed the important role of feature selection in classifying gene expression data. By performing feature selection, the classification accuracy can be significantly boosted by using a small number of genes. The relationship of features selected in different feature selection methods is investigated and the most frequent features selected in each fold among all methods for both datasets are evaluated.

Consistency of sparse PCA in High Dimension, Low Sample Size contexts

Sparse Principal Component Analysis (PCA) methods are efficient tools to reduce the dimension (or number of variables) of complex data. Sparse principal components (PCs) are easier to interpret than conventional PCs, because most loadings are zero. We study the asymptotic properties of these sparse PC directions for scenarios with fixed sample size and increasing dimension (i.e. High Dimension, Low Sample Size (HDLSS)). We consider the previously studied single spike covariance model and assume in addition that the maximal eigenvector is sparse. We extend the existing HDLSS asymptotic consistency and strong inconsistency results of conventional PCA in an entirely new direction. We find a large set of sparsity assumptions under which sparse PCA is still consistent even when conventional PCA is strongly inconsistent. The consistency of sparse PCA is characterized along with rates of convergence. Furthermore, we clearly identify the mathematical boundaries of the sparse PCA consistency, by showing strong inconsistency for an oracle version of sparse PCA beyond the consistent region, as well as its inconsistency on the boundaries of the consistent region. Simulation studies are performed to validate the asymptotic results in finite samples.

Comparison of binary discrimination methods for high dimension low sample size data

A comparison of some binary discrimination methods is done in the high dimension low sample size context for Gaussian data with common diagonal covariance matrix. In particular we obtain results about the asymptotic behavior of the methods Support Vector Machine, Mean Difference (i.e. Centroid Rule), Distance Weighted Discrimination, Maximal Data Piling and Naive Bayes when the dimension d of the data sets tends to infinity and the sample sizes of the classes are fixed. It is concluded that, under appropriate conditions, the first four methods are asymptotically equivalent, but the Naive Bayes method can have a different asymptotic behavior when d tends to infinity.

Pattern recognition based on canonical correlations in a high dimension low sample size context

This paper is concerned with pattern recognition for 2-class problems in a High Dimension Low Sample Size (hdlss) setting. The proposed method is based on canonical correlations between the predictors X and responses Y. The paper proposes a modified version of the canonical correlation matrix ΣX−1/2ΣXYΣY−1/2 which is suitable for discrimination with class labels Y in a hdlss context. The modified canonical correlation matrix yields ranking vectors for variable selection, a discriminant direction and a rule which is essentially equivalent to the naive Bayes rule. The paper examines the asymptotic behavior of the ranking vectors and the discriminant direction and gives precise conditions for hdlss consistency in terms of the growth rates of the dimension and sample size. The feature selection induced by the discriminant direction as ranking vector is shown to work efficiently in simulations and in applications to real hdlss data.

Clustering high dimension, low sample size data using the maximal data piling distance

We propose a new hierarchical clustering method for high dimension, low sample size (HDLSS) data. The method utilizes the fact that each individ- ual data vector accounts for exactly one dimension in the subspace generated by HDLSS data. The linkage that is used for measuring the distance between clus- ters is the orthogonal distance between affine subspaces generated by each cluster. The ideal implementation would be to consider all possible binary splits of the data and choose the one that maximizes the distance in between. Since this is not computationally feasible in general, we use the singular value decomposition for its approximation. We provide theoretical justification of the method by studying high dimensional asymptotics. Also we obtain the probability distribution of the distance measure under the null hypothesis of no split, which we use to propose a criterion for determining the number of clusters. Simulation and data analysis with microarray data show competitive clustering performance of the proposed method.

Boundary behavior in High Dimension, Low Sample Size asymptotics of PCA

In High Dimension, Low Sample Size (HDLSS) data situations, where the dimension d is much larger than the sample size n, principal component analysis (PCA) plays an important role in statistical analysis. Under which conditions does the sample PCA well reflect the population covariance structure? We answer this question in a relevant asymptotic context where d grows and n is fixed, under a generalized spiked covariance model. Specifically, we assume the largest population eigenvalues to be of the order dα, where α<, =, or >1. Earlier results show the conditions for consistency and strong inconsistency of eigenvectors of the sample covariance matrix. In the boundary case, α=1, where the sample PC directions are neither consistent nor strongly inconsistent, we show that eigenvalues and eigenvectors do not degenerate but have limiting distributions. The result smoothly bridges the phase transition represented by the other two cases, and thus gives a spectrum of limits for the sample PCA in the HDLSS asymptotics. While the results hold under a general situation, the limiting distributions under Gaussian assumption are illustrated in greater detail. In addition, the geometric representation of HDLSS data is extended to give three different representations, that depend on the magnitude of variances in the first few principal components.

On the border of extreme and mild spiked models in the HDLSS framework

In the spiked covariance model for High Dimension Low Sample Size (HDLSS) asymptotics where the dimension tends to infinity while the sample size is fixed, a few largest eigenvalues are assumed to grow as the dimension increases. The rate of growth is crucial as the asymptotic behavior of the sample Principal Component (PC) directions changes dramatically, from consistency to strong inconsistency at the boundary of the extreme and mild spiked covariance models. Yet, the behavior at the boundary spiked model is unexplored. We study the HDLSS asymptotic behavior of the eigenvalues and the eigenvectors of the sample covariance matrix at the boundary spiked model and observe that they show intermediate behavior between the extreme and mild spiked models.