High Dimension, Low Sample Size Data Research Articles

Discriminating Tensor Spectral Clustering for High-Dimension-Low-Sample-Size Data.

Tensor spectral clustering (TSC) is a recently proposed approach to robustly group data into underlying clusters. Unlike the traditional spectral clustering (SC), which merely uses pairwise similarities of data in an affinity matrix, TSC aims at exploring their multiwise similarities in an affinity tensor to achieve better performance. However, the performance of TSC highly relies on the design of multiwise similarities, and it remains unclear especially for high-dimension-low-sample-size (HDLSS) data. To this end, this article has proposed a discriminating TSC (DTSC) for HDLSS data. Specifically, DTSC uses the proposed discriminating affinity tensor that encodes the pair-to-pair similarities, which are particularly constructed by the anchor-based distance. HDLSS asymptotic analysis shows that the proposed affinity tensor can explicitly differentiate samples from different clusters when the feature dimension is large. This theoretical property allows DTSC to improve the clustering performance on HDLSS data. Experimental results on synthetic and benchmark datasets demonstrate the effectiveness and robustness of the proposed method in comparison to several baseline methods.

Discriminating Tensor Spectral Clustering for High-Dimension-Low-Sample-Size Data.

Tensor spectral clustering (TSC) is a recently proposed approach to robustly group data into underlying clusters. Unlike the traditional spectral clustering (SC), which merely uses pairwise similarities of data in an affinity matrix, TSC aims at exploring their multiwise similarities in an affinity tensor to achieve better performance. However, the performance of TSC highly relies on the design of multiwise similarities, and it remains unclear especially for high-dimension-low-sample-size (HDLSS) data. To this end, this article has proposed a discriminating TSC (DTSC) for HDLSS data. Specifically, DTSC uses the proposed discriminating affinity tensor that encodes the pair-to-pair similarities, which are particularly constructed by the anchor-based distance. HDLSS asymptotic analysis shows that the proposed affinity tensor can explicitly differentiate samples from different clusters when the feature dimension is large. This theoretical property allows DTSC to improve the clustering performance on HDLSS data. Experimental results on synthetic and benchmark datasets demonstrate the effectiveness and robustness of the proposed method in comparison to several baseline methods.

Design of input assignment and feedback gain for re‐stabilizing undirected networks with High‐Dimension Low‐Sample‐Size data

AbstractThere exists a critical transition before dramatic deterioration of a complex dynamical system. Recently, a method to predict such shifts based on High‐Dimension Low‐Sample‐Size (HDLSS) data has been developed. Thus based on the prediction, it is important to make the system more stable by feedback control just before such critical transitions, which we call re‐stabilization. However, the re‐stabilization cannot be achieved by traditional stabilization methods such as pole placement method because the available HDLSS data is not enough to get a mathematical system model by system identification. In this article, a model‐free pole placement method for re‐stabilization is proposed to design the optimal input assignment and feedback gain for undirected network systems only with HDLSS data. The proposed method is validated by numerical simulations.

Data-Driven Re-stabilization of Gene Regulatory Network Towards Early Medical Treatment

The Dynamical Network Biomarkers (DNBs) theory has been proposed to detect early-warning signals of critical transitions in gene regulatory networks only with High-Dimension Low-Sample-Size (HDLSS) data of the system state. Towards giving a theoretical foundation for early medical treatment, this paper proposes a data-driven approach for the re-stabilization of gene regulatory networks based on HDLSS data. In the proposed re-stabilization method, only the diagonal elements of the system matrix need to be adjusted. Namely, only the self-feedback loops of mRNA expression for genes are intervened in, which reduces the complexity of the early medical treatment based on gene regulation and makes it practical to be implemented. The proposed re-stabilization method is generalized to the systems with either saddle-node bifurcation or Hopf bifurcation. Numerical simulations have been implemented to validate the effectiveness of the proposed method.

Fuzzy cluster-scaled principal component analysis for mixed data and its application of educational effect based on device selection

This paper proposes a fuzzy cluster-scaled principal component analysis (fuzzy cluster-scaled PCA) for mixed data, which consists of both numerical and categorical data with respect to quantitative and qualitative variables. The fuzzy cluster-scaled PCA has been proposed for high-dimension, low-sample size (HDLSS) data in which the number of variables (or dimensions) is much larger than the number of objects (or samples). In this case, the target data is only for the numerical data. The proposed fuzzy cluster-scaled PCA in this paper can be applied to the mixed type of HDLSS data by utilizing the feature of the fuzzy cluster-scaled correlation, which is decomposed into two parts: the first part is the correlation of classification structures between variables and the second part is the correlation between variables. Then, the first part can be adapted to the categorical data, and the second part can be used for the numerical data through the same objects. Several numerical examples used data from a survey concerned with the relationship between students’ choice of devices and scores of examination/class marks to measure the educational effect show a better performance of this proposed method, and insightful, important information on educational effectiveness is clarified.

Improving covariance-regularized discriminant analysis for EHR-based predictive analytics of diseases

Linear Discriminant Analysis (LDA) is a well-known technique for feature extraction and dimension reduction. The performance of classical LDA however, significantly degrades on the High Dimension Low Sample Size (HDLSS) data for the ill-posed inverse problem. Existing approaches for HDLSS data classification typically assume the data in question are with Gaussian distribution and deal the HDLSS classification problem with regularization. However, these assumptions are too strict to hold in many emerging real-life applications, such as enabling personalized predictive analysis using Electronic Health Records (EHRs) data collected from an extremely limited number of patients who have been diagnosed with or without the target disease for prediction. In this paper, we revised the problem of predictive analysis of disease using personal EHR data and LDA classifier. To fill the gap, in this paper, we first studied an analytical model that understands the accuracy of LDA for classifying data with arbitrary distribution. The model gives a theoretical upper bound of LDA error rate that is controlled by two factors: (1) the statistical convergence rate of (inverse) covariance matrix estimators and (2) the divergence of the training/testing datasets to fitted distributions. To this end, we could lower the error rate by balancing the two factors for better classification performance. Hereby, we further proposed a novel LDA classifier De-Sparse that leverages De-sparsified Graphical Lasso to improve the estimation of LDA, which outperforms state-of-the-art LDA approaches developed for HDLSS data. Such advances and effectiveness are further demonstrated by both theoretical analysis and extensive experiments on EHR datasets https://www.overleaf.com/project/5d2728c718f6ff3b2bcf5991 .

Ensemble feature selection in high dimension, low sample size datasets: Parallel and serial combination approaches

Feature selection in high dimension, low sample size (HDLSS) data is always an important data pre-processing task. In the literature, the concept of ensemble learning has been applied to improve single feature selection methods, the so-called ensemble feature selection techniques. The most widely used approach is to combine multiple feature selection methods and their selection results via some sort of aggregation function in a parallel manner. Another ensemble strategy is based on the serial combination approach where the selection results of the first feature selection stage are used as input for the second stage of feature selection to produce the final output. The aim of this paper is to fully explore the performance of parallel and serial combination approaches for ensemble feature selection over HDLSS data. In particular, we strive to answer two research questions: whether parallel and serial based ensemble feature selection can outperform single feature selection and which combination approach is the better choice for ensemble feature selection. The experimental results based on comparing nine parallel and nine serial combinations, as well as three single baseline feature selection methods, including principal component analysis (PCA), genetic algorithm (GA), and C4.5 decision tree, show that ensemble feature selection performs better than single feature selection in terms of classification accuracy. However, there are no significant differences in performance between the single best baseline method (i.e. GA) and the top three parallel and serial combinations. On the other hand, the serial combination approach produces the largest feature reduction rate.

Geometric consistency of principal component scores for high‐dimensional mixture models and its application

AbstractIn this article, we consider clustering based on principal component analysis (PCA) for high‐dimensional mixture models. We present theoretical reasons why PCA is effective for clustering high‐dimensional data. First, we derive a geometric representation of high‐dimension, low‐sample‐size (HDLSS) data taken from a two‐class mixture model. With the help of the geometric representation, we give geometric consistency properties of sample principal component scores in the HDLSS context. We develop ideas of the geometric representation and provide geometric consistency properties for multiclass mixture models. We show that PCA can cluster HDLSS data under certain conditions in a surprisingly explicit way. Finally, we demonstrate the performance of the clustering using gene expression datasets.

Interpretable deep neural network for cancer survival analysis by integrating genomic and clinical data

BackgroundUnderstanding the complex biological mechanisms of cancer patient survival using genomic and clinical data is vital, not only to develop new treatments for patients, but also to improve survival prediction. However, highly nonlinear and high-dimension, low-sample size (HDLSS) data cause computational challenges to applying conventional survival analysis.ResultsWe propose a novel biologically interpretable pathway-based sparse deep neural network, named Cox-PASNet, which integrates high-dimensional gene expression data and clinical data on a simple neural network architecture for survival analysis. Cox-PASNet is biologically interpretable where nodes in the neural network correspond to biological genes and pathways, while capturing the nonlinear and hierarchical effects of biological pathways associated with cancer patient survival. We also propose a heuristic optimization solution to train Cox-PASNet with HDLSS data. Cox-PASNet was intensively evaluated by comparing the predictive performance of current state-of-the-art methods on glioblastoma multiforme (GBM) and ovarian serous cystadenocarcinoma (OV) cancer. In the experiments, Cox-PASNet showed out-performance, compared to the benchmarking methods. Moreover, the neural network architecture of Cox-PASNet was biologically interpreted, and several significant prognostic factors of genes and biological pathways were identified.ConclusionsCox-PASNet models biological mechanisms in the neural network by incorporating biological pathway databases and sparse coding. The neural network of Cox-PASNet can identify nonlinear and hierarchical associations of genomic and clinical data to cancer patient survival. The open-source code of Cox-PASNet in PyTorch implemented for training, evaluation, and model interpretation is available at: https://github.com/DataX-JieHao/Cox-PASNet.

Kernel naive Bayes discrimination for high‐dimensional pattern recognition

SummaryKernel discriminant analysis translates the original classification problem into feature space and solves the problem with dimension and sample size interchanged. In high‐dimension low sample size (HDLSS) settings, this reduces the ‘dimension’ to that of the sample size. For HDLSS two‐class problems we modify Mika's kernel Fisher discriminant function which – in general – remains ill‐posed even in a kernel setting; see Mika et al. (1999). We propose a kernel naive Bayes discriminant function and its smoothed version, using first‐ and second‐degree polynomial kernels. For fixed sample size and increasing dimension, we present asymptotic expressions for the kernel discriminant functions, discriminant directions and for the error probability of our kernel discriminant functions. The theoretical calculations are complemented by simulations which show the convergence of the estimators to the population quantities as the dimension grows. We illustrate the performance of the new discriminant rules, which are easy to implement, on real HDLSS data. For such data, our results clearly demonstrate the superior performance of the new discriminant rules, and especially their smoothed versions, over Mika's kernel Fisher version, and typically also over the commonly used naive Bayes discriminant rule.

Some high-dimensional one-sample tests based on functions of interpoint distances

The multivariate one-sample location problem is well studied in the literature, and several tests are available for it. But most of the existing one-sample tests perform poorly for high-dimensional data, and many of them are not even applicable when the dimension of the data exceeds the sample size. In this article, we develop and investigate some nonparametric one-sample tests based on functions of interpoint distances. These proposed tests can be conveniently used in high dimension, low sample size (HDLSS) situations, and good power properties of these tests for HDLSS data have been established using theoretical as well as numerical results.

Sparse HDLSS discrimination with constrained data piling

Regularization is a key component in high dimensional data analyses. In high dimensional discrimination with binary classes, the phenomenon of data piling occurs when the projection of data onto a discriminant vector is dichotomous, one for each class. Regularizing the degree of data piling yields a new class of discrimination rules for high dimension–low sample size data. A discrimination method that regularizes the degree of data piling while achieving sparsity is proposed and solved via a linear programming. Computational efficiency is further improved by a sign-preserving regularization that forces the signs of the estimator to be the same as the mean difference. The proposed classifier shows competitive performances for simulated and real data examples including speech recognition and gene expressions.

High dimensional low sample size activity recognition using geometric classifiers

Research on high dimension, low sample size (HDLSS) data has revealed their neighborless nature. This paper addresses the classification of HDLSS image or video data for human activity recognition. Existing approaches often use off-the-shelf classifiers such as nearest neighbor techniques or support vector machines and tend to ignore the geometry of underlying feature distributions. Addressing this issue, we investigate different geometric classifiers and affirm the lack of neighborhoods within HDLSS data. As this undermines proximity based methods and may cause over-fitting for discriminant methods, we propose a QR factorization approach to Nearest Affine Hull (NAH) classification which remedies the HDLSS dilemma and noticeably reduces time and memory requirements of existing methods. We show that the resulting non-parametric models provide smooth decision surfaces and yield efficient and accurate solutions in multiclass HDLSS scenarios. On several action recognition benchmarks, the proposed NAH classifier outperforms other instance based methods and shows competitive or superior performance than SVMs. In addition, for online settings, the proposed NAH method is faster than online SVMs.

A Variable Selection Method Considering Cluster Loading for Labeled High Dimension Low Sample Size Data

As the information society rapidly develops, there is an increased importance placed of dealing with high dimension low sample size (HDLSS) data, whose number of variables is much larger than the number of objects. Moreover, the selection of effective variables for HDLSS data is becoming more crucial. In this paper, a variable selection method considering cluster loading for labeled HDLSS data is proposed. Related to cluster loading, the conventional model considering principal component analysis has been proposed. However, the model can not be used for HDLSS data. Therefore, we propose a cluster loading model using a clustering result. By using the obtained cluster loading, we can select variables which belong to clusters unrelated with the given discrimination information represented by the labels of objects. Several numerical examples show a better performance of the proposed method.

Using visual statistical inference to better understand random class separations in high dimension, low sample size data

Statistical graphics play an important role in exploratory data analysis, model checking and diagnosis. With high dimensional data, this often means plotting low-dimensional projections, for example, in classification tasks projection pursuit is used to find low-dimensional projections that reveal differences between labelled groups. In many contemporary data sets the number of observations is relatively small compared to the number of variables, which is known as a high dimension low sample size (HDLSS) problem. This paper explores the use of visual inference on understanding low-dimensional pictures of HDLSS data. Visual inference helps to quantify the significance of findings made from graphics. This approach may be helpful to broaden the understanding of issues related to HDLSS data in the data analysis community. Methods are illustrated using data from a published paper, which erroneously found real separation in microarray data, and with a simulation study conducted using Amazon's Mechanical Turk.

HDLSS Discrimination With Adaptive Data Piling

We propose new discrimination methods for classification of high dimension, low sample size (HDLSS) data that regularize the degree of data piling. The within-class scatter of the HDLSS data, when projected onto a low-dimensional discriminant subspace, can be selected to be arbitrarily small. Using this fact, we develop two different ways of tuning the amount of within-class scatter, or equivalently, the degree of data piling. In the first approach, we consider a linear path connecting the maximal data piling and the least data piling directions. We also formulate a problem of finding the optimal classifier under a constraint on data piling. The data piling regularization methods are extended to multicategory problems. Simulated and real data examples show competitive performances of the proposed classification methods. Supplementary materials for this article are available online on the journal web site.

Clustering high dimension, low sample size data using the maximal data piling distance

We propose a new hierarchical clustering method for high dimension, low sample size (HDLSS) data. The method utilizes the fact that each individ- ual data vector accounts for exactly one dimension in the subspace generated by HDLSS data. The linkage that is used for measuring the distance between clus- ters is the orthogonal distance between affine subspaces generated by each cluster. The ideal implementation would be to consider all possible binary splits of the data and choose the one that maximizes the distance in between. Since this is not computationally feasible in general, we use the singular value decomposition for its approximation. We provide theoretical justification of the method by studying high dimensional asymptotics. Also we obtain the probability distribution of the distance measure under the null hypothesis of no split, which we use to propose a criterion for determining the number of clusters. Simulation and data analysis with microarray data show competitive clustering performance of the proposed method.

Boundary behavior in High Dimension, Low Sample Size asymptotics of PCA

In High Dimension, Low Sample Size (HDLSS) data situations, where the dimension d is much larger than the sample size n, principal component analysis (PCA) plays an important role in statistical analysis. Under which conditions does the sample PCA well reflect the population covariance structure? We answer this question in a relevant asymptotic context where d grows and n is fixed, under a generalized spiked covariance model. Specifically, we assume the largest population eigenvalues to be of the order dα, where α<, =, or >1. Earlier results show the conditions for consistency and strong inconsistency of eigenvectors of the sample covariance matrix. In the boundary case, α=1, where the sample PC directions are neither consistent nor strongly inconsistent, we show that eigenvalues and eigenvectors do not degenerate but have limiting distributions. The result smoothly bridges the phase transition represented by the other two cases, and thus gives a spectrum of limits for the sample PCA in the HDLSS asymptotics. While the results hold under a general situation, the limiting distributions under Gaussian assumption are illustrated in greater detail. In addition, the geometric representation of HDLSS data is extended to give three different representations, that depend on the magnitude of variances in the first few principal components.

Some considerations of classification for high dimension low-sample size data

We review in this article several classification methods, especially for high-dimensional and low-sample size data. We discuss several desirable properties for classifiers in such settings, including predictability, consistency, generality, stability, robustness and sparsity. Specifically, a good classifier should have a small prediction error (predictability); converge to the Bayes-rule classifier asymptotically (consistency); be stable when adding/removing an observation (generality); be stable for different data sets of the same kind (stochastic stability); be stable when there are a small number of contaminated observations (robustness); and have a small number of variables in the classifier (interpretability or sparsity). Several simulation examples and real applications are used to illustrate the usefulness of the existing popular classifiers and compare their performance.

Effective PCA for high-dimension, low-sample-size data with noise reduction via geometric representations

In this article, we propose a new estimation methodology to deal with PCA for high-dimension, low-sample-size (HDLSS) data. We first show that HDLSS datasets have different geometric representations depending on whether a ρ -mixing-type dependency appears in variables or not. When the ρ -mixing-type dependency appears in variables, the HDLSS data converge to an n -dimensional surface of unit sphere with increasing dimension. We pay special attention to this phenomenon. We propose a method called the noise-reduction methodology to estimate eigenvalues of a HDLSS dataset. We show that the eigenvalue estimator holds consistency properties along with its limiting distribution in HDLSS context. We consider consistency properties of PC directions. We apply the noise-reduction methodology to estimating PC scores. We also give an application in the discriminant analysis for HDLSS datasets by using the inverse covariance matrix estimator induced by the noise-reduction methodology.