High Dimension Low Sample Size Research Articles

Clustering of high-dimensional observations

We present a novel clustering method for high-dimensional, low sample size (HDLSS) data. The method is distance-based, takes advantage of the distance concentration phenomenon and the limiting values of the dissimilarity indices to construct clusters. We describe an algorithm that orders each row of the dissimilarity matrix to estimate the change points, which define cluster boundaries. We construct an agreement matrix of the Rand indices of the row clusters. The minimum of the row sum of the agreement matrix provides us with the best clusters. We prove that the new method achieves perfect clustering as the number of features diverges for a fixed sample size. Several examples are presented to illustrate the proposed method. We compare the new method with four other clustering techniques, including high-dimensional k-means, minimal spanning tree and Hierarchical Scan. The clustering methods are applied to the Lymphoma data set.

High-dimensional Statistical Analysis and Its Application to an ALMA Map of NGC 253

In astronomy, if we denote the dimension of data as d and the number of samples as n, we often find a case with n ≪ d. Traditionally, such a situation is regarded as ill-posed, and there was no choice but to discard most of the information in data dimensions to let d < n. The data with n ≪ d is referred to as the high-dimensional low sample size (HDLSS). To deal with HDLSS problems, a method called high-dimensional statistics has rapidly developed in the last decade. In this work, we first introduce high-dimensional statistical analysis to the astronomical community. We apply two representative methods in the high-dimensional statistical analysis methods, noise-reduction principal component analysis (NRPCA) and automatic sparse principal component analysis (A-SPCA), to a spectroscopic map of a nearby archetype starburst galaxy NGC 253 taken by the Atacama Large Millimeter/submillimeter Array (ALMA). The ALMA map is an example of a typical HDLSS data set. First, we analyzed the original data, including the Doppler shift due to the systemic rotation. High-dimensional PCA can precisely describe the spatial structure of the rotation. We then applied to the Doppler-shift corrected data to analyze more subtle spectral features. NRPCA and R-SPCA were able to quantify the very complicated characteristics of the ALMA spectra. Particularly, we were able to extract information on the global outflow from the center of NGC 253. This method can also be applied not only to spectroscopic survey data, but also to any type of data with a small sample size and large dimension.

Discriminating Tensor Spectral Clustering for High-Dimension-Low-Sample-Size Data.

Tensor spectral clustering (TSC) is a recently proposed approach to robustly group data into underlying clusters. Unlike the traditional spectral clustering (SC), which merely uses pairwise similarities of data in an affinity matrix, TSC aims at exploring their multiwise similarities in an affinity tensor to achieve better performance. However, the performance of TSC highly relies on the design of multiwise similarities, and it remains unclear especially for high-dimension-low-sample-size (HDLSS) data. To this end, this article has proposed a discriminating TSC (DTSC) for HDLSS data. Specifically, DTSC uses the proposed discriminating affinity tensor that encodes the pair-to-pair similarities, which are particularly constructed by the anchor-based distance. HDLSS asymptotic analysis shows that the proposed affinity tensor can explicitly differentiate samples from different clusters when the feature dimension is large. This theoretical property allows DTSC to improve the clustering performance on HDLSS data. Experimental results on synthetic and benchmark datasets demonstrate the effectiveness and robustness of the proposed method in comparison to several baseline methods.

Asymptotic properties of hierarchical clustering in high-dimensional settings

In this study, three asymptotic behaviors of hierarchical clustering are defined and studied with strict conditions under several asymptotic settings, from large samples to high dimensionality, when having two independent populations. We proceed with the current comprehension of the asymptotic properties of hierarchical clustering in high-dimensional, low-sample-size (HDLSS) settings. For high-dimensional data, the asymptotic properties of hierarchical clustering are demonstrated under mild and practical settings, and we present simulation studies and hierarchical clustering performance discussions. Furthermore, hierarchical clustering was theoretically investigated when both the dimension and sample size approach infinity, and we generalized a latent number of populations considering hierarchical clustering in multiclass HDLSS settings.

Random forest kernel for high-dimension low sample size classification

Random forest kernel for high-dimension low sample size classification

Multiple-instance ensemble for construction of deep heterogeneous committees for high-dimensional low-sample-size data

Deep ensemble learning, where we combine knowledge learned from multiple individual neural networks, has been widely adopted to improve the performance of neural networks in deep learning. This field can be encompassed by committee learning, which includes the construction of neural network cascades. This study focuses on the high-dimensional low-sample-size (HDLS) domain and introduces multiple instance ensemble (MIE) as a novel stacking method for ensembles and cascades. In this study, our proposed approach reformulates the ensemble learning process as a multiple-instance learning problem. We utilise the multiple-instance learning solution of pooling operations to associate feature representations of base neural networks into joint representations as a method of stacking. This study explores various attention mechanisms and proposes two novel committee learning strategies with MIE. In addition, we utilise the capability of MIE to generate pseudo-base neural networks to provide a proof-of-concept for a “growing” neural network cascade that is unbounded by the number of base neural networks. We have shown that our approach provides (1) a class of alternative ensemble methods that performs comparably with various stacking ensemble methods and (2) a novel method for the generation of high-performing “growing” cascades. The approach has also been verified across multiple HDLS datasets, achieving high performance for binary classification tasks in the low-sample size regime.

Design of input assignment and feedback gain for re‐stabilizing undirected networks with High‐Dimension Low‐Sample‐Size data

AbstractThere exists a critical transition before dramatic deterioration of a complex dynamical system. Recently, a method to predict such shifts based on High‐Dimension Low‐Sample‐Size (HDLSS) data has been developed. Thus based on the prediction, it is important to make the system more stable by feedback control just before such critical transitions, which we call re‐stabilization. However, the re‐stabilization cannot be achieved by traditional stabilization methods such as pole placement method because the available HDLSS data is not enough to get a mathematical system model by system identification. In this article, a model‐free pole placement method for re‐stabilization is proposed to design the optimal input assignment and feedback gain for undirected network systems only with HDLSS data. The proposed method is validated by numerical simulations.

Fuzzy clustering based classifier for extraction of individualities from high dimension low sample size data

The target of this research is high dimension low sample size (HDLSS) data in which the number of variables (or dimensions) is much larger than the number of objects. For such a data, it is well known that mathematically (or statistically), we cannot obtain correct solutions as eigen values of the covariance matrix of variables, therefore, a lot of multivariate analyses cannot apply to this type of data. To overcome this problem, we have proposed a methodology for dimensional reduction, which is a fuzzy cluster scaled principal component analysis (fuzzy cluster scaled PCA). This paper presents a study of the applicability of the previously proposed fuzzy cluster scaled PCA for the discrimination of individual subjects observed by sensors worn on the body during several activities. The analysis of this data is useful for healthcare, considering the individuality of the history of activities, such as the implementation of a custom-made system for healthcare.

Data Augmentation in High Dimensional Low Sample Size Setting Using a Geometry-Based Variational Autoencoder.

In this paper, we propose a new method to perform data augmentation in a reliable way in the High Dimensional Low Sample Size (HDLSS) setting using a geometry-based variational autoencoder (VAE). Our approach combines the proposal of 1) a new VAE model, the latent space of which is modeled as a Riemannian manifold and which combines both Riemannian metric learning and normalizing flows and 2) a new generation scheme which produces more meaningful samples especially in the context of small data sets. The method is tested through a wide experimental study where its robustness to data sets, classifiers and training samples size is stressed. It is also validated on a medical imaging classification task on the challenging ADNI database where a small number of 3D brain magnetic resonance images (MRIs) are considered and augmented using the proposed VAE framework. In each case, the proposed method allows for a significant and reliable gain in the classification metrics. For instance, balanced accuracy jumps from 66.3% to 74.3% for a state-of-the-art convolutional neural network classifier trained with 50 MRIs of cognitively normal (CN) and 50 Alzheimer disease (AD) patients and from 77.7% to 86.3% when trained with 243 CN and 210 AD while improving greatly sensitivity and specificity metrics.

Tree enhanced deep adaptive network for cancer prediction with high dimension low sample size microarray data

Cancer prediction based on microarray data can facilitate the molecular exploration of cancers, thus building more accurate cancer prediction models is essential. This study focuses on a deep learning-based cancer prediction model. However, using a deep neural network to predict cancer is a difficult task due to the complexity of the underlying biological patterns and high dimension low sample size (HDLSS) of microarray data, which could bring about over-fitting and large training gradient variance. Therefore, a tree-enhanced deep adaptive network (TEDAN) is proposed to address these issues. Firstly, we employ the idea of the ensemble tree as a feature transformation method to alleviate the over-fitting problem, which generates a feature with a lower dimension and a more discriminative pattern. Secondly, a deep adaptive network (DAN) based on a self-attention mechanism is proposed to model the underlying biological interaction between different genes. Thirdly, a low sample size training (LSST) method is proposed to further reduce the large training gradient variance. Experiment results on six public cancer prediction datasets demonstrate that the TEDAN outperforms other strong baseline models.

James–Stein for the leading eigenvector

Recent research identifies and corrects bias, such as excess dispersion, in the leading sample eigenvector of a factor-based covariance matrix estimated from a high-dimension low sample size (HL) data set. We show that eigenvector bias can have a substantial impact on variance-minimizing optimization in the HL regime, while bias in estimated eigenvalues may have little effect. We describe a data-driven eigenvector shrinkage estimator in the HL regime called "James-Stein for eigenvectors" (JSE) and its close relationship with the James-Stein (JS) estimator for a collection of averages. We show, both theoretically and with numerical experiments, that, for certain variance-minimizing problems of practical importance, efforts to correct eigenvalues have little value in comparison to the JSE correction of the leading eigenvector. When certain extra information is present, JSE is a consistent estimator of the leading eigenvector.

Assessment of Projection Pursuit Index for Classifying High Dimension Low Sample Size Data in R

Analyzing “large p small n” data is becoming increasingly paramount in a wide range of application fields. As a projection pursuit index, the Penalized Discriminant Analysis ($\mathrm{PDA}$) index, built upon the Linear Discriminant Analysis ($\mathrm{LDA}$) index, is devised in Lee and Cook (2010) to classify high-dimensional data with promising results. Yet, there is little information available about its performance compared with the popular Support Vector Machine ($\mathrm{SVM}$). This paper conducts extensive numerical studies to compare the performance of the $\mathrm{PDA}$ index with the $\mathrm{LDA}$ index and $\mathrm{SVM}$, demonstrating that the $\mathrm{PDA}$ index is robust to outliers and able to handle high-dimensional datasets with extremely small sample sizes, few important variables, and multiple classes. Analyses of several motivating real-world datasets reveal the practical advantages and limitations of individual methods, suggesting that the $\mathrm{PDA}$ index provides a useful alternative tool for classifying complex high-dimensional data. These new insights, along with the hands-on implementation of the $\mathrm{PDA}$ index functions in the R package classPP, facilitate statisticians and data scientists to make effective use of both sets of classification tools.

Deep belief network-based approach for detecting Alzheimer's disease using the multi-omics data

Alzheimer's disease (AD) is the most uncertain form of Dementia in terms of finding out the mechanism. AD does not have a vital genetic factor to relate to. There were no reliable techniques and methods to identify the genetic risk factors associated with AD in the past. Most of the data available were from the brain images. However, recently, there have been drastic advancements in the high-throughput techniques in bioinformatics. It has led to focused researches in discovering the AD causing genetic risk factors. Recent analysis has resulted in considerable prefrontal cortex data with which classification and prediction models can be developed for AD. We have developed a Deep Belief Network-based prediction model using the DNA Methylation and Gene Expression Microarray Data, with High Dimension Low Sample Size (HDLSS) issues. To overcome the HDLSS challenge, we performed a two-layer feature selection considering the biological aspects of the features as well. In the two-layered feature selection approach, first the differentially expressed genes and differentially methylated positions are identified, then both the datasets are combined using Jaccard similarity measure. As the second step, an ensemble-based feature selection approach is implemented to further narrow down the gene selection. The results show that the proposed feature selection technique outperforms the existing commonly used feature selection techniques, such as Support Vector Machine Recursive Feature Elimination (SVM-RFE), and Correlation-based Feature Selection (CBS). Furthermore, the Deep Belief Network-based prediction model performs better than the widely used Machine Learning models. Also, the multi-omics dataset shows promising results compared to the single omics.

Fuzzy cluster-scaled principal component analysis for mixed data and its application of educational effect based on device selection

This paper proposes a fuzzy cluster-scaled principal component analysis (fuzzy cluster-scaled PCA) for mixed data, which consists of both numerical and categorical data with respect to quantitative and qualitative variables. The fuzzy cluster-scaled PCA has been proposed for high-dimension, low-sample size (HDLSS) data in which the number of variables (or dimensions) is much larger than the number of objects (or samples). In this case, the target data is only for the numerical data. The proposed fuzzy cluster-scaled PCA in this paper can be applied to the mixed type of HDLSS data by utilizing the feature of the fuzzy cluster-scaled correlation, which is decomposed into two parts: the first part is the correlation of classification structures between variables and the second part is the correlation between variables. Then, the first part can be adapted to the categorical data, and the second part can be used for the numerical data through the same objects. Several numerical examples used data from a survey concerned with the relationship between students’ choice of devices and scores of examination/class marks to measure the educational effect show a better performance of this proposed method, and insightful, important information on educational effectiveness is clarified.

Data-Driven Re-stabilization of Gene Regulatory Network Towards Early Medical Treatment

The Dynamical Network Biomarkers (DNBs) theory has been proposed to detect early-warning signals of critical transitions in gene regulatory networks only with High-Dimension Low-Sample-Size (HDLSS) data of the system state. Towards giving a theoretical foundation for early medical treatment, this paper proposes a data-driven approach for the re-stabilization of gene regulatory networks based on HDLSS data. In the proposed re-stabilization method, only the diagonal elements of the system matrix need to be adjusted. Namely, only the self-feedback loops of mRNA expression for genes are intervened in, which reduces the complexity of the early medical treatment based on gene regulation and makes it practical to be implemented. The proposed re-stabilization method is generalized to the systems with either saddle-node bifurcation or Hopf bifurcation. Numerical simulations have been implemented to validate the effectiveness of the proposed method.

Interpreting models interpreting brain dynamics

Brain dynamics are highly complex and yet hold the key to understanding brain function and dysfunction. The dynamics captured by resting-state functional magnetic resonance imaging data are noisy, high-dimensional, and not readily interpretable. The typical approach of reducing this data to low-dimensional features and focusing on the most predictive features comes with strong assumptions and can miss essential aspects of the underlying dynamics. In contrast, introspection of discriminatively trained deep learning models may uncover disorder-relevant elements of the signal at the level of individual time points and spatial locations. Yet, the difficulty of reliable training on high-dimensional low sample size datasets and the unclear relevance of the resulting predictive markers prevent the widespread use of deep learning in functional neuroimaging. In this work, we introduce a deep learning framework to learn from high-dimensional dynamical data while maintaining stable, ecologically valid interpretations. Results successfully demonstrate that the proposed framework enables learning the dynamics of resting-state fMRI directly from small data and capturing compact, stable interpretations of features predictive of function and dysfunction.

Prediction of Microcystis Occurrences and Analysis Using Machine Learning in High-Dimension, Low-Sample-Size and Imbalanced Water Quality Data

Machine learning, Deep learning, and water quality data have been used in recent years to predict the outbreak of harmful algae, especially Microcystis, and analyze outbreak causes. However, for various reasons, water quality data are often High-Dimension, Low-Sample- Size (HDLSS), meaning the sample size is lower than the number of dimensions. Moreover, imbalance problems may arise due to bias in the occurrence frequency of Microcystis. These problems make predicting the occurrence of Microcystis and analyzing its causes with machine learning difficult. In this study, a machine learning model that applies Feature Engineering (FE) and Feature Selection (FS) algorithms are used to predict outbreaks of Microcystis and analyze the outbreak factors from imbalanced HDLSS water quality data. The prediction performance was verified with binary classification to determine whether Microcystis would occur in the future by applying three machine learning models to four data patterns. The cause analysis of Microcystis occurrence was performed by visualizing the results of applying FE and FS. For the test data, the predictive performance of FE and FS methods was significantly better than that of the conventional method, with an accuracy of .108 points and an F-value of .691 points higher than the conventional method. A prediction performance increase was observed with a smaller model capacity. Data-driven analysis suggested that total nitrogen, chemical oxygen demand, chlorophyll-a, dissolved oxygen saturation, and water temperature are associated with Microcystis occurrences. The results also indicated that basic statistics of the water quality distribution (especially mean, standard deviation, and skewness) over a year, not the concentrations of water components, are related to the occurrence of Microcystis. These are new findings not found in previous studies and are expected to contribute significantly to future studies of algae. This study provides a method for analyzing water quality data with high-dimensionality and small samples, imbalance problems, or both.

Classification for high-dimension low-sample size data

High-dimension and low-sample-size (HDLSS) data sets have posed great challenges to many machine learning methods. To deal with practical HDLSS problems, development of new classification techniques is highly desired. After the cause of the over-fitting phenomenon is identified, a new classification criterion for HDLSS data sets, termed tolerance similarity, is proposed to emphasize maximization of within-class variance on the premise of class separability. Leveraging on this criterion, a novel linear binary classifier, termed No-separated Data Maximum Dispersion classifier (NPDMD), is designed. The main idea of the NPDMD is to spread samples of two classes in a large interval in the respective positive or negative space along the projecting direction when the distance between the projection means for two classes is large enough. The salient features of the proposed NPDMD are: (1) The NPDMD operates well on HDLSS data sets; (2) The NPDMD solves the objective function in the entire feature space to avoid the data-piling phenomenon. (3) The NPDMD leverages on the low-rank property of the covariance matrix for HDLSS data sets to accelerate the computation speed. (4) The NPDMD is suitable for different real-word applications. (5) The NPDMD can be implemented readily using Quadratic Programming. Not only theoretical properties of the NPDMD have been derived, but also a series of evaluations have been conducted on one simulated and six real-world benchmark data sets, including face classification and mRNA classification. Experimental results and comprehensive studies demonstrate the superiority of the NPDMD in terms of correct classification rate, mean within-group correct classification rate and the area under the ROC curve.

Network-based dimensionality reduction of high-dimensional, low-sample-size datasets

In the field of data science, there are a variety of datasets that suffer from the high-dimensional, low-sample-size (HDLSS) problem; however, only a few dimensionality reduction methods exist that are applicable to address this type of problem, and there is no nonparametric solution to date. The purpose of this work is to develop a novel network-based (nonparametric) dimensionality reduction analysis (NDA) method, that can be effectively applied to HDLSS data. First, with the NDA method, the correlation graph of variables is specified. With a modularity-based community detection method, the set of modules is specified. Then, the linear combination of variables weighted by their eigenvector centralities (EVCs), defined as LVs, is determined. In the optional phase of variable selection, variables with low EVCs and low communality are ignored. Then, the set of LVs and the set of indicators belonging to the LVs are specified using the NDA method. NDA is applied to publicly available databases and compared with principal factoring with community analysis (PFA) methods. The results show that NDA can be effectively applied to HDLSS datasets as it outperforms the existing methods in terms of interpretability. In addition, the application of NDA is easier, since there is no need to specify the number of latent variables due to its nonparametric nature.

Covariance shrinkage can assess and improve functional connectomes

Connectomes derived from resting-state functional MRI scans have significantly benefited from the development of dedicated fMRI motion correction and denoising algorithms. But they are based on empirical correlations that can produce unreliable results in high dimension low sample size settings. A family of statistical estimators, the covariance shrinkage methods, could mitigate this issue. Unfortunately, these methods have rarely been used to correct functional connectomes and no extensive experiment has been conducted so far to compare the shrinkage methods available for this task. In this work, we propose to fix this issue by processing a benchmark dataset made of a thousand high-resolution resting-state fMRI scans provided by the Human Connectome Project to compare the ability of five prominent covariance shrinkage methods to produce reliable functional connectomes at different spatial resolutions and scans duration: the pioneer linear covariance shrinkage method introduced by Ledoit and Wolf, the Oracle Approximating Shrinkage, the QuEST method, the NERCOME method, and a recent analytical approximation of the QuEST approach. Our experiments establish that all covariance shrinkage methods significantly improve functional connectomes derived from short fMRI scans. The Oracle Approximating Shrinkage and the QuEST method produced the best results. Lastly, we present shrinkage intensity charts that can be used for designing and analyzing fMRI studies. These charts indicate that sparse connectomes are difficult to estimate from short fMRI scans, and they describe a range of settings where dynamic functional connectivity should not be computed.