Clustering High-dimensional Data Research Articles

Tuning-free sparse clustering via alternating hard-thresholding

Model-based clustering is a commonly-used technique to partition heterogeneous data into homogeneous groups. When the analysis is to be conducted with a large number of features, analysts face simultaneous challenges in model interpretability, clustering accuracy, and computational efficiency. Several Bayesian and penalization methods have been proposed to select important features for model-based clustering. However, the performance of those methods relies on a careful algorithmic tuning, which can be time-consuming for high-dimensional cases. In this paper, we propose a new sparse clustering method based on alternating hard-thresholding. The new method is conceptually simple and tuning-free. With a user-specified sparsity level, it efficiently detects a set of key features by eliminating a large number of features that are less useful for clustering. Based on the selected key features, one can readily obtain an effective clustering of the original high-dimensional data under a general sparse covariance structure. Under mild conditions, we show that the new method leads to clusters with a misclassification rate consistent to the optimal rate as if the underlying true model were used. The promising performance of the new method is supported by both simulated and real data examples.

Robust and compact maximum margin clustering for high-dimensional data

In the field of machine learning, clustering has become an increasingly popular research topic due to its critical importance. Many clustering algorithms have been proposed utilizing a variety of approaches. This study focuses on clustering of high-dimensional data using the maximum margin clustering approach. In this paper, two methods are introduced: The first method employs the classical maximum margin clustering approach, which separates data into two clusters with the greatest margin between them. The second method takes cluster compactness into account and searches for two parallel hyperplanes that best fit to the cluster samples while also being as far apart from each other as possible. Additionally, robust variants of these clustering methods are introduced to handle outliers and noise within the data samples. The stochastic gradient algorithm is used to solve the resulting optimization problems, enabling all proposed clustering methods to scale well with large-scale data. Experimental results demonstrate that the proposed methods are more effective than existing maximum margin clustering methods, particularly in high-dimensional clustering problems, highlighting the efficacy of the proposed methods.

Clustering High-Dimensional Noisy Categorical Data

Clustering is a widely used unsupervised learning technique that groups data into homogeneous clusters. However, when dealing with real-world data that contain categorical values, existing algorithms can be computationally costly in high dimensions and can struggle with noisy data that has missing values. Furthermore, except for one algorithm, no others provide theoretical guarantees of clustering accuracy. In this article, we propose a general categorical data encoding method and a computationally efficient spectral-based algorithm to cluster high-dimensional noisy categorical data (nominal or ordinal). Under a statistical model for data on m attributes from n subjects in r clusters with missing probability ϵ, we show that our algorithm exactly recovers the true clusters with high probability when m n ( 1 − ϵ ) ≥ C M r 2 log 3 M , with M = max ( n , m ) and a fixed constant C. In addition, we show that m n ( 1 − ϵ ) 2 ≥ r δ / 2 with 0 < δ < 1 is necessary for any algorithm to succeed with probability at least ( 1 + δ ) / 2 . In cases where m = n and r are fixed, the sufficient condition matches with the necessary condition up to a polylog ( n ) factor. In numerical studies our algorithm outperforms several existing algorithms in both clustering accuracy and computational efficiency. Supplementary materials for this article are available online.

Improved Selective Deep-Learning-Based Clustering Ensemble

Clustering ensemble integrates multiple base clustering results to improve the stability and robustness of the single clustering method. It consists of two principal steps: a generation step, which is about the creation of base clusterings, and a consensus function, which is the integration of all clusterings obtained in the generation step. However, most of the existing base clustering algorithms used in the generation step are shallow clustering algorithms such as k-means. These shallow clustering algorithms do not work well or even fail when dealing with large-scale, high-dimensional unstructured data. The emergence of deep clustering algorithms provides a solution to address this challenge. Deep clustering combines the unsupervised commonality of deep representation learning to address complex high-dimensional data clustering, which has achieved excellent performance in many fields. In light of this, we introduce deep clustering into clustering ensemble and propose an improved selective deep-learning-based clustering ensemble algorithm (ISDCE). ISDCE exploits the deep clustering algorithm with different initialization parameters to generate multiple diverse base clusterings. Next, ISDCE constructs ensemble quality and diversity evaluation metrics of base clusterings to select higher-quality and rich-diversity candidate base clusterings. Finally, a weighted graph partition consensus function is utilized to aggregate the candidate base clusterings to obtain a consensus clustering result. Extensive experimental results on various types of datasets demonstrate that ISDCE performs significantly better than existing clustering ensemble approaches.

Mixtures of factor analysers with censored responses and external covariates: An application to educational data from Peru.

Analysing data from educational tests allows governments to make decisions for improving the quality of life of individuals in a society. One of the key responsibilities of statisticians is to develop models that provide decision-makers with pertinent information about the latent process that educational tests seek to represent. Mixtures of factor analysers (MtFA) have emerged as a powerful device for model-based clustering and classification of high-dimensional data containing one or several groups of observations with fatter tails or anomalous outliers. This paper considers an extension of MtFA for robust clustering of censored data, referred to as the MtFAC model, by incorporating external covariates. The enhanced flexibility of including covariates in MtFAC enables cluster-specific multivariate regression analysis of dependent variables with censored responses arising from upper and/or lower detection limits of experimental equipment. An alternating expectation conditional maximization (AECM) algorithm is developed for maximum likelihood estimation of the proposed model. Two simulation experiments are conducted to examine the effectiveness of the techniques presented. Furthermore, the proposed methodology is applied to Peruvian data from the 2007 Early Grade Reading Assessment, and the results obtained from the analysis provide new insights regarding the reading skills of Peruvian students.

Statistical method for clustering high-dimensional data based on fuzzy mathematical modeling

Abstract In this paper, firstly, the fuzzy mathematical model and its types and algorithms of fuzzy numbers are studied. Then, the concept of fuzzy mathematics is added to cluster analysis, and fuzzy cluster analysis is carried out on samples or objects after data standardization and the construction of a similarity matrix. Then, the fuzzy C-mean FCM algorithm is proposed, and the FCM algorithm is improved by introducing the point density of data objects, the maximum minimum distance method of point density sampling, and the reduction of computation to improve the statistical function of clustering high-dimensional data. Finally, the algorithm of this paper is compared with other algorithms by AC, PR, RE, convergence speed, running time, DBI index and other indexes in order to analyze the advantages of the FCM algorithm in high dimensional data clustering statistics. The results show that the F value of the FCM algorithm in the Zoo dataset reaches 0.976, which improves 61% compared to FKM and nearly 12.6% compared to the IIFKM0 algorithm, with the best clustering effect. The DBI index of the FCM algorithm is the lowest in the four datasets, which is basically stabilized in the range of 0.5-0.6. This is better than other algorithms.

Incomplete multiview subspace clustering based on multiple kernel low-redundant representation learning

Subspace clustering is a widely used technique for clustering high-dimensional data. However, its effectiveness is limited in the context of incomplete multiview clustering, where intact subspaces cannot be obtained due to missing instances. To address this issue, we present a novel approach for incomplete multiview subspace clustering based on multiple kernel completion, low-redundant representation learning, and weighted tensor low-rank constraint. First, a carefully designed kernel completion scheme is employed to obtain intact kernels, from which the complete low-redundant representations are learned to obtain intact and compact subspaces. Second, unlike the traditional pairwise subspace fusion, we propose to fuse the multiview subspaces with a weighted tensor low-rank constraint, which not only explores higher-order relationships among views but also assigns appropriate weights to each view. Finally, we propose a unified model that jointly learns low-redundant representations, view-specific subspaces, and their low-rank tensor structure. Extensive experiments conducted on four publicly available datasets demonstrate the effectiveness of the proposed method.

Adaptive local Principal Component Analysis improves the clustering of high-dimensional data

In local Principal Component Analysis (PCA), a distribution is approximated by multiple units, each representing a local region by a hyper-ellipsoid obtained through PCA. We present an extension for local PCA which adaptively adjusts both the learning rate of each unit and the potential function which guides the competition between the local units. Our local PCA method is an online neural network method where unit centers and shapes are modified after the presentation of each data point. For several benchmark distributions, we demonstrate that our method improves the overall quality of clustering, especially for high-dimensional distributions where many conventional methods do not perform satisfactorily. Our online method is also well suited for the processing of streaming data: The two adaptive mechanisms lead to a quick reorganization of the clustering when the underlying distribution changes.

Hypergraph-Clustering Method Based on an Improved Apriori Algorithm

With the complexity and variability of data structures and dimensions, traditional clustering algorithms face various challenges. The integration of network science and clustering has become a popular field of exploration. One of the main challenges is how to handle large-scale and complex high-dimensional data effectively. Hypergraphs can accurately represent multidimensional heterogeneous data, making them important for improving clustering performance. In this paper, we propose a hypergraph-clustering method dubbed the “high-dimensional data clustering method” based on hypergraph partitioning using an improved Apriori algorithm (HDHPA). First, the method constructs a hypergraph based on the improved Apriori association rule algorithm, where frequent itemsets existing in high-dimensional data are treated as hyperedges. Then, different frequent itemsets are mined in parallel to obtain hyperedges with corresponding ranks, avoiding the generation of redundant rules and improving mining efficiency. Next, we use the dense subgraph partition (DSP) algorithm to divide the hypergraph into multiple subclusters. Finally, we merge the subclusters through dense sub-hypergraphs to obtain the clustering results. The advantage of this method lies in its use of the hypergraph model to discretize the association between data in space, which further enhances the effectiveness and accuracy of clustering. We comprehensively compare the proposed HDHPA method with several advanced hypergraph-clustering methods using seven different types of high-dimensional datasets and then compare their running times. The results show that the clustering evaluation index values of the HDHPA method are generally superior to all other methods. The maximum ARI value can reach 0.834, an increase of 42%, and the average running time is lower than other methods. All in all, HDHPA exhibits an excellent comparable performance on multiple real networks. The research results of this paper provide an effective solution for processing and analyzing large-scale network datasets and are also conducive to broadening the application range of clustering techniques.

Kernel Block Diagonal Representation Subspace Clustering with Similarity Preservation

Subspace clustering methods based on the low-rank and sparse model are effective strategies for high-dimensional data clustering. However, most existing low-rank and sparse methods with self-expression can only deal with linear structure data effectively, but they cannot handle data with complex nonlinear structure well. Although kernel subspace clustering methods can efficiently deal with nonlinear structure data, some similarity information between samples may be lost when the original data are reconstructed in the kernel space. Moreover, these kernel subspace clustering methods may not obtain an affinity matrix with an optimal block diagonal structure. In this paper, we propose a novel subspace clustering method termed kernel block diagonal representation subspace clustering with similarity preservation (KBDSP). KBDSP contains three contributions: (1) an affinity matrix with block diagonal structure is generated by introducing a block diagonal representation term; (2) a similarity-preserving regularizer is constructed and embedded into our model by minimizing the discrepancy between inner products of original data and inner products of reconstructed data in the kernel space, which better preserve the similarity information between original data; (3) the KBDSP model is proposed by integrating the block diagonal representation term and similarity-preserving regularizer into the kernel self-expressing frame. The optimization of our proposed model is solved efficiently by utilizing the alternating direction method of multipliers (ADMM). Experimental results on nine datasets demonstrate the effectiveness of the proposed method.

Sparse kernel k-means for high-dimensional data

The kernel k-means method usually loses its power when clustering high-dimensional data, due to a large number of irrelevant features. We propose a novel sparse kernel k-means clustering (SKKM) to extend the advantages of kernel k-means to the high-dimensional cases. We assign each feature a 0-1 indicator and optimize an equivalent kernel k-means loss function while penalizing the sum of the indicators. An alternating minimization algorithm is proposed to estimate both the class labels and the feature indicators. We prove the consistency of both clustering and feature selection of the proposed method. In addition, we apply the proposed framework to the normalized cut. In the numerical experiments, we demonstrate that the proposed method provides better/comparable performance compared to the existing high-dimensional clustering methods.

Retracted: An Ensemble Clustering Approach (Consensus Clustering) for High-Dimensional Data

Retracted: An Ensemble Clustering Approach (Consensus Clustering) for High-Dimensional Data

UEQMS: UMAP Embedded Quick Mean Shift Algorithm for High Dimensional Clustering

The mean shift algorithm is a simple yet very effective clustering method widely used for image and video segmentation as well as other exploratory data analysis applications. Recently, a new algorithm called MeanShift++ (MS++) for low-dimensional clustering was proposed with a speedup of 4000 times over the vanilla mean shift. In this work, starting with a first-of-its-kind theoretical analysis of MS++, we extend its reach to high-dimensional data clustering by integrating the Uniform Manifold Approximation and Projection (UMAP) based dimensionality reduction in the same framework. Analytically, we show that MS++ can indeed converge to a non-critical point. Subsequently, we suggest modifications to MS++ to improve its convergence characteristics. In addition, we propose a way to further speed up MS++ by avoiding the execution of the MS++ iterations for every data point. By incorporating UMAP with modified MS++, we design a faster algorithm, named UMAP embedded quick mean shift (UEQMS), for partitioning data with a relatively large number of recorded features. Through extensive experiments, we showcase the efficacy of UEQMS over other state-of-the-art algorithms in terms of accuracy and runtime.

A Proposed Clustering Algorithm for Efficient Clustering of High-Dimensional Data

To partition transaction data values, clustering algorithms are used. To analyse the relationships between transactions, similarity measures are utilized. Similarity models based on vectors perform well with low-dimensional data. High-dimensional data values are clustered using subspace clustering techniques. Clustering high-dimensional data is difficult due to the curse of dimensionality. Projective clustering seeks out projected clusters in subsets of a data space's dimensions. In high-dimensional data space, a probability model represents predicted clusters. A model-based fuzzy projection clustering method to find clusters with overlapping boundaries in different projection subspaces. The system employs the Model Based Projective Clustering (MPC) method. To cluster high-dimensional data, projective clustering algorithms are used. A subspace clustering technique is the model-based projective clustering algorithm. Similarity analysis use non-axis-subspaces. Anomaly transactions are segmented using projected clusters. The suggested system is intended to cluster objects in high-dimensional spaces. The similarity analysis includes non-access subspaces. The clustering procedure validates anomaly data values with similarity. The subspace selection procedure has been optimized. A subspace clustering approach is the model-based projective clustering algorithm. Similarity analysis use non-axis-subspaces. Anomaly transactions are segmented using projected clusters. The suggested system is intended to cluster objects in high-dimensional spaces. The similarity analysis includes non-access subspaces. The clustering procedure validates anomaly data values with similarity. The subspace selection procedure has been improved.

Joint feature selection and optimal bipartite graph learning for subspace clustering

Recently, there has been tremendous interest in developing graph-based subspace clustering in high-dimensional data, which does not require a priori knowledge of the number of dimensions and subspaces. The general steps of such algorithms are dictionary representation and spectral clustering. Traditional methods use the dataset itself as a dictionary when performing dictionary representation. There are some limitations that the redundant information present in the dictionary and features may make the constructed graph structure unclear and require post-processing to obtain labels. To address these problems, we propose a novel subspace clustering model that first introduces feature selection to process the input data, randomly selects some samples to construct a dictionary to remove redundant information and learns the optimal bipartite graph with K-connected components under the constraint of the (normalized) Laplacian rank. Finally, the labels are obtained directly from the graphs. The experimental results on motion segmentation and face recognition datasets demonstrate the superior effectiveness and stability of our algorithm.

Deep Multi-Constraint Soft Clustering Analysis for Single-Cell RNA-Seq Data via Zero-Inflated Autoencoder Embedding.

Clustering cells into subgroups plays a critical role in single cell-based analyses, which facilitates to reveal cell heterogeneity and diversity. Due to the ever-increasing scRNA-seq data and low RNA capture rate, it has become challenging to cluster high-dimensional and sparse scRNA-seq data. In this study, we propose a single-cell Multi-Constraint deep soft K-means Clustering(scMCKC) framework. Based on zero-inflated negative binomial (ZINB) model-based autoencoder, scMCKC constructs a novel cell-level compactness constraint by considering association between similar cell, to emphasize the compactness between clusters. Besides, scMCKC utilizes pairwise constraint encoded by prior information to guide clustering. Meanwhile, a weighted soft K-means algorithm is leveraged to determine the cell populations, which assigns the label based on affinity between data and clustering center. Experiments on eleven scRNA-seq datasets demonstrate that scMCKC is superior to the state-of-the-art methods and notably improves cluster performance. Moreover, we validate the robustness on human kidney dataset, which demonstrates that scMCKC exhibits comprehensively excellent performance on clustering analysis. The ablation study on eleven datasets proves that the novel cell-level compactness constraint is conductive to the clustering results.

HSCFC: High-dimensional streaming data clustering algorithm based on feedback control system

At present, few studies have considered the clustering analysis of high-dimensional data collected by many sensors in a real-time streaming environment. Existing clustering analysis algorithms for high-dimensional data are primarily based on batch processing models, and most of them cannot meet the requirements of incremental high-dimensional data streams, which are extremely common in practical applications. To address the aforementioned problems, this paper focuses on the study of high-dimensional data clustering based on stream processing, and proposes a high-dimensional data stream clustering algorithm based on a feedback control system, which comprises three stages: window principal component analysis, feedback stream clustering, and feedback controller. The classic exponentially weighted attenuation function is used in the window principal component analysis to avoid concept drift in the data stream, and incremental feature extraction executes through the sliding window to improve the iterative efficiency of the data in the window. To minimize the errors caused by variability in projection angles during dimensionality reduction, a feedback stream clustering stage is designed with alternating iterations of window clustering and cluster aggregation. Aiming at the problems caused by manually adjusting the hyperparameters used in high-dimensional data stream clustering, a feedback controller is developed to adjust the hyperparameters in the two other stages by analyzing the clustering results in real time and using a discriminant score to adopt corresponding feedback strategies. The experimental comparisons between the proposed and the traditional algorithms on multiple datasets demonstrate the effectiveness of the former.

Marigold: Efficientk-Means Clustering in High Dimensions

How can we efficiently and scalably cluster high-dimensional data? Thek-means algorithm clusters data by iteratively reducing intra-cluster Euclidean distances until convergence. While it finds applications from recommendation engines to image segmentation, its application to high-dimensional data is hindered by the need to repeatedly compute Euclidean distances among points and centroids. In this paper, we propose Marigold (k-means for high-dimensional data), a scalable algorithm fork-means clustering in high dimensions. Marigold prunes distance calculations by means of (i) a tight distance-bounding scheme; (ii) a stepwise calculation over a multiresolution transform; and (iii) exploiting the triangle inequality. To our knowledge, such an arsenal of pruning techniques has not been hitherto applied tok-means. Our work is motivated by time-critical Angle-Resolved Photoemission Spectroscopy (ARPES) experiments, where it is vital to detect clusters among high-dimensional spectra in real time. In a thorough experimental study with real-world data sets we demonstrate that Marigold efficiently clusters high-dimensional data, achieving approximately one order of magnitude improvement over prior art.

CoIn: Correlation Induced Clustering for Cognition of High Dimensional Bioinformatics Data.

Analysis of high dimensional biomedical data such as microarray gene expression data and mass spectrometry images, is crucial to provide better medical services including cancer subtyping, protein homology detection,etc. Clustering is a fundamental cognitive task which aims to group unlabeled data into multiple clusters based on their intrinsic similarities. The K-means algorithm is one of the most widely used clustering heuristics that aims at grouping the data objects into meaningful clusters such that the sum of squared Euclidean distances within each cluster is minimized. Its conceptual simplicity and computational efficiency make it easy to be used for wide applications of different data types. However, all features of data in K-means are considered equally in relevance, which distorts the performance when clustering high-dimensional data such as microarray gene expression data, mass spectrometry images, where there exist many redundant variables and correlated variables. In this paper, we propose a new correlation induced clustering, CoIn, to capture complex correlations among high dimensional data and guarantee the correlation consistency within each cluster. We evaluate the proposed method on a high dimensional mass spectrometry dataset of liver cancer tumor to explore the metabolic differences on tissues and discover the intra-tumor heterogeneity (ITH). By comparing the results of baselines and ours, it has been found that our method produces more explainable and understandable results for clinical analysis, which demonstrates the proposed clustering paradigm has the potential with application to knowledge discovery in high dimensional bioinformatics data.

Subspace Clustering in High-Dimensional Data Streams: A Systematic Literature Review

Clustering high dimensional data is challenging as data dimensionality increases the distance between data points, resulting in sparse regions that degrade clustering performance. Subspace clustering is a common approach for processing high-dimensional data by finding relevant features for each cluster in the data space. Subspace clustering methods extend traditional clustering to account for the constraints imposed by data streams. Data streams are not only high-dimensional, but also unbounded and evolving. This necessitates the development of subspace clustering algorithms that can handle high dimensionality and adapt to the unique characteristics of data streams. Although many articles have contributed to the literature review on data stream clustering, there is currently no specific review on subspace clustering algorithms in high-dimensional data streams. Therefore, this article aims to systematically review the existing literature on subspace clustering of data streams in high-dimensional streaming environments. The review follows a systematic methodological approach and includes 18 articles for the final analysis. The analysis focused on two research questions related to the general clustering process and dealing with the unbounded and evolving characteristics of data streams. The main findings relate to six elements: clustering process, cluster search, subspace search, synopsis structure, cluster maintenance, and evaluation measures. Most algorithms use a two-phase clustering approach consisting of an initialization stage, a refinement stage, a cluster maintenance stage, and a final clustering stage. The density-based top-down subspace clustering approach is more widely used than the others because it is able to distinguish true clusters and outliers using projected micro-clusters. Most algorithms implicitly adapt to the evolving nature of the data stream by using a time fading function that is sensitive to outliers. Future work can focus on the clustering framework, parameter optimization, subspace search techniques, memory-efficient synopsis structures, explicit cluster change detection, and intrinsic performance metrics. This article can serve as a guide for researchers interested in high-dimensional subspace clustering methods for data streams.