Deep Clustering Research Articles

Retraction Note: Regularized deep clustering approach for effective categorization of maize diseases

Retraction Note: Regularized deep clustering approach for effective categorization of maize diseases

Deep clustering analysis via variational autoencoder with Gamma mixture latent embeddings

This article proposes a novel deep clustering model based on the variational autoencoder (VAE), named GamMM-VAE, which can learn latent representations of training data for clustering in an unsupervised manner. Most existing VAE-based deep clustering methods use the Gaussian mixture model (GMM) as a prior on the latent space. We employ a more flexible asymmetric Gamma mixture model to achieve higher quality embeddings of the data latent space. Second, since the Gamma is defined for strictly positive variables, in order to exploit the reparameterization trick of VAE, we propose a transformation method from Gaussian distribution to Gamma distribution. This method can also be considered a Gamma distribution reparameterization trick, allows gradients to be backpropagated through the sampling process in the VAE. Finally, we derive the evidence lower bound (ELBO) based on the Gamma mixture model in an effective way for the stochastic gradient variational Bayesian (SGVB) estimator to optimize the proposed model. ELBO, a variational inference objective, ensures the maximization of the approximation of the posterior distribution, while SGVB is a method used to perform efficient inference and learning in VAEs. We validate the effectiveness of our model through quantitative comparisons with other state-of-the-art deep clustering models on six benchmark datasets. Moreover, due to the generative nature of VAEs, the proposed model can generate highly realistic samples of specific classes without supervised information.

Contrastive learning-based multi-view clustering for incomplete multivariate time series

Incomplete multivariate time series (MTS) clustering is a prevalent research topic in time series analysis, aimed at partitioning MTS containing missing data into distinct clusters. Contrastive learning-based multi-view clustering methods are a promising approach to address this issue. However, existing methods are typically not designed for time series. Specifically, most of these methods struggle to capture the inherent properties of time series, and are susceptible to losing their interdimensional correlations, thereby compromising data integrity. Furthermore, they commonly utilize data augmentation techniques to generate sample pairs for contrastive learning. These existing data augmentation techniques are not suitable for time series, and introduce uncertainty factors, which can diminish the representation learning capacity of contrastive learning. To address the challenges, we propose a contrastive learning-based multi-view clustering method for incomplete multivariate time series (MVCIMTS). In this method, each variable within the MTS is treated as a separate view, enabling a multi-view learning approach. To better leverage the intrinsic information of time series, we utilize a GRU-based model architecture that integrates imputation and clustering within a unified deep learning framework. In this way, missing views can be effectively inferred, and representations suitable for clustering can be learned, thereby enhancing the clustering performance for incomplete time series. Furthermore, we introduce an innovative contrastive learning approach specifically tailored for MTS, which ensures that the exploration of common semantics and clustering consistency across views remains unaffected by uncertainty factors. It assumes that each time series variable within the same sample has similar representations, thereby taking into account the correlation between variables and enhancing the quality of the representations. To the best of our knowledge, this is the first attempt at applying contrastive learning-based multi-view deep clustering to incomplete MTS. We conduct extensive comparative experiments with five multi-view clustering methods and two time series clustering methods on seven benchmark datasets. The results demonstrate that our proposed method is superior to other state-of-the-art methods.

CCEGAN: Enhancing GAN clustering through contrastive clustering ensemble

Clustering algorithms play a crucial role in various domains, and recent advancements in Generative Adversarial Network (GAN) techniques have opened new possibilities for improving clustering effectiveness. This paper aims to enhance the performance of GAN clustering by addressing the challenge of generating high-quality labeled samples. We propose a novel contrastive network and a voting-based method to progressively filter and fuse information from synthetic samples. These methods are incorporated into a deep clustering ensemble framework, which combines the advantages of GAN clustering and ensemble learning. Through comprehensive empirical analysis on diverse datasets, including both image and non-image datasets, we demonstrate the superiority of our proposed method in terms of effectiveness and robustness. Our approach outperforms existing GAN clustering methods while maintaining a reasonable computational time. This work contributes to the field of clustering algorithms by providing a more effective and robust approach for leveraging GANs in the clustering process. The code is available at https://github.com/Jarvisyan/CCEGAN-pytorch.

CDFL: Efficient Federated Human Activity Recognition Using Contrastive Learning and Deep Clustering

CDFL: Efficient Federated Human Activity Recognition Using Contrastive Learning and Deep Clustering

A Comprehensive Survey on Deep Clustering: Taxonomy, Challenges, and Future Directions

Clustering is a fundamental machine learning task, which aim at assigning instances into groups so that similar samples belong to the same cluster while dissimilar samples belong to different clusters. Shallow clustering methods usually assume that data are collected and expressed as feature vectors within which clustering is performed. However, clustering high-dimensional data, such as images, texts, videos, and graphs, poses significant challenges for clustering tasks, such as indiscriminate representation and intricate relationships among instances. Over the past decades, deep learning has achieved remarkable success in effective representation learning and modeling complex relationships. Motivated by these advancements, Deep Clustering seeks to improve clustering outcomes through deep learning techniques, garnering considerable interest from both academia and industry. Despite many contributions to this vibrant area of research, the lack of systematic analysis and a comprehensive taxonomy has hindered progress in this field. In this survey, we first explore how deep learning can be integrated into deep clustering and identify two fundamental components: the representation learning module and the clustering module. Then, we summarize and analyze the representative design of these two modules. Furthermore, we introduce a novel taxonomy of deep clustering based on how these two modules interact, specifically through multistage, generative, iterative, and simultaneous approaches. In addition, we present well-known benchmark datasets, evaluation metrics, and open-source tools to clearly demonstrate different experimental approaches. Finally, we examine the practical applications of deep clustering and propose challenging areas for future research.

Functional imaging derived ADHD biotypes based on deep clustering: a study on personalized medication therapy guidance

Functional imaging derived ADHD biotypes based on deep clustering: a study on personalized medication therapy guidance

Deep Multirepresentation Learning for Data Clustering.

Deep clustering incorporates embedding into clustering in order to find a lower-dimensional space suitable for clustering tasks. Conventional deep clustering methods aim to obtain a single global embedding subspace (aka latent space) for all the data clusters. In contrast, in this article, we propose a deep multirepresentation learning (DML) framework for data clustering whereby each difficult-to-cluster data group is associated with its own distinct optimized latent space and all the easy-to-cluster data groups are associated with a general common latent space. Autoencoders (AEs) are employed for generating cluster-specific and general latent spaces. To specialize each AE in its associated data cluster(s), we propose a novel and effective loss function which consists of weighted reconstruction and clustering losses of the data points, where higher weights are assigned to the samples more probable to belong to the corresponding cluster(s). Experimental results on benchmark datasets demonstrate that the proposed DML framework and loss function outperform state-of-the-art clustering approaches. In addition, the results show that the DML method significantly outperforms the SOTA on imbalanced datasets as a result of assigning an individual latent space to the difficult clusters.

Retraction Note: Deep clustering based on embedded auto-encoder

Retraction Note: Deep clustering based on embedded auto-encoder

An optimized siamese neural network with deep linear graph attention model for gynaecological abdominal pelvic masses classification.

An adnexal mass, also known as a pelvic mass, is a growth that develops in or near the uterus, ovaries, fallopian tubes, and supporting tissues. For women suspected of having ovarian cancer, timely and accurate detection of a malignant pelvic mass is crucial for effective triage, referral, and follow-up therapy. While various deep learning techniques have been proposed for identifying pelvic masses, current methods are often not accurate enough and can be computationally intensive. To address these issues, this manuscript introduces an optimized Siamese circle-inspired neural network with deep linear graph attention (SCINN-DLGN) model designed for pelvic mass classification. The SCINN-DLGN model is intended to classify pelvic masses into three categories: benign, malignant, and healthy. Initially, real-time MRI pelvic mass images undergo pre-processing using semantic-aware structure-preserving median morpho-filtering to enhance image quality. Following this, the region of interest (ROI) within the pelvic mass images is segmented using an EfficientNet-based U-Net framework, which reduces noise and improves the accuracy of segmentation. The segmented images are then analysed using the SCINN-DLGN model, which extracts geometric features from the ROI. These features are classified into benign, malignant, or healthy categories using a deep clustering algorithm integrated into the linear graph attention model. The proposed system is implemented on a Python platform, and its performance is evaluated using real-time MRI pelvic mass datasets. The SCINN-DLGN model achieves an impressive 99.9% accuracy and 99.8% recall, demonstrating superior efficiency compared to existing methods and highlighting its potential for further advancement in the field.

Deep attributed graph clustering with feature consistency contrastive and topology enhanced network

Deep attributed graph clustering has attracted considerable interest lately due to its capability to uncover meaningful latent knowledge from heterogeneous spaces, thereby improving our comprehension of real-world systems. However, ensuring the consistency of the clustering assignments generated from topological and attribute information remains a key issue, which is one of the reasons for the low performance of clustering. To tackle these issues, a novel deep clustering approach with Feature Consistency Contrastive and Topology Enhanced Network (FCC-TEN) is proposed, which consists of GAT and AE that can mine the topological and attributed information and achieve consistency contrastive learning to improve clustering performance. First, a Fusion Graph Convolutional Auto-encoder module is proposed to fuse the attribute information captured by each layer of the AE and enrich topological information for improving the feature extraction capability of AE. Then, using a Feature Consistency Contrastive module to uncover consistency information of the GAT and AE through contrastive learning at the feature and label level. Finally, clustering results are obtained directly by the clustering assignment obtained at the label level. Comprehensive testing on five improved datasets shows that our method provides advanced clustering performance. Moreover, visual analyses of the clustering results corroborate a gradual refinement of the clustering structure, proving the validity of our approach.

A clustering ensemble algorithm for handling deep embeddings using cluster confidence

Abstract Clustering ensemble, which aims to learn a robust consensus clustering from multiple weak base clusterings, has achieved promising performance on various applications. With the development of big data, the scale and complexity of data is constantly increasing. However, most existing clustering ensemble methods typically employ shallow clustering algorithms to generate base clusterings. When confronted with high-dimensional complex data, these shallow algorithms fail to fully utilize the intricate features present in the latent data space. As a result, the quality and diversity of the generated base clusterings are insufficient, thus affecting the subsequent ensemble performance. To address this issue, we propose a novel clustering ensemble algorithm for handling deep embeddings using cluster confidence (CEDECC) to improve the robustness and performance. Instead of simply combining deep clustering with clustering ensembles, we take into consideration that the performance of existing deep clustering methods heavily relies on the quality of low-dimensional embeddings generated during the pre-training stage. The quality of embeddings is unstable due to the influence of different initialization parameters. In CEDECC, specifically, we first construct a cluster confidence measure to evaluate the quality of low-dimensional embeddings. Typically, high-quality low-dimensional embeddings yield accurate clustering results with the same model parameters. Then, we utilize multiple high-quality embeddings to generate the base partitions. In the ensemble strategy phase, we consider the cluster-wise diversity and propose a novel ensemble cluster estimation to improve the overall consensus performance of the model. Extensive experiments on three benchmark datasets and four real-world biological datasets have demonstrated that the proposed CEDECC consistently outperforms the state-of-the-art clustering ensemble methods.

Graph convolutional network for water network partitioning

The partitioning of a water distribution network into several district-metered areas (DMAs) yields is of great benefits for the management of water distribution systems, including enhanced leak control, pollution monitoring, and pressure optimization. However, achieving an effective partitioning that simultaneously incorporates node characteristics and network connections is challenging. Previous studies have tackled water network partitioning by focusing on either node features or pipe connections individually. In response to the limitations of current approaches for water distribution network partitioning, this study introduces an innovative unsupervised clustering approach for node partitioning based on graph convolutional network (GCN) techniques. Initially, the network nodes are clustered using the Cross-Attention Fusion Based Enhanced Graph Convolutional Network (CaEGCN) deep clustering algorithm, considering both the network’s topological layout and hydraulic properties. The resulting clusters categorize the network nodes into distinct regions. We evaluate our approach and other methods through a real world water distribution network using five different metrics commonly used to evaluate applied in assessing water supply network zoning issues. By comparing and analyzing the performance of experiments with other schemes, the experimental results demonstrate that the CaEGCN deep clustering method featured a remarkable performance. The proposed method showcases its potential for practical application by offering a dependable and easily interpretable water network partitioning solution.

A prior knowledge-embedded deformable convolutional network for vision-based fabric defect detection

The vision-based defect detection of textile surface is an essential problem in evaluating the appearance quality. In previous studies on vision-based defect detection, the following two points were neglected: (1) The proportion of defects on fabric is small, resulting in a low signal-to-noise ratio of defect images and a lack of sampling features; (2) the irregular shape of defects can overlap in position, leading to localization difficulty. In this paper, we propose a prior knowledge-embedded deformable convolutional network (PKE-DCNet) based on deep learning to address these two issues. First, a feature extraction method with prior information is designed with a shape-matching deep clustering module and a region-biased sampling module for detecting defects with complex shape. Then, a defect boundary boxes adaptive generation method is proposed with an anchor-free search mechanism and irregular edge contour computation to detect the key points in the region. Extensive experiments with mixed fabric defect datasets demonstrated that PKE-DCNet reached an overall mAP of 95.36% for six types of defects within a detection speed of 322 FPS, which was better than state-of-the-art methods.

Rethinking deep clustering paradigms: Self-supervision is all you need

The recent advances in deep clustering have been made possible by significant progress in self-supervised and pseudo-supervised learning. However, the trade-off between self-supervision and pseudo-supervision can give rise to three primary issues. The joint training causes Feature Randomness and Feature Drift, whereas the independent training causes Feature Randomness and Feature Twist. In essence, using pseudo-labels generates random and unreliable features. The combination of pseudo-supervision and self-supervision drifts the reliable clustering-oriented features. Moreover, moving from self-supervision to pseudo-supervision can twist the curved latent manifolds. This paper addresses the limitations of existing deep clustering paradigms concerning Feature Randomness, Feature Drift, and Feature Twist. We propose a new paradigm with a new strategy that replaces pseudo-supervision with a second round of self-supervision training. The new strategy makes the transition between instance-level self-supervision and neighborhood-level self-supervision smoother and less abrupt. Moreover, it prevents the drifting effect that is caused by the strong competition between instance-level self-supervision and clustering-level pseudo-supervision. Moreover, the absence of the pseudo-supervision prevents the risk of generating random features. With this novel approach, our paper introduces a Rethinking of the Deep Clustering Paradigms, denoted by R-DC. Our model is specifically designed to address three primary challenges encountered in Deep Clustering: Feature Randomness, Feature Drift, and Feature Twist. Experimental results conducted on six datasets have shown that the two-level self-supervision training yields substantial improvements, as evidenced by the results of the clustering and ablation study. Furthermore, experimental comparisons with nine state-of-the-art clustering models have clearly shown that our strategy leads to a significant enhancement in performance.

Deep Clustering With Hybrid-Grained Contrastive and Discriminative Learning

Deep Clustering With Hybrid-Grained Contrastive and Discriminative Learning

Simple Contrastive Graph Clustering.

Contrastive learning has recently attracted plenty of attention in deep graph clustering due to its promising performance. However, complicated data augmentations and time-consuming graph convolutional operations undermine the efficiency of these methods. To solve this problem, we propose a simple contrastive graph clustering (SCGC) algorithm to improve the existing methods from the perspectives of network architecture, data augmentation, and objective function. As to the architecture, our network includes two main parts, that is, preprocessing and network backbone. A simple low-pass denoising operation conducts neighbor information aggregation as an independent preprocessing, and only two multilayer perceptrons (MLPs) are included as the backbone. For data augmentation, instead of introducing complex operations over graphs, we construct two augmented views of the same vertex by designing parameter unshared Siamese encoders and perturbing the node embeddings directly. Finally, as to the objective function, to further improve the clustering performance, a novel cross-view structural consistency objective function is designed to enhance the discriminative capability of the learned network. Extensive experimental results on seven benchmark datasets validate our proposed algorithm's effectiveness and superiority. Significantly, our algorithm outperforms the recent contrastive deep clustering competitors with at least seven times speedup on average. The code of SCGC is released at SCGC. Besides, we share a collection of deep graph clustering, including papers, codes, and datasets at ADGC.

Context-Based Meta-Reinforcement Learning With Bayesian Nonparametric Models.

Deep reinforcement learning agents usually need to collect a large number of interactions to solve a single task. In contrast, meta-reinforcement learning (meta-RL) aims to quickly adapt to new tasks using a small amount of experience by leveraging the knowledge from training on a set of similar tasks. State-of-the-art context-based meta-RL algorithms use the context to encode the task information and train a policy conditioned on the inferred latent task encoding. However, most recent works are limited to parametric tasks, where a handful of variables control the full variation in the task distribution, and also failed to work in non-stationary environments due to the few-shot adaptation setting. To address those limitations, we propose MEta-reinforcement Learning with Task Self-discovery (MELTS), which adaptively learns qualitatively different nonparametric tasks and adapts to new tasks in a zero-shot manner. We introduce a novel deep clustering framework (DPMM-VAE) based on an infinite mixture of Gaussians, which combines the Dirichlet process mixture model (DPMM) and the variational autoencoder (VAE), to simultaneously learn task representations and cluster the tasks in a self-adaptive way. Integrating DPMM-VAE into MELTS enables it to adaptively discover the multi-modal structure of the nonparametric task distribution, which previous methods using isotropic Gaussian random variables cannot model. In addition, we propose a zero-shot adaptation mechanism and a recurrence-based context encoding strategy to improve the data efficiency and make our algorithm applicable in non-stationary environments. On various continuous control tasks with both parametric and nonparametric variations, our algorithm produces a more structured and self-adaptive task latent space and also achieves superior sample efficiency and asymptotic performance compared with state-of-the-art meta-RL algorithms.

A Dual-Masked Deep Structural Clustering Network With Adaptive Bidirectional Information Delivery.

Structured clustering networks, which alleviate the oversmoothing issue by delivering hidden features from autoencoder (AE) to graph convolutional networks (GCNs), involve two shortcomings for the clustering task. For one thing, they used vanilla structure to learn clustering representations without considering feature and structure corruption; for another thing, they exhibit network degradation and vanishing gradient issues after stacking multilayer GCNs. In this article, we propose a clustering method called dual-masked deep structural clustering network (DMDSC) with adaptive bidirectional information delivery (ABID). Specifically, DMDSC enables generative self-supervised learning to mine deeper interstructure and interfeature correlations by simultaneously reconstructing corrupted structures and features. Furthermore, DMDSC develops an ABID module to establish an information transfer channel between each pairwise layer of AE and GCNs to alleviate the oversmoothing and vanishing gradient problems. Numerous experiments on six benchmark datasets have shown that the proposed DMDSC outperforms the most advanced deep clustering algorithms.

NsDCC: dual-level contrastive clustering with nonuniform sampling for scRNA-seq data analysis.

Dimensionality reduction and clustering are crucial tasks in single-cell RNA sequencing (scRNA-seq) data analysis, treated independently in the current process, hindering their mutual benefits. The latest methods jointly optimize these tasks through deep clustering. However, contrastive learning, with powerful representation capability, can bridge the gap that common deep clustering methods face, which requires pre-defined cluster centers. Therefore, a dual-level contrastive clustering method with nonuniform sampling (nsDCC) is proposed for scRNA-seq data analysis. Dual-level contrastive clustering, which combines instance-level contrast and cluster-level contrast, jointly optimizes dimensionality reduction and clustering. Multi-positive contrastive learning and unit matrix constraint are introduced in instance- and cluster-level contrast, respectively. Furthermore, the attention mechanism is introduced to capture inter-cellular information, which is beneficial for clustering. The nsDCC focuses on important samples at category boundaries and in minority categories by the proposed nearest boundary sparsest density weight assignment algorithm, making it capable of capturing comprehensive characteristics against imbalanced datasets. Experimental results show that nsDCC outperforms the six other state-of-the-art methods on both real and simulated scRNA-seq data, validating its performance on dimensionality reduction and clustering of scRNA-seq data, especially for imbalanced data. Simulation experiments demonstrate that nsDCC is insensitive to "dropout events" in scRNA-seq. Finally, cluster differential expressed gene analysis confirms the meaningfulness of results from nsDCC. In summary, nsDCC is a new way of analyzing and understanding scRNA-seq data.