Deep Clustering Research Articles

Phenotyping urban built and natural environments with high-resolution satellite images and unsupervised deep learning

Cities in the developing world are expanding rapidly, and undergoing changes to their roads, buildings, vegetation, and other land use characteristics. Timely data are needed to ensure that urban change enhances health, wellbeing and sustainability. We present and evaluate a novel unsupervised deep clustering method to classify and characterise the complex and multidimensional built and natural environments of cities into interpretable clusters using high-resolution satellite images. We applied our approach to a high-resolution (0.3 m/pixel) satellite image of Accra, Ghana, one of the fastest growing cities in sub-Saharan Africa, and contextualised the results with demographic and environmental data that were not used for clustering. We show that clusters obtained solely from images capture distinct interpretable phenotypes of the urban natural (vegetation and water) and built (building count, size, density, and orientation; length and arrangement of roads) environment, and population, either as a unique defining characteristic (e.g., bodies of water or dense vegetation) or in combination (e.g., buildings surrounded by vegetation or sparsely populated areas intermixed with roads). Clusters that were based on a single defining characteristic were robust to the spatial scale of analysis and the choice of cluster number, whereas those based on a combination of characteristics changed based on scale and number of clusters. The results demonstrate that satellite data and unsupervised deep learning provide a cost-effective, interpretable and scalable approach for real-time tracking of sustainable urban development, especially where traditional environmental and demographic data are limited and infrequent.

Deep Clustering With a Constraint for Topological Invariance Based on Symmetric InfoNCE.

We consider the scenario of deep clustering, in which the available prior knowledge is limited. In this scenario, few existing state-of-the-art deep clustering methods can perform well for both noncomplex topology and complex topology data sets. To address the problem, we propose a constraint utilizing symmetric InfoNCE, which helps an objective of the deep clustering method in the scenario of training the model so as to be efficient for not only noncomplex topology but also complex topology data sets. Additionally, we provide several theoretical explanations of the reason that the constraint can enhances the performance of deep clustering methods. To confirm the effectiveness of the proposed constraint, we introduce a deep clustering method named MIST, which is a combination of an existing deep clustering method and our constraint. Our numerical experiments via MIST demonstrate that the constraint is effective. In addition, MIST outperforms other state-of-the-art deep clustering methods for most of the commonly used 10 benchmark data sets.

ScDFC: A deep fusion clustering method for single-cell RNA-seq data.

Clustering methods have been widely used in single-cell RNA-seq data for investigating tumor heterogeneity. Since traditional clustering methods fail to capture the high-dimension methods, deep clustering methods have drawn increasing attention these years due to their promising strengths on the task. However, existing methods consider either the attribute information of each cell or the structure information between different cells. In other words, they cannot sufficiently make use of all of this information simultaneously. To this end, we propose a novel single-cell deep fusion clustering model, which contains two modules, i.e. an attributed feature clustering module and a structure-attention feature clustering module. More concretely, two elegantly designed autoencoders are built to handle both features regardless of their data types. Experiments have demonstrated the validity of the proposed approach, showing that it is efficient to fuse attributes, structure, and attention information on single-cell RNA-seq data. This work will be further beneficial for investigating cell subpopulations and tumor microenvironment. The Python implementation of our work is now freely available at https://github.com/DayuHuu/scDFC.

Conditional deep clustering based transformed spatio-temporal features and fused distance for efficient video retrieval

Conditional deep clustering based transformed spatio-temporal features and fused distance for efficient video retrieval

Microservice extraction using graph deep clustering based on dual view fusion

With the increasing scale of software, traditional monolithic architecture applications are challenging to maintain and scale on cloud platforms. Many companies increasingly adopt microservices architecture as a more flexible choice. However, microservice migration is still challenging due to the lack of higher-quality microservice extraction methods. Traditional microservice extraction methods cannot effectively combine the structural dependency and business functions of monolithic applications; thus, their performance warrants improvement. This paper proposes a graph deep clustering method based on dual view fusion (GDC-DVF) for microservice extraction. GDC-DVF constructs a graph of invocation relationships between classes, which is the structural dependency view, using the runtime trace data of a monolithic application. Then the business function view is constructed by the random walk algorithm and uniform random sampling using the structural dependency view. Next, the fused node feature embedding representations of the two views are learned using a graph encoder based on a graph attention adaptive residual network. Clustering is performed on the fused feature embedding representations to obtain microservice extraction proposals. GDC-DVF is tested on four open-source monolithic applications and achieves better performance compared with comparison methods. Experimental results show that GDC-DVF can extract high-quality microservice collections and validate the effectiveness and scalability of the graph neural network (GNN) for microservice extraction problems.

Learning Representation for Clustering Via Prototype Scattering and Positive Sampling.

Existing deep clustering methods rely on either contrastive or non-contrastive representation learning for downstream clustering task. Contrastive-based methods thanks to negative pairs learn uniform representations for clustering, in which negative pairs, however, may inevitably lead to the class collision issue and consequently compromise the clustering performance. Non-contrastive-based methods, on the other hand, avoid class collision issue, but the resulting non-uniform representations may cause the collapse of clustering. To enjoy the strengths of both worlds, this paper presents a novel end-to-end deep clustering method with prototype scattering and positive sampling, termed ProPos. Specifically, we first maximize the distance between prototypical representations, named prototype scattering loss, which improves the uniformity of representations. Second, we align one augmented view of instance with the sampled neighbors of another view-assumed to be truly positive pair in the embedding space-to improve the within-cluster compactness, termed positive sampling alignment. The strengths of ProPos are avoidable class collision issue, uniform representations, well-separated clusters, and within-cluster compactness. By optimizing ProPos in an end-to-end expectation-maximization framework, extensive experimental results demonstrate that ProPos achieves competing performance on moderate-scale clustering benchmark datasets and establishes new state-of-the-art performance on large-scale datasets. Source code is available at https://github.com/Hzzone/ProPos.

Deep Clustering by Graph Attention Contrastive Learning

Contrastive learning shows great potential in deep clustering. It uses constructed pairs to discover the feature distribution that is required for the clustering task. In addition to conventional augmented pairs, recent methods have introduced more methods of creating highly confident pairs, such as nearest neighbors, to provide more semantic prior knowledge. However, existing works only use partial pairwise similarities to construct semantic pairs locally without capturing the entire sample’s relationships from a global perspective. In this paper, we propose a novel clustering framework called graph attention contrastive learning (GACL) to aggregate more semantic information. To this end, GACL is designed to simultaneously perform instance-level and graph-level contrast. Specifically, with its novel graph attention mechanism, our model explores more undiscovered pairs and selectively focuses on informative pairs. To ensure local and global clustering consistency, we jointly use the designed graph-level and instance-level contrastive losses. Experiments on six challenging image benchmarks demonstrate the superiority of our proposed approach over state-of-the-art methods.

Automatic Deep Sparse Multi-Trial Vector-based Differential Evolution clustering with manifold learning and incremental technique

Most deep clustering methods despite utilizing complex networks to learn better from data, use a shallow clustering method. These methods have difficulty in finding good clusters due to the lack of ability to handle between local search and global search to prevent premature convergence. In other words, they do not consider different aspects of the search and it causes them to get stuck in the local optimum. In addition, the majority of existing deep clustering approaches perform clustering with the knowledge of the number of clusters, which is not practical in most real scenarios where such information is not available. To address these problems, this paper presents a novel automatic deep sparse clustering approach based on an evolutionary algorithm called Multi-Trial Vector-based Differential Evolution (MTDE). Sparse auto-encoder is first applied to extract embedded features. Manifold learning is then adopted to obtain representation and extract the spatial structure of features. Afterward, MTDE clustering is performed without prior information on the number of clusters to find the optimal clustering solution. The proposed approach was evaluated on various datasets, including images and time-series. The results demonstrate that the proposed method improved MTDE by 18.94% on average and compared to the most recent deep clustering algorithms, is consistently among the top three in the majority of datasets.

Classification of Anomaly Patterns in Integrated Energy Systems Based on Conditional Variational Autoencoder and Attention Mechanism

By studying the classification of anomaly patterns in integrated energy systems, a deeper understanding of their operational status can be gained, leading to improved reliability and efficiency. This can ultimately result in reduced energy consumption and carbon emissions, contributing to sustainability efforts. This paper proposes a classification method that employs a conditional variational autoencoder and attention mechanism for deep clustering to identify anomaly patterns and distinguish between normal and anomaly datasets. The proposed model effectively addresses the issue of detecting anomalous periods in traditional anomaly detection methods. To account for the periodicity and coupling relationships of different loads, the model integrates sliding windows and attention mechanisms to improve the accuracy of detecting anomaly patterns. Firstly, during the encoder stage, a spatial attention mechanism is incorporated to extract features at each time step of the model input. Secondly, during the decoder stage, a temporal attention mechanism is introduced to perform feature extraction among the multiple time-step hidden layer states of the model input. The proposed method is applied to a typical integrated energy system and compared with existing methods. The experimental results demonstrate the effectiveness of the proposed method in accurately classifying the normal and anomaly patterns of integrated energy systems due to the internal clustering evaluation index.

Deep clustering for abdominal organ classification in ultrasound imaging.

The purpose of this study is to examine the utilization of unlabeled data for abdominal organ classification in multi-label (non-mutually exclusive classes) ultrasound images, as an alternative to the conventional transfer learning approach. We present a new method for classifying abdominal organs in ultrasound images. Unlike previous approaches that only relied on labeled data, we consider the use of both labeled and unlabeled data. To explore this approach, we first examine the application of deep clustering for pretraining a classification model. We then compare two training methods, fine-tuning with labeled data through supervised learning and fine-tuning with both labeled and unlabeled data using semisupervised learning. All experiments were conducted on a large dataset of unlabeled images () and a small set of labeled images () comprising progressively 10%, 20%, 50%, and 100% of the images. We show that for supervised fine-tuning, deep clustering is an effective pre-training method, with performance matching that of ImageNet pre-training using five times less labeled data. For semi-supervised learning, deep clustering pre-training also yields higher performance when the amount of labeled data is limited. Best performance is obtained with deep clustering pre-training combined with semi-supervised learning and 2742 labeled example images with an -score weighted average of 84.1%. This method can be used as a tool to preprocess large unprocessed databases, thus reducing the need for prior annotations of abdominal ultrasound studies for the training of image classification algorithms, which in turn could improve the clinical use of ultrasound images.

Model adaptation via credible local context representation

AbstractConventional model transfer techniques, requiring the labelled source data, are not applicable in the privacy‐protected medical fields. For the challenging scenarios, recent source data‐free domain adaptation (SFDA) has become a mainstream solution but losing focus on the inter‐sample class information. This paper proposes a new Credible Local Context Representation approach for SFDA. Our main idea is to exploit the credible local context for more discriminative representation. Specifically, we enhance the source model's discrimination by information regulating. To capture the context, a discovery method is developed that performs fixed steps walking in deep space and takes the credible features in this path as the context. In the epoch‐wise adaptation, deep clustering‐like training is conducted with two major updates. First, the context for all target data is constructed and then the context‐fused pseudo‐labels providing semantic guidance are generated. Second, for each target data, a weighting fusion on its context forms the anchored neighbourhood structure; thus, the deep clustering is switched from individual‐based to coarse‐grained. Also, a new regularisation building is developed on the anchored neighbourhood to drive the deep coarse‐grained learning. Experiments on three benchmarks indicate that the proposed method can achieve state‐of‐the‐art results.

A Novel Intrusion Detection System for Internet of Healthcare Things Based on Deep Subclasses Dispersion Information

Despite the significant benefits that the Internet of Healthcare Things (IoHT) offered to the medical sector, there are concerns and risks regarding these systems which can delay their wide deployment as they handle sensitive and often life-critical medical information. In addition to the IoHT concerns and risks, there are security constraints which include hardware, software and network constraints that pose a security challenge to these systems. Therefore, security measures need to be deployed that can overcome the concerns, mitigate the risks and meet the constraints of the IoHT. For these reasons, a subclasses intrusion detection system for the IoHT is proposed in this research work based on a novel variation of the standard One-class Support Vector Machine (OSVM), namely Deep Subclass Dispersion One-class Support Vector Machine (Deep SDOSVM), which considers subclasses in the target class, i.e. normal class, in order to minimize the data dispersion within and between subclasses, thereby improving the discriminative power and classification performance of the intrusion detection system. A deep clustering model is used for subclasses generation in the proposed Deep SDOSVM approach, namely the Dynamic Autoencoder Model (DynAE), to overcome the drawbacks of the classical clustering algorithms and further enhance the classification performance of the intrusion detection system. The proposed deep clustering subclasses intrusion detection system was evaluated on the real-world TON_IoT dataset and compared to other state-of-the-art one-class classifiers. Experimentation results have shown that the proposed approach outperformed the other relevant one-class classifiers for network intrusion detection.

Self-paced deep clustering with learning loss

Deep clustering has become a popular technique in various fields due to its superior performance over conventional clustering. However, it can be challenging to classify data located near the decision boundary, and difficult or noisy samples can confuse or mislead the training process of deep neural networks, ultimately impacting clustering performance. To address this issue, we propose a novel self-paced deep clustering method that gradually increases the number of samples input into the network from easy to difficult. Our approach involves attaching a loss prediction module to the convolutional neural network to judge the difficulty of samples. The module is robust as it relies on the input contents, not statistical estimates of uncertainty from outputs. Moreover, this module selects the most informative data for the training model, thus enabling the network to converge stably and reach a good optimal solution. Finally, we consider the reconstruction error, clustering loss, and loss prediction error to construct the loss of the model. Experimental results on four image datasets show that our method outperforms state-of-the-art works. The main code can be found at https://github.com/clustering106/SDCLL.

Deep joint optimized clustering model for life cycle identification of enterprise electricity consumption

Motivation analysis is significant in inspecting illegal power consumption behavior. Based on enterprise life cycle theory, this study proposes enterprise electricity consumption life cycle to conceptually aid the analysis of motives for enterprise illegal power consumption behavior. The identification of the aforementioned life cycle is achieved by clustering. We propose a deep clustering model based on the joint optimization of Gated Recurrent Unit (GRU) autoencoder and an improved K-prototypes algorithm. The proposed model utilizes GRU autoencoder to learn the latent features of high-dimensional enterprise electricity data and dynamically adjust the distance metric in K-prototypes algorithm based on the reconstruction error of GRU autoencoder. By jointly optimizing the two components, the proposed model can effectively cluster the enterprise electricity consumers and enterprises’ electricity consumption life cycle is identified based on the features and patterns of enterprises in each cluster. Experiment demonstrates that the proposed model can effectively capture the non-linear transformation from electricity data to its latent space and yield superior clustering results, and can provide data support for the identification of enterprise electricity consumption life cycle.

Clustering Single-Cell RNA Sequence Data Using Information Maximized and Noise-Invariant Representations.

Single-cell RNA sequencing (scRNA-seq) is a revolutionary methodology that helps to analyze transcriptome or genome information from a single cell. However, high dimensionality and sparsity in data due to dropout events pose computational challenges for existing state-of-the-art scRNA-seq clustering methods. Learning efficient representations becomes even more challenging due to the presence of noise in scRNA-seq data. To overcome the effect of noise and learn effective representations, this paper proposes sc-INDC (Single-Cell Information Maximized Noise-Invariant Deep Clustering), a deep neural network that facilitates learning of informative and noise-invariant representations of scRNA-seq data. Furthermore, the time complexity of the proposed sc-INDC is significantly lower compared to state-of-the-art scRNA-seq clustering methods. Extensive experimentation on fourteen publicly available scRNA-seq datasets illustrates the efficacy of the proposed model. Additionally, visualizations of t-SNE plots and several ablation studies are also conducted to provide insights into the improved representation ability of sc-INDC. Code of the proposed sc-INDC will be available at: https://github.com/arnabkmondal/sc-INDC.

Semi-Supervised Medical Image Classification Combined with Unsupervised Deep Clustering

An effective way to improve the performance of deep neural networks in most computer vision tasks is to improve the quantity of labeled data and the quality of labels. However, in the analysis and processing of medical images, high-quality annotation depends on the experience and professional knowledge of experts, which makes it very difficult to obtain a large number of high-quality annotations. Therefore, we propose a new semi-supervised framework for medical image classification. It combines semi-supervised classification with unsupervised deep clustering. Spreading label information to unlabeled data by alternately running two tasks helps the model to extract semantic information from unlabeled data, and prevents the model from overfitting to a small amount of labeled data. Compared with current methods, our framework enhances the robustness of the model and reduces the influence of outliers. We conducted a comparative experiment on the public benchmark medical image dataset to verify our method. On the ISIC 2018 Dataset, our method surpasses other methods by more than 0.85% on AUC and 1.08% on Sensitivity. On the ICIAR BACH 2018 dataset, our method achieved 94.12% AUC, 77.92% F1-score, 77.69% Recall, and 78.16% Precision. The error rate is at least 1.76% lower than that of other methods. The result shows the effectiveness of our method in medical image classification.

Semi-supervised deep embedded clustering with pairwise constraints and subset allocation

Semi-supervised deep clustering methods attract much attention due to their excellent performance on the end-to-end clustering task. However, it is hard to obtain satisfying clustering results since many overlapping samples in industrial text datasets strongly and incorrectly influence the learning process. Existing methods incorporate prior knowledge in the form of pairwise constraints or class labels, which not only largely ignore the correlation between these two supervision information but also cause the problem of weak-supervised constraint or incorrect strong-supervised label guidance. In order to tackle these problems, we propose a semi-supervised method based on pairwise constraints and subset allocation (PCSA-DEC). We redefine the similarity-based constraint loss by forcing the similarity of samples in the same class much higher than other samples and design a novel subset allocation loss to precisely learn strong-supervised information contained in labels which consistent with unlabeled data. Experimental results on the two industrial text datasets show that our method can yield 8.2%–8.7% improvement in accuracy and 13.4%–19.8% on normalized mutual information over the state-of-the-art method.

Deep Clustering Survival Machines with Interpretable Expert Distributions.

We develop deep clustering survival machines to simultaneously predict survival information and characterize data heterogeneity that is not typically modeled by conventional survival analysis methods. By modeling timing information of survival data generatively with a mixture of parametric distributions, referred to as expert distributions, our method learns weights of the expert distributions for individual instances based on their features discriminatively such that each instance's survival information can be characterized by a weighted combination of the learned expert distributions. Extensive experiments on both real and synthetic datasets have demonstrated that our method is capable of obtaining promising clustering results and competitive time-to-event predicting performance.

Flight risk evaluation based on flight state deep clustering network

Flight risk evaluation based on data-driven approach is an essential topic of aviation safety management. Existing risk analysis methods ignore the coupling and time-variant characteristics of flight parameters, and cannot accurately establish the mapping relationship between flight state and loss-of-control risk. To deal with the problem, a flight state deep clustering network (FSDCN) model was proposed to mine latent loss-of-control risk information implicating in raw flight parameters. FSDCN integrates the feature extraction and clustering into an end-to-end deep hybrid network to extract latent risk features from multivariate time-series flight parameters and cluster them. In the FSDCN model, a sequential multi-attention encoder–decoder network is designed to extract embedded risk features, and the feature clustering layer is designed to iteratively refine clustering effects and feature extraction. Besides, a loss-of-control classifier is added to optimize the risk feature vector expression and ensure sufficient dividing feature for facilitate clustering. The multi-task joint learning strategy is adopted to improve the clustering performance of the model further. According to extracted risk features and similarity metrics, the optimal clusters number of flight states is set as 5. Comparative experiments show that FSDCN significantly performs better than other clustering models with performance percentage error below 6%. Through statistical analysis of clustering results, the risk level is quantified for each cluster. Three high-difficulty maneuver cases are presented to demonstrate FSDCN for flight risk evaluation. The flight parameter sequences of the maneuver cases are input into the well-trained FSDCN to obtain the risk prediction results. The spatiotemporal distribution characteristics of the risk-quantized results are consistent with flight parameters over-limit situations, which demonstrates the effectiveness of FSDCN on clustering flight states. The experimental results on flight maneuver cases show that FSDCN can find potential loss-of-control risk features according to multivariate time-series flight data and provide support for in-flight risk warnings.

Neural clustering based on implicit maximum likelihood

Clustering is one of the most fundamental unsupervised learning tasks with numerous applications in various fields. Clustering methods based on neural networks, called deep clustering methods, leverage the representational power of neural networks to enhance clustering performance. ClusterGan constitutes a generative deep clustering method that exploits generative adversarial networks (GANs) to perform clustering. However, it inherits some deficiencies of GANs, such as mode collapse, vanishing gradients and training instability. In order to tackle those deficiencies, the generative approach of implicit maximum likelihood estimation (IMLE) has been recently proposed. In this paper, we present a clustering method based on generative neural networks, called neural implicit maximum likelihood clustering, which adopts ideas from both ClusterGAN and IMLE. The proposed method has been compared with ClusterGAN and other neural clustering methods on both synthetic and real datasets, demonstrating promising results.