Deep Clustering Research Articles

Maximizing bi-mutual information of features for self-supervised deep clustering

Self-supervised learning based on mutual information makes good use of classification models and label information produced by clustering tasks to train networks parameters, and then updates the downstream clustering assignment with respect to maximizing mutual information between label information. This kind of methods have attracted more and more attention and obtained better progress, but there is still a larger improvement space compared with the methods of supervised learning, especially on the challenge image datasets. To this end, a self-supervised deep clustering method by maximizing mutual information is proposed (bi-MIM-SSC), where deep convolutional network is employed as a feature encoder. The first term is to maximize mutual information between output-feature pairs for importing more semantic meaning to the output features. The second term is to maximize mutual information between an input image and its feature generated by the encoder for keeping the useful information of an original image in latent space as possible. Furthermore, pre-training is carried out to further enhance the representation ability of the encoder, and the auxiliary over-clustering is added in clustering network. The performance of the proposed method bi-MIM-SSC is compared with other clustering methods on the CIFAR10, CIFAR100 and STL10 datasets. Experimental results demonstrate that the proposed bi-MIM-SSC method has better feature representation ability and provide better clustering results.

Hypergraph-Supervised Deep Subspace Clustering

Auto-encoder (AE)-based deep subspace clustering (DSC) methods aim to partition high-dimensional data into underlying clusters, where each cluster corresponds to a subspace. As a standard module in current AE-based DSC, the self-reconstruction cost plays an essential role in regularizing the feature learning. However, the self-reconstruction adversely affects the discriminative feature learning of AE, thereby hampering the downstream subspace clustering. To address this issue, we propose a hypergraph-supervised reconstruction to replace the self-reconstruction. Specifically, instead of enforcing the decoder in the AE to merely reconstruct samples themselves, the hypergraph-supervised reconstruction encourages reconstructing samples according to their high-order neighborhood relations. By the back-propagation training, the hypergraph-supervised reconstruction cost enables the deep AE to capture the high-order structure information among samples, facilitating the discriminative feature learning and, thus, alleviating the adverse effect of the self-reconstruction cost. Compared to current DSC methods, relying on the self-reconstruction, our method has achieved consistent performance improvement on benchmark high-dimensional datasets.

Deep Time-Series Clustering: A Review

We present a comprehensive, detailed review of time-series data analysis, with emphasis on deep time-series clustering (DTSC), and a case study in the context of movement behavior clustering utilizing the deep clustering method. Specifically, we modified the DCAE architectures to suit time-series data at the time of our prior deep clustering work. Lately, several works have been carried out on deep clustering of time-series data. We also review these works and identify state-of-the-art, as well as present an outlook on this important field of DTSC from five important perspectives.

GW-DC: A Deep Clustering Model Leveraging Two-Dimensional Image Transformation and Enhancement

Traditional time-series clustering methods usually perform poorly on high-dimensional data. However, image clustering using deep learning methods can complete image annotation and searches in large image databases well. Therefore, this study aimed to propose a deep clustering model named GW_DC to convert one-dimensional time-series into two-dimensional images and improve cluster performance for algorithm users. The proposed GW_DC consisted of three processing stages: the image conversion stage, image enhancement stage, and image clustering stage. In the image conversion stage, the time series were converted into four kinds of two-dimensional images by different algorithms, including grayscale images, recurrence plot images, Markov transition field images, and Gramian Angular Difference Field images; this last one was considered to be the best by comparison. In the image enhancement stage, the signal components of two-dimensional images were extracted and processed by wavelet transform to denoise and enhance texture features. Meanwhile, a deep clustering network, combining convolutional neural networks with K-Means, was designed for well-learning characteristics and clustering according to the aforementioned enhanced images. Finally, six UCR datasets were adopted to assess the performance of models. The results showed that the proposed GW_DC model provided better results.

Image deep clustering based on local-topology embedding

• The feature representation consists of two parts: data and its local-topology information. • We propose a replacement strategy to find local-topology representation of data. • Data augmentation is applied to improve the generalization performance of the model. • A two-stage image deep clustering algorithm is presented based on local-topology embedding called ITEC. Reasonable feature representation plays an important role in improving the performance of clustering algorithms. However, recent deep clustering studies only focusing on feature representation at the pixel level leads to feature representation with low discrimination. Our key insight is that considering local-topology information between images would help to get a highly discriminative representation, and therefore we design a replacement strategy to find local-topology representation of data, and propose a two-stage image deep clustering algorithm based on local-topology embedding called ITEC. Specifically, we take advantage of data augmentation technique to improve the generalization performance of the learning models; then local-topology representation of data is embedded into the representation of data itself, so as to better complete tasks of image clustering. Extensive experiments demonstrate that local-topology information effectively promotes the performance of deep clustering significantly.

Significant DBSCAN+: Statistically Robust Density-based Clustering

Cluster detection is important and widely used in a variety of applications, including public health, public safety, transportation, and so on. Given a collection of data points, we aim to detect density-connected spatial clusters with varying geometric shapes and densities, under the constraint that the clusters are statistically significant. The problem is challenging, because many societal applications and domain science studies have low tolerance for spurious results, and clusters may have arbitrary shapes and varying densities. As a classical topic in data mining and learning, a myriad of techniques have been developed to detect clusters with both varying shapes and densities (e.g., density-based, hierarchical, spectral, or deep clustering methods). However, the vast majority of these techniques do not consider statistical rigor and are susceptible to detecting spurious clusters formed as a result of natural randomness. On the other hand, scan statistic approaches explicitly control the rate of spurious results, but they typically assume a single “hotspot” of over-density and many rely on further assumptions such as a tessellated input space. To unite the strengths of both lines of work, we propose a statistically robust formulation of a multi-scale DBSCAN, namely Significant DBSCAN+, to identify significant clusters that are density connected. As we will show, incorporation of statistical rigor is a powerful mechanism that allows the new Significant DBSCAN+ to outperform state-of-the-art clustering techniques in various scenarios. We also propose computational enhancements to speed-up the proposed approach. Experiment results show that Significant DBSCAN+ can simultaneously improve the success rate of true cluster detection (e.g., 10–20% increases in absolute F1 scores) and substantially reduce the rate of spurious results (e.g., from thousands/hundreds of spurious detections to none or just a few across 100 datasets), and the acceleration methods can improve the efficiency for both clustered and non-clustered data.

Deep Convolutional Neural Network with KNN Regression for Automatic Image Annotation

Automatic image annotation is an active field of research in which a set of annotations are automatically assigned to images based on their content. In literature, some works opted for handcrafted features and manual approaches of linking concepts to images, whereas some others involved convolutional neural networks (CNNs) as black boxes to solve the problem without external interference. In this work, we introduce a hybrid approach that combines the advantages of both CNN and the conventional concept-to-image assignment approaches. J-image segmentation (JSEG) is firstly used to segment the image into a set of homogeneous regions, then a CNN is employed to produce a rich feature descriptor per area, and then, vector of locally aggregated descriptors (VLAD) is applied to the extracted features to generate compact and unified descriptors. Thereafter, the not too deep clustering (N2D clustering) algorithm is performed to define local manifolds constituting the feature space, and finally, the semantic relatedness is calculated for both image–concept and concept–concept using KNN regression to better grasp the meaning of concepts and how they relate. Through a comprehensive experimental evaluation, our method has indicated a superiority over a wide range of recent related works by yielding F1 scores of 58.89% and 80.24% with the datasets Corel 5k and MSRC v2, respectively. Additionally, it demonstrated a relatively high capacity of learning more concepts with higher accuracy, which results in N+ of 212 and 22 with the datasets Corel 5k and MSRC v2, respectively.

Unsupervised deep clustering via contractive feature representation and focal loss

Deep clustering aims to promote clustering tasks by combining deep learning and clustering together to learn the clustering-oriented representation, and many approaches have shown their validity. However, the feature learning modules in existing methods hardly learn a discriminative representation. In addition, the label assignment mechanism becomes inefficient when dealing with some hard samples. To address these issues, a new joint optimization clustering framework is proposed through introducing the contractive representation in feature learning and utilizing focal loss in the clustering layer. The contractive penalty term added in feature learning would cause the local feature space contraction, resulting in learning more discriminative features. To our certain knowledge, this is also the first work to utilize the focal loss to improve the label assignment in deep clustering method. Moreover, the construction of the joint optimization framework enables the proposed method to learn feature representation and label assignment simultaneously in an end-to-end way. Finally, we comprehensively compare with some state-of-the-art clustering approaches on several clustering tasks to demonstrate the effectiveness of the proposed method.

End-to-end deep representation learning for time series clustering: a comparative study

Time series are ubiquitous in data mining applications. Similar to other types of data, annotations can be challenging to acquire, thus preventing from training time series classification models. In this context, clustering methods can be an appropriate alternative as they create homogeneous groups allowing a better analysis of the data structure. Time series clustering has been investigated for many years and multiple approaches have already been proposed. Following the advent of deep learning in computer vision, researchers recently started to study the use of deep clustering to cluster time series data. The existing approaches mostly rely on representation learning (imported from computer vision), which consists of learning a representation of the data and performing the clustering task using this new representation. The goal of this paper is to provide a careful study and an experimental comparison of the existing literature on time series representation learning for deep clustering. In this paper, we went beyond the sole comparison of existing approaches and proposed to decompose deep clustering methods into three main components: (1) network architecture, (2) pretext loss, and (3) clustering loss. We evaluated all combinations of these components (totaling 300 different models) with the objective to study their relative influence on the clustering performance. We also experimentally compared the most efficient combinations we identified with existing non-deep clustering methods. Experiments were performed using the largest repository of time series datasets (the UCR/UEA archive) composed of 128 univariate and 30 multivariate datasets. Finally, we proposed an extension of the class activation maps method to the unsupervised case which allows to identify patterns providing highlights on how the network clustered the time series.

Multi-Class Cell Detection Using Spatial Context Representation.

In digital pathology, both detection and classification of cells are important for automatic diagnostic and prognostic tasks. Classifying cells into subtypes, such as tumor cells, lymphocytes or stromal cells is particularly challenging. Existing methods focus on morphological appearance of individual cells, whereas in practice pathologists often infer cell classes through their spatial context. In this paper, we propose a novel method for both detection and classification that explicitly incorporates spatial contextual information. We use the spatial statistical function to describe local density in both a multi-class and a multi-scale manner. Through representation learning and deep clustering techniques, we learn advanced cell representation with both appearance and spatial context. On various benchmarks, our method achieves better performance than state-of-the-arts, especially on the classification task. We also create a new dataset for multi-class cell detection and classification in breast cancer and we make both our code and data publicly available.

ADCAS: Adversarial Deep Clustering of Android Streams

The data sequences which are used in malware analysis can be attacked in many applications. However, adversarial attacks are rarely regarded in these types of data. In this paper, a deep learning-based malware clustering approach for sequential data was proposed and the impact of deploying adversarial attacks on it was investigated. An input data stream of android applications was considered as a sequence and the proposed method was tested with the extracted static features of android applications. Three android benchmark datasets, Drebin, Genome, and Contagio, are used to assess the proposed approach. In most experiments, the False Positive Rate (FPR) values of deep clustering algorithms increase to over 60% after the attack, according to the obtained results. Also, the accuracy rates drop to less than 83% in all cases. But by applying the proposed defense method the FPR values reduced while accuracy rates increased.

IAE-ClusterGAN: A new Inverse autoencoder for Generative Adversarial Attention Clustering network

Clustering is a challenging and crucial task in unsupervised learning. Recently, though many clustering algorithms combined with deep learning have been proposed, we observe that the existing deep clustering algorithms do not considerably preserve the clustering structure and information of raw data in the learned latent space. To address this issue, we propose a Generative Adversarial Attention Clustering network Based on Inverse autoencoder (IAE-ClusterGAN), which can control the distribution type of the learned latent code without additional constraints so that unsupervised clustering tasks can be done efficiently. Meanwhile, we integrate the attention mechanism into the network to make the latent code contain more useful clustering information. Moreover, we utilize hyperspherical mapping in the discriminator to improve the stability of model training and reduce the training parameters. Experimental results demonstrate that IAE-ClusterGAN achieves competitive results compared to the state-of-the-art models on five benchmark datasets.

A Novel Deep Clustering Method and Indicator for Time Series Soft Partitioning

The aerospace industry develops prognosis and health management algorithms to ensure better safety on board, particularly for in-flight controls where jamming is dreaded. For that, vibration signals are monitored to predict future defect occurrences. However, time series are not labeled according to severity level, and the user can only assess the system health from the data mining procedure. To that extent, a clustering algorithm using a deep neural network core is developed. Time series are encoded into pictures to be fed into an artificially trained neural network: U-NET. From the segmented output, one-dimensional information on cluster frontiers is extracted and filtered without any parameter selection. Then, a kernel density estimation finally transforms the signal into an empirical density. Ultimately, a Gaussian mixture model extracts the latter independent components. The method empowered us to reveal different degrees of severity faults in the studied data, with their respective likelihoods, without prior knowledge. It was then compared to state-of-the-art machine learning algorithms. However, internal clustering results evaluation for time series is an open question. As the state-of-the-art indexes were not producing relevant results, a new indicator was built to fulfill this task. We applied the whole method to an actuator consisting of an induction machine linked to a ball screw. This study lays the groundwork for future training of diagnosis and prognosis structures in the health management framework.

Unsupervised Deep Clustering of Seismic Data: Monitoring the Ross Ice Shelf, Antarctica

AbstractAdvances in machine learning (ML) techniques and computational capacity have yielded state‐of‐the‐art methodologies for processing, sorting, and analyzing large seismic data sets. In this study, we consider an application of ML for automatically identifying dominant types of impulsive seismicity contained in observations from a 34‐station broadband seismic array deployed on the Ross Ice Shelf (RIS), Antarctica from 2014 to 2017. The RIS seismic data contain signals and noise generated by many glaciological processes that are useful for monitoring the integrity and dynamics of ice shelves. Deep clustering was employed to efficiently investigate these signals. Deep clustering automatically groups signals into hypothetical classes without the need for manual labeling, allowing for the comparison of their signal characteristics and spatial and temporal distribution with potential source mechanisms. The method uses spectrograms as input and encodes their salient features into a lower‐dimensional latent representation using an autoencoder, a type of deep neural network. For comparison, two clustering methods are applied to the latent data: a Gaussian mixture model (GMM) and deep embedded clustering (DEC). Eight classes of dominant seismic signals were identified and compared with environmental data such as temperature, wind speed, tides, and sea ice concentration. The greatest seismicity levels occurred at the RIS front during the 2016 El Niño summer, and near grounding zones near the front throughout the deployment. We demonstrate the spatial and temporal association of certain classes of seismicity with seasonal changes at the RIS front, and with tidally driven seismicity at Roosevelt Island.

Deep Clustering Algorithm Based on Denoising and Self-Attention

Deep Clustering Algorithm Based on Denoising and Self-Attention

Unsupervised embedded feature learning for deep clustering with stacked sparse auto-encoder

Deep clustering attempts to capture the feature representation that benefits the clustering issue. Although the existing deep clustering methods have achieved encouraging performance in many research fields, there still present some shortcomings, such as the lack of consideration of local structure retention and sparse characteristics of input data. To this end, we propose a deep stacked sparse embedded clustering method in this paper, which considers both the local structure preservation and sparse property of inputs. The proposed method is trained to capture the feature representation for an input data by the guidance of the clustering and reconstruction loss, where the reconstruction loss prevents the corruption of feature space and preserve the local structure. Besides, sparse constraint is added to the encoder to avoid learning of unimportant features. Through simultaneously minimizing the reconstruction and clustering loss, the proposed method is able to jointly learn the clustering oriented features and optimize the assignment of cluster labels. Then we conduct amounts of comparative experiments, which consists of seven clustering methods and six publicly available image data sets. Eventually, comprehensive experiments validate the effectiveness of introducing sparse property and preserving local structure in the proposed method.

Vertical Niche Partitioning of Archaea and Bacteria Linked to Shifts in Dissolved Organic Matter Quality and Hydrography in North Atlantic Waters

Understanding the factors that modulate prokaryotic assemblages and their niche partitioning in marine environments is a longstanding challenge in marine microbial ecology. This study analyzes amplicon sequence variant (ASV) diversity and co-occurrence of prokaryotic (Archaea and Bacteria) communities through coastal-oceanic gradients in the NW Iberian upwelling system and adjacent open-ocean (Atlantic Ocean). Biogeographic patterns were investigated in relation with environmental conditions, mainly focusing on the optical signature of the dissolved organic matter (DOM). Alpha- and beta-diversity were horizontally homogeneous [with the only exception of Archaea (∼1700 m depth), attributed to the influence of Mediterranean water, MW], while beta-diversity was significantly vertically stratified. Prokaryotic communities were structured in four clusters (upper subsurface, lower subsurface, intermediate, and deep clusters). Deep (>2000 m) archaeal and bacterial assemblages, and intermediate (500-2000 m) Bacteria (mainly SAR202 and SAR406), were significantly related to humic-like DOM (FDOM-M), while intermediate Archaea were additionally related to biogeochemical attributes of the high-salinity signature of MW. Lower subsurface (100-500 m) Archaea (particularly one ASV belonging to the genus Candidatus Nitrosopelagicus) were mainly related to the imprint of high-salinity MW, while upper subsurface (≤100 m) archaeal assemblages (particularly some ASVs belonging to Marine Group II) were linked to protein-like DOM (aCDOM254). Conversely, both upper and lower subsurface bacterial assemblages were mainly linked to aCDOM254 (particularly ASVs belonging to Rhodobacteraceae, Cyanobacteria, and Flavobacteriaceae) and nitrite concentration (mainly members of Planctomycetes). Most importantly, our analysis unveiled depth-ecotypes, such as the ASVs MarG.II_1 belonging to the archaeal deep cluster (linked to FDOM-M) and MarG.II_2 belonging to the upper subsurface cluster (related to FDOM-T and aCDOM254). This result strongly suggests DOM-mediated vertical niche differentiation, with further implications for ecosystem functioning. Similarly, positive and negative co-occurrence relationships also suggested niche partitioning (e.g., between the closely related ASVs Thaum._Nit._Nit._Nit._1 and _2) and competitive exclusion (e.g., between Thaum._Nit._Nit._Nit._4 and _5), supporting the finding of non-randomly, vertically structured prokaryotic communities. Overall, differences between Archaea and Bacteria and among closely related ASVs were revealed in their preferential relationship with compositional changes in the DOM pool and environmental forcing. Our results provide new insights on the ecological processes shaping prokaryotic assembly and biogeography.

Deep convolutional self-paced clustering

Deep convolutional self-paced clustering

Deep attributed graph clustering with self-separation regularization and parameter-free cluster estimation

Detecting clusters over attributed graphs is a fundamental task in the graph analysis field. The goal is to partition nodes into dense clusters based on both their attributes and structures. Modern graph neural networks provide facilitation to jointly capture the above information in attributed graphs with a feature aggregation manner, and have achieved great success in attributed graph clustering. However, existing methods mainly focus on capturing the proximity information in graphs and often fail to learn cluster-friendly features during the training of models. Besides, similar to many deep clustering frameworks, current methods based on graph neural networks require a preassigned cluster number before estimating the clusters. To address these limitations, we propose in this paper a deep attributed clustering method based on self-separated graph neural networks and parameter-free cluster estimation. First, to learn cluster-friendly features, we jointly optimize a jumping graph convolutional auto-encoder with a self-separation regularizer, which learns clusters with changing sizes while keeping dense intra-cluster structures and sparse inter structures. Second, an additional softmax auto-encoder is trained to determine the natural cluster number from the data. The hidden units capture cluster structures and can be used to estimate the number of clusters. Extensive experiments show the effectiveness of the proposed model.

Mixture-of-Experts Variational Autoencoder for clustering and generating from similarity-based representations on single cell data.

Clustering high-dimensional data, such as images or biological measurements, is a long-standing problem and has been studied extensively. Recently, Deep Clustering has gained popularity due to its flexibility in fitting the specific peculiarities of complex data. Here we introduce the Mixture-of-Experts Similarity Variational Autoencoder (MoE-Sim-VAE), a novel generative clustering model. The model can learn multi-modal distributions of high-dimensional data and use these to generate realistic data with high efficacy and efficiency. MoE-Sim-VAE is based on a Variational Autoencoder (VAE), where the decoder consists of a Mixture-of-Experts (MoE) architecture. This specific architecture allows for various modes of the data to be automatically learned by means of the experts. Additionally, we encourage the lower dimensional latent representation of our model to follow a Gaussian mixture distribution and to accurately represent the similarities between the data points. We assess the performance of our model on the MNIST benchmark data set and challenging real-world tasks of clustering mouse organs from single-cell RNA-sequencing measurements and defining cell subpopulations from mass cytometry (CyTOF) measurements on hundreds of different datasets. MoE-Sim-VAE exhibits superior clustering performance on all these tasks in comparison to the baselines as well as competitor methods.