Self-supervised Clustering Research Articles

Progressive Self-Supervised Clustering With Novel Category Discovery

These days, clustering is one of the most classical themes to analyze data structures in machine learning and pattern recognition. Recently, the anchor-based graph has been widely adopted to promote the clustering accuracy of plentiful graph-based clustering techniques. In order to achieve more satisfying clustering performance, we propose a novel clustering approach referred to as the progressive self-supervised clustering method with novel category discovery (PSSCNCD), which consists of three separate procedures specifically. First, we propose a new semisupervised framework with novel category discovery to guide label propagation processing, which is reinforced by the parameter-insensitive anchor-based graph obtained from balanced K -means and hierarchical K -means (BKHK). Second, we design a novel representative point selected strategy based on our semisupervised framework to discover each representative point and endow pseudolabel progressively, where every pseudolabel hypothetically corresponds to a real category in each self-supervised label propagation. Third, when sufficient representative points have been found, the labels of all samples will be finally predicted to obtain terminal clustering results. In addition, the experimental results on several toy examples and benchmark data sets comprehensively demonstrate that our method outperforms other clustering approaches.

Adaptive graph convolutional clustering network with optimal probabilistic graph

The graph convolutional network (GCN)-based clustering approaches have achieved the impressive performance due to strong ability of exploiting the topological structure. The adjacency graph seriously affects the clustering performance, especially for non-graph data. Existing approaches usually conduct two independent steps, i.e., constructing a fixed graph structure and then graph embedding representation learning by GCN. However, the constructed graph structure may be unreliable one due to noisy data, resulting in sub-optimal graph embedding representation. In this paper, we propose an adaptive graph convolutional clustering network (AGCCN) to alternatively learn the similarity graph structure and node embedding representation in a unified framework. Our AGCCN learns the weighted adjacency graph adaptively from the node representations by solving the optimization problem of graph learning, in which adaptive and optimal neighbors for each sample are assigned with probabilistic way according to local connectivity. Then, the attribute feature extracted by parallel Auto-Encoder (AE) module is fused into the input of adaptive graph convolution module layer-by-layer to learn the comprehensive node embedding representation and strengthen its representation ability. This also skillfully alleviates the over-smoothing problem of GCN. To further improve the discriminant ability of node representation, a dual self-supervised clustering mechanism is designed to guide model optimization with pseudo-labels information. Extensive experimental results on various real-world datasets consistently show the superiority and effectiveness of the proposed deep graph clustering method.

Fast Self-Supervised Clustering With Anchor Graph.

Benefit from avoiding the utilization of labeled samples, which are usually insufficient in the real world, unsupervised learning has been regarded as a speedy and powerful strategy on clustering tasks. However, clustering directly from primal data sets leads to high computational cost, which limits its application on large-scale and high-dimensional problems. Recently, anchor-based theories are proposed to partly mitigate this problem and field naturally sparse affinity matrix, while it is still a challenge to get excellent performance along with high efficiency. To dispose of this issue, we first presented a fast semisupervised framework (FSSF) combined with a balanced K -means-based hierarchical K -means (BKHK) method and the bipartite graph theory. Thereafter, we proposed a fast self-supervised clustering method involved in this crucial semisupervised framework, in which all labels are inferred from a constructed bipartite graph with exactly k connected components. The proposed method remarkably accelerates the general semisupervised learning through the anchor and consists of four significant parts: 1) obtaining the anchor set as interim through BKHK algorithm; 2) constructing the bipartite graph; 3) solving the self-supervised problem to construct a typical probability model with FSSF; and 4) selecting the most representative points regarding anchors from BKHK as an interim and conducting label propagation. The experimental results on toy examples and benchmark data sets have demonstrated that the proposed method outperforms other approaches.

Self-Supervised Clustering for Leaf Disease Identification

Plant diseases have been one of the most threatening scenarios to farmers. Although most plant diseases can be identified by observing leaves, it often requires human expertise. The recent improvements in computer vision have led to introduce disease classification systems through observing leaf images. Nevertheless, most disease classification systems are specific to diseases and plants, limiting method’s usability. The methods are also costly as they require vast labeled data, which can only be done by experts. This paper introduces a self-supervised leaf disease clustering system that can be used for classifying plant diseases. As self-supervision does not require labeled data, the proposed method can be inexpensive and can be implemented for most types of plants. The method implements a siamese deep convolutional neural network (DCNN) for generating clusterable embeddings from leaf images. The training strategy of the embedding network is conducted using AutoEmbedder approach with randomly augmented image pairs. The self-supervised embedding model training involves three different data pair linkage scenarios: can-link, cannot-link, and may-link pairs. The embeddings are further clustered using k-means algorithm in the final classification stage. The experiment is conducted to individually classify diseases of eight different fruit leaves. The results indicate that the proposed leaf disease identification method performs better than the existing self-supervised clustering systems. The paper indicates that end-to-end siamese networks can outperform well-designed sequentially trained self-supervised methods.

Deep self-supervised clustering with embedding adjacent graph features

Deep clustering uses neural networks to learn the low-dimensional feature representations suitable for clustering tasks. Numerous studies have shown that learning embedded features and defining the clustering loss properly contribute to better performance. However, most of the existing studies focus on the deep local features and ignore the global spatial characteristics of the original data space. To address this issue, this paper proposes deep self-supervised clustering with embedding adjacent graph features (DSSC-EAGF). The significance of our efforts is three-fold: 1) To obtain the deep representation of the potential global spatial structure, a dedicated adjacent graph matrix is designed and used to train the autoencoder in the original data space; 2) In the deep encoding feature space, the KNN algorithm is used to obtain the virtual clusters for devising a self-supervised learning loss. Then, the reconstruction loss, clustering loss, and self-supervised loss are integrated, and a novel overall loss measurement is proposed for DSSC-EAGF. 3) An inverse-Y-shaped network model is designed to well learn the features of both the local and the global structures of the original data, which greatly improves the clustering performance. The experimental studies prove the superiority of the proposed DSSC-EAGF against a few state-of-the-art deep clustering methods.

Self-supervised clustering of mass spectrometry imaging data using contrastive learning.

Mass spectrometry imaging (MSI) is widely used for the label-free molecular mapping of biological samples. The identification of co-localized molecules in MSI data is crucial to the understanding of biochemical pathways. One of key challenges in molecular colocalization is that complex MSI data are too large for manual annotation but too small for training deep neural networks. Herein, we introduce a self-supervised clustering approach based on contrastive learning, which shows an excellent performance in clustering of MSI data. We train a deep convolutional neural network (CNN) using MSI data from a single experiment without manual annotations to effectively learn high-level spatial features from ion images and classify them based on molecular colocalizations. We demonstrate that contrastive learning generates ion image representations that form well-resolved clusters. Subsequent self-labeling is used to fine-tune both the CNN encoder and linear classifier based on confidently classified ion images. This new approach enables autonomous and high-throughput identification of co-localized species in MSI data, which will dramatically expand the application of spatial lipidomics, metabolomics, and proteomics in biological research.

A three-stage learning approach to cross-domain person re-identification

Cross-domain person re-identification (Re-id) is far less discussed than supervised Re-id, since there are not labeled data that can be used to guide the person Re-id. Even though some works have been done on unsupervised Re-id, cross-domain Re-id is much challenging, and needs further research works. In this paper, without background transfer and other data enhancement of source domain training set, we propose a three-stage cross-domain Re-id model which considers the domain adaptation, self-supervised clustering re-training and joint loss training in one schema. In order to improve the adaptability of the model trained on the source data, cross-domain and cross-camera losses are considered. Then we do the self-supervised learning of clustering re-training in target dataset to learn the discriminative features in the target domain on the basis of the pre-trained model. Finally, we train the target data using the pseudo labels obtained by clustering with joint metric learning and representation learning. In particular, we replace the classic cross-entropy loss with label smoothing regularization loss so as to reduce fitting of false pseudo labels. Performance evaluation on benchmark datasets demonstrates the effectiveness of the proposed approach.

Contrastive self-supervised clustering of scRNA-seq data

BackgroundSingle-cell RNA sequencing (scRNA-seq) has emerged has a main strategy to study transcriptional activity at the cellular level. Clustering analysis is routinely performed on scRNA-seq data to explore, recognize or discover underlying cell identities. The high dimensionality of scRNA-seq data and its significant sparsity accentuated by frequent dropout events, introducing false zero count observations, make the clustering analysis computationally challenging. Even though multiple scRNA-seq clustering techniques have been proposed, there is no consensus on the best performing approach. On a parallel research track, self-supervised contrastive learning recently achieved state-of-the-art results on images clustering and, subsequently, image classification.ResultsWe propose contrastive-sc, a new unsupervised learning method for scRNA-seq data that perform cell clustering. The method consists of two consecutive phases: first, an artificial neural network learns an embedding for each cell through a representation training phase. The embedding is then clustered in the second phase with a general clustering algorithm (i.e. KMeans or Leiden community detection). The proposed representation training phase is a new adaptation of the self-supervised contrastive learning framework, initially proposed for image processing, to scRNA-seq data. contrastive-sc has been compared with ten state-of-the-art techniques. A broad experimental study has been conducted on both simulated and real-world datasets, assessing multiple external and internal clustering performance metrics (i.e. ARI, NMI, Silhouette, Calinski scores). Our experimental analysis shows that constastive-sc compares favorably with state-of-the-art methods on both simulated and real-world datasets.ConclusionOn average, our method identifies well-defined clusters in close agreement with ground truth annotations. Our method is computationally efficient, being fast to train and having a limited memory footprint. contrastive-sc maintains good performance when only a fraction of input cells is provided and is robust to changes in hyperparameters or network architecture. The decoupling between the creation of the embedding and the clustering phase allows the flexibility to choose a suitable clustering algorithm (i.e. KMeans when the number of expected clusters is known, Leiden otherwise) or to integrate the embedding with other existing techniques.

Self-supervised cross-iterative clustering for unlabeled plant disease images

Current annotation for plant disease images depends on manual sorting and handcrafted features by agricultural experts, which is time-consuming and labour-intensive. In this paper, we propose a self-supervised clustering framework for grouping plant disease images based on the vulnerability of Kernel K-means. The main idea is to establish a cross iterative under-clustering algorithm based on Kernel K-means to produce the pseudo-labeled training set and a chaotic cluster to be further classified by a deep learning module. In order to verify the effectiveness of our proposed framework, we conduct extensive experiments on three different plant disease datatsets with five plants and 17 plant diseases. The experimental results show the high superiority of our method to do image-based plant disease classification over balanced and unbalanced datasets by comparing with five state-of-the-art existing works in terms of different metrics.

Adversarial self-supervised clustering with cluster-specificity distribution

As we embark on the big data era, numerous unlabeled data has generated increasingly, and thus deep clustering analysis has become prevalent in artificial intelligence. Yet, most existing deep clustering methods still have the following demerits: 1) They fail to take cluster-specificity distribution into account, resulting in suboptimal latent representation. 2) They suffer from the scale issue of the distributions between the given sample and cluster centers, resulting in unstable clustering performance. 3) They fail to utilize the obtained clustering labels, resulting in suboptimal clustering performances. To fill these gaps, we propose a new deep clustering solution, namely Adversarial Self-supervised Clustering With Cluster-specificity Distribution (ASC2D). Specifically, by imposing the cluster-specificity constraint, which is measured by the ℓ1,2-norm, the learned latent representation well encodes the cluster structure. Meanwhile, by introducing the thought of adversarial learning, ASC2D well eliminates the gaps between distributions. Moreover, ASC2D utilize the clustering label to supervise the learning of representation, where the latter is used in turn to conduct the subsequent clustering. By this way, clustering and representation learning are seamlessly connected, with the aim to achieve better clustering performance. Extensive experimental results show that ASC2D is superior to 14 state-of-the-art baselines on six image datasets in terms of three evaluation metrics, especially on Fashion-MNIST datasets, ASC2D brings about 4.1% and 7.1% improvement over the best baseline in terms of ACC and NMI metrics.

Self-Supervised Representation Learning With Path Integral Clustering for Speaker Diarization

Automatic speaker diarization techniques typically involve a two-stage processing approach where audio segments of fixed duration are converted to vector representations in the first stage. This is followed by an unsupervised clustering of the representations in the second stage. In most of the prior approaches, these two stages are performed in an isolated manner with independent optimization steps. In this paper, we propose a representation learning and clustering algorithm that can be iteratively performed for improved speaker diarization. The representation learning is based on principles of self-supervised learning while the clustering algorithm is a graph structural method based on path integral clustering (PIC). The representation learning step uses the cluster targets from PIC and the clustering step is performed on embeddings learned from the self-supervised deep model. This iterative approach is referred to as self-supervised clustering (SSC). The diarization experiments are performed on CALLHOME and AMI meeting datasets. In these experiments, we show that the SSC algorithm improves significantly over the baseline system (relative improvements of 13% and 59% on CALLHOME and AMI datasets respectively in terms of diarization error rate (DER)). In addition, the DER results reported in this work improve over several other recent approaches for speaker diarization.

Deep Clustering to Identify Sources of Urban Seismic Noise in Long Beach, California

Abstract Ambient seismic noise consists of emergent and impulsive signals generated by natural and anthropogenic sources. Developing techniques to identify specific cultural noise signals will benefit studies performing seismic imaging from continuous records. We examine spectrograms of urban cultural noise from a spatially dense seismic array located in Long Beach, California. The spectral features of the waveforms are used to develop a self-supervised clustering model for differentiating cultural noise into separable types of signals. We use 161 hr of seismic data from 5200 geophones that contain impulsive signals originating from human activity. The model uses convolutional autoencoders, a self-supervised machine-learning technique, to learn latent features from spectrograms produced from the data. The latent features are evaluated using a deep clustering algorithm to separate the noise signals into different classes. We evaluate the separation of data and analyze the classes to identify the likely sources of the signals present in the data. To interpret the model performance, we examine the time–frequency domain features of the signals and the spatiotemporal evolution observed for each class. We demonstrate that clustering using deep autoencoders is a useful approach to characterizing seismic noise and identifying novel signals in the data.

Self-Supervised Time Series Clustering With Model-Based Dynamics.

Time series clustering is usually an essential unsupervised task in cases when category information is not available and has a wide range of applications. However, existing time series clustering methods usually either ignore temporal dynamics of time series or isolate the feature extraction from clustering tasks without considering the interaction between them. In this article, a time series clustering framework named self-supervised time series clustering network (STCN) is proposed to optimize the feature extraction and clustering simultaneously. In the feature extraction module, a recurrent neural network (RNN) conducts a one-step time series prediction that acts as the reconstruction of the input data, capturing the temporal dynamics and maintaining the local structures of the time series. The parameters of the output layer of the RNN are regarded as model-based dynamic features and then fed into a self-supervised clustering module to obtain the predicted labels. To bridge the gap between these two modules, we employ spectral analysis to constrain the similar features to have the same pseudoclass labels and align the predicted labels with pseudolabels as well. STCN is trained by iteratively updating the model parameters and the pseudoclass labels. Experiments conducted on extensive time series data sets show that STCN has state-of-the-art performance, and the visualization analysis also demonstrates the effectiveness of the proposed model.

Constrained Self-Supervised Clustering for Discovering New Intents (Student Abstract)

Discovering new user intents is an emerging task in the dialogue system. In this paper, we propose a self-supervised clustering method that can naturally incorporate pairwise constraints as prior knowledge to guide the clustering process and does not require intensive feature engineering. Extensive experiments on three benchmark datasets show that our method can yield significant improvements over strong baselines.