Deep Clustering Method Research Articles

SGAE: Stacked Graph Autoencoder for Deep Clustering

Unsupervised clustering is a crucial issue in data mining and pattern recognition. Based on deep learning paradigms, deep clustering algorithms have been studied extensively and obtained superior performance in various applications. However, most of previous methods did not use helpful information from neighborhood relations to form group-separated space, and the feature embedding is usually distorted during the training process. To tackle the former limitation, we develop a graph convolution based unsupervised learning algorithm named Stacked Graph Autoencoder (SGAE). Specifically, SGAE utilizes the message passing mechanism to aggregate information from neighbors and obtain a meaningful and separated latent representation. Since the adjacency matrix is unavailable in clustering tasks, a graph construction approach with two pruning strategies is introduced to generate a transition matrix. To reduce the distortion caused by the multi-layered network training process, we further propose a topological structure preservation mechanism. It uses the constructed adjacency graph as supervised information, to maintain the relationship between nodes in the original space. Experiments on several popular benchmark datasets show that SGAE achieves significant improvements compared to unsupervised and semi-supervised deep clustering methods.

Self-supervised clustering with assistance from off-the-shelf classifier

Deep clustering outperforms conventional clustering by mutually promoting representation learning and cluster assignment. However, most existing deep clustering methods suffer from two major drawbacks. Firstly, most cluster assignment methods are highly dependent on the intermediate target distribution generated by a handcrafted nonlinear mapping function. Secondly, the clustering results can be easily guided towards wrong direction by the misassigned samples in each cluster. The existing deep clustering methods are incapable of discriminating such samples. These facts largely limit the possible performance that deep clustering methods can reach. To address these issues, a novel Self-Supervised Clustering (SSC) framework is constructed, which boosts the clustering performance by classification in an unsupervised manner. Fuzzy theory is used to score the membership of each sample to the clusters in terms of probability in each training epoch, which evaluates the intermediate clustering result certainty of each sample. The most reliable samples can be selected with the help of a sample selection method according to the membership and enhanced by data augmentation method. These augmented data are employed to fine-tune an off-the-shelf deep network classifier with the labels provided by the clustering in a self-supervised way. The classification results of the original dataset are used as the target distribution to guide the training process of the deep clustering model. The proposed framework can efficiently discriminate sample outliers and generate better target distribution with the assistance of the powerful classifier. Extensive experiments indicate that the proposed framework remarkably outperforms state-of-the-art deep clustering methods on four benchmark datasets.

AugDMC: Data Augmentation Guided Deep Multiple Clustering

Clustering aims to group similar objects together while separating dissimilar ones apart. Thereafter, structures hidden in data can be identified to help understand data in an unsupervised manner. Traditional clustering methods such as k-means provide only a single clustering for one data set. Deep clustering methods such as auto-encoder based clustering methods have shown a better performance, but still provide a single clustering. However, a given dataset might have multiple clustering structures and each represents a unique perspective of the data. Therefore, some multiple clustering methods have been developed to discover multiple independent structures hidden in data. Although deep multiple clustering methods provide better performance, how to efficiently capture the alternative perspectives in data is still a problem. In this paper, we propose AugDMC, a novel data Augmentation guided Deep Multiple Clustering method, to tackle the challenge. Specifically, AugDMC leverages data augmentations to automatically extract features related to a certain aspect of the data using a self-supervised prototype-based representation learning, where different aspects of the data can be preserved under different data augmentations. Moreover, a stable optimization strategy is proposed to alleviate the unstable problem from different augmentations. Thereafter, multiple clusterings based on different aspects of the data can be obtained. Experimental results on three real-world datasets compared with state-of-the-art methods validate the effectiveness of the proposed method.

Deep Reinforcement Clustering

Deep clustering has attracted plentiful attention in various domains owning to the superior performance. However, the previous deep clustering methods are guided by pre-specified clustering strategies that lack sustained explorations of data structures, degrading recognition of intrinsic patterns hidden in data. To address this challenge, deep reinforcement clustering (DRC) is proposed to learn an adaptive partition policy for pattern mining, which can fully explore structure knowledge of data in an adaptive manner. DRC is defined as a Markov decision process of data partitions, which chooses the optimal cluster prototype for data via maximizing the cumulative reward in state transition of environment. To implement the definition, a Bernoulli action prototype is devised to capture decision distributions in the transition of states, where the heavy-tailed Cauchy distribution precisely measures the structure divergences of data. Furthermore, a reward maximizing policy is designed to guide sustained explorations of data structures, which ensures intra-cluster compactness and inter-cluster separation of data partitions. Finally, extensive experiments are conducted on eight benchmark datasets, and the results demonstrate that DRC outperforms the state-of-the-art baseline methods.

Latent Features Embedded Dynamic Graph Evolution Deep Clustering Network

As a typical unsupervised machine learning task, clustering is always a hot research topic. Motivated by deep learning approaches, deep clustering has become prevalent in recent years, and achieves appealing performance. Most of current deep clustering methods focus on learning a new discriminative representation to enhance separability of original data, i.e., autoencoder, multilayer perceptrons and deep belief networks. However, the structure information which is very important in unsupervised learning, attracts little attention in previous deep feature representation learning clustering works. In this paper, a dynamic graph evolution based graph convolutional network (DGE-GCN) is introduced for clustering task, in which the data structural information and learned latent features are integrated into a unified network for deep clustering. Instead of using a fixed graph during the graph convolution process, we design a dynamic graph evolution strategy to refine the initial graph which could be not accurate. In addition, the latent representations learned by an autoencoder are embedded for refining the graph in a layer-wise manner. In such a manner, the latent features can help improve the graph structure, while the refined graph can in turn constrain the autoencoder to learn more discriminative features. In order to unify the graph convolutional network branch and autoencoder branch, a dual self-supervised mechanism is designed to guide the parameter learning of the whole network architecture. Comprehensive experiments demonstrate that the proposed network consistently performs better than several state-of-the-art methods on various benchmark datasets.

DSCPL: A Deep Cloud Manufacturing Service Clustering Method Using Pseudo-Labels

Research on service clustering for manufacturing network by deep learning is an effective method for service discovery and service management in manufacturing industries. Manufacturing service clustering can be used to solve the problem of service selection and composition in cloud manufacturing. After clustering characteristics of graph data, this paper analyzes and studies the manufacturing network structure, and finds the network characteristics formed by clustering. A deep manufacturing cloud service clustering model using pseudo-labels (DSCPL) is proposed, which combines graph topology and node features to cluster nodes with similar attributes. The specific contributions of the paper are as follows: 1) Defining a parametric nonlinear map embedded the graph space containing graph structure (adjacency information) and node information (features) into a low-dimensional feature space. 2)Building an auxiliary target distribution to realize the self-learning mechanism in order to adapt to the clustering task, and 3) Realizing the unsupervised clustering method. Comprehensive experiments show that the clustering effect of this method is better than the current advanced deep clustering methods on public datasets and simulated datasets.

Financial Market Correlation Analysis and Stock Selection Application Based on TCN-Deep Clustering

It is meaningful to analyze the market correlations for stock selection in the field of financial investment. Since it is difficult for existing deep clustering methods to mine the complex and nonlinear features contained in financial time series, in order to deeply mine the features of financial time series and achieve clustering, a new end-to-end deep clustering method for financial time series is proposed. It contains two modules: an autoencoder feature extraction network based on TCN (temporal convolutional neural) networks and a temporal clustering optimization algorithm with a KL (Kullback–Leibler) divergence. The features of financial time series are represented by the causal convolution and the dilated convolution of TCN networks. Then, the pre-training results based on the KL divergence are fine-tuned to make the clustering results discriminative. The experimental results show that the proposed method outperforms existing deep clustering and general clustering algorithms in the CSI 300 and S&P 500 index markets. In addition, the clustering results combined with an inference strategy can be used to select stocks that perform well or poorly, thus guiding actual stock market trades.

RffAe-S: Autoencoder Based on Random Fourier Feature With Separable Loss for Unsupervised Signal Modulation Clustering

Unsupervised signal modulation clustering is becoming increasingly important due to its application in the dynamic spectrum access process of 5G wireless communication and threat detection at the physical layer of Internet of Things. The need for better clustering results makes it a challenge to avoid feature drift and improve feature separability. This article proposes a novel separable loss function to address the issue. Besides, the high-level semantic properties of modulation types make it difficult for networks to extract their features. An autoencoder structure based on the random Fourier feature (RffAe) is proposed to simulate the demodulation process of unknown signals. Combined with the separable loss of RffAe (RffAe-S), it has excellent feature extraction ability. Great experiments were carried out on RADIOML 2016.10 A and RADIOML 2016.10 B. Experimental evaluations on these datasets show that our approach RffAe-S achieves state-of-the-art results compared to classical and the most relevant deep clustering methods.

Graph Convolutional Network Combined with Semantic Feature Guidance for Deep Clustering

The performances of semisupervised clustering for unlabeled data are often superior to those of unsupervised learning, which indicates that semantic information attached to clusters can significantly improve feature representation capability. In a graph convolutional network (GCN), each node contains information about itself and its neighbors that is beneficial to common and unique features among samples. Combining these findings, we propose a deep clustering method based on GCN and semantic feature guidance (GFDC) in which a deep convolutional network is used as a feature generator, and a GCN with a softmax layer performs clustering assignment. First, the diversity and amount of input information are enhanced to generate highly useful representations for downstream tasks. Subsequently, the topological graph is constructed to express the spatial relationship of features. For a pair of datasets, feature correspondence constraints are used to regularize clustering loss, and clustering outputs are iteratively optimized. Three external evaluation indicators, i.e., clustering accuracy, normalized mutual information, and the adjusted Rand index, and an internal indicator, i.e., the Davidson-Bouldin index (DBI), are employed to evaluate clustering performances. Experimental results on eight public datasets show that the GFDC algorithm is significantly better than the majority of competitive clustering methods, i.e., its clustering accuracy is 20% higher than the best clustering method on the United States Postal Service dataset. The GFDC algorithm also has the highest accuracy on the smaller Amazon and Caltech datasets. Moreover, DBI indicates the dispersion of cluster distribution and compactness within the cluster.

Exploring Human Mobility for Multi-Pattern Passenger Prediction: A Graph Learning Framework

Traffic flow prediction is an integral part of an intelligent transportation system and thus fundamental for various traffic-related applications. Buses are an indispensable way of moving for urban residents with fixed routes and schedules, which leads to latent travel regularity. However, human mobility patterns, specifically the complex relationships between bus passengers, are deeply hidden in this fixed mobility mode. Although many models exist to predict traffic flow, human mobility patterns have not been well explored in this regard. To reduce this research gap and learn human mobility knowledge from this fixed travel behaviors, we propose a multi-pattern passenger flow prediction framework, MPGCN, based on Graph Convolutional Network (GCN). Firstly, we construct a novel sharing-stop network to model relationships between passengers based on bus record data. Then, we employ GCN to extract features from the graph by learning useful topology information and introduce a deep clustering method to recognize mobility patterns hidden in bus passengers. Furthermore, to fully utilize Spatio-temporal information, we propose GCN2Flow to predict passenger flow based on various mobility patterns. To the best of our knowledge, this paper is the first work to adopt a multipattern approach to predict the bus passenger flow from graph learning. We design a case study for optimizing routes. Extensive experiments upon a real-world bus dataset demonstrate that MPGCN has potential efficacy in passenger flow prediction and route optimization.

Deviation Point Curriculum Learning for Trajectory Outlier Detection in Cooperative Intelligent Transport Systems

Cooperative Intelligent Transport Systems (C-ITS) are emerging in the field of transportation systems, which can be used to provide safety, sustainability, efficiency, communication and cooperation between vehicles, roadside units, and traffic command centres. With improved network structure and traffic mobility, a large amount of trajectory-based data is generated. Trajectory-based knowledge graphs help to give semantic and interconnection capabilities for intelligent transport systems. Prior works consider trajectory as the single point of deviation for the individual outliers. However, in real-world transportation systems, trajectory outliers can be seen in the groups, e.g., a group of vehicles that deviates from a single point based on the maintenance of streets in the vicinity of the intelligent transportation system. In this paper, we propose a trajectory deviation point embedding and deep clustering method for outlier detection. We first initiate network structure and nodes’ neighbours to construct a structural embedding by preserving nodes relationships. We then implement a method to learn the latent representation of deviation points in road network structures. A hierarchy multilayer graph is designed with a biased random walk to generate a set of sequences. This sequence is implemented to tune the node embeddings. After that, embedding values of the node were averaged to get the trip embedding. Finally, LSTM-based pairwise classification method is initiated to cluster the embedding with similarity-based measures. The results obtained from the experiments indicate that the proposed learning trajectory embedding captured structural identity and increased <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">F-measure</i> by 5.06% and 2.4% while compared with generic <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">Node2Vec</i> and <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">Struct2Vec</i> methods.

Learning from biased crowdsourced labeling with deep clustering

With the rapid development of crowdsourcing learning, amount of labels can be obtained from crowd workers fast and cheaply. However, crowdsourcing learning also faces challenges due to the varied qualities of amateurish crowd workers. To improve the quality of crowd labels, many researchers focus on inferring the ground truth from noisy labels, and take different factors, e.g. the reliability of workers and the difficulty of instances, into consideration to infer the aggregated labels. Nevertheless, to the best of our knowledge, label aggregation for biased crowdsourced labeling scenarios has not been sufficiently studied. Actually, the phenomenon of biased labeling exists in many crowdsourcing annotation tasks and affects the performance of label aggregation. To this end, this paper proposes a novel framework termed Biased Crowdsourcing Learning with Deep Clustering (BCLDC), which involves label aggregation and prediction using deep clustering to improve the quality of aggregated labels and learned models in biased labeling scenarios. BCLDC utilizes a deep clustering method to detect the labeling bias and then eliminates the bias by adjusting the number of labels belonging to the minority class which has fewer labels. Finally, a classifier is trained simultaneously with the aggregated labels inferred by an EM algorithm. Experimental results on six real-world datasets and five synthetic datasets consistently show that the proposed BCLDC outperforms other state-of-the-art algorithms in terms of ground truth inference and prediction.

Learning the Precise Feature for Cluster Assignment.

Clustering is one of the fundamental tasks in computer vision and pattern recognition. Recently, deep clustering methods (algorithms based on deep learning) have attracted wide attention with their impressive performance. Most of these algorithms combine deep unsupervised representation learning and standard clustering together. However, the separation of representation learning and clustering will lead to suboptimal solutions because the two-stage strategy prevents representation learning from adapting to subsequent tasks (e.g., clustering according to specific cues). To overcome this issue, efforts have been made in the dynamic adaption of representation and cluster assignment, whereas current state-of-the-art methods suffer from heuristically constructed objectives with the representation and cluster assignment alternatively optimized. To further standardize the clustering problem, we audaciously formulate the objective of clustering as finding a precise feature as the cue for cluster assignment. Based on this, we propose a general-purpose deep clustering framework, which radically integrates representation learning and clustering into a single pipeline for the first time. The proposed framework exploits the powerful ability of recently developed generative models for learning intrinsic features, and imposes an entropy minimization on the distribution of the cluster assignment by a dedicated variational algorithm. The experimental results show that the performance of the proposed method is superior, or at least comparable to, the state-of-the-art methods on the handwritten digit recognition, fashion recognition, face recognition, and object recognition benchmark datasets.

Manifold spatial clustering via asymmetric convolutional denoising autoencoder

Deep unsupervised learning extracts meaningful features from unlabeled images and simultaneously serves downstream tasks in computer vision. The basic process of deep clustering methods can include features learning and clustering assignment. To enhance the discriminative ability of the features and further improve the clustering performances, a new deep clustering method namely ACMEC (asymmetric convolutional denoising autoencoder with manifold spatial embedding clustering) is proposed. In this method, an asymmetric convolution denoising autoencoder is employed to extract visual features from images, and a manifold learning algorithm is used to obtain more distinctive features, followed by a Gaussian Mixture Model (GMM) is for clustering learning. The stability of feature space is guaranteed using separately training mechanism. In addition, reconstruction from noisy images enhances the robustness of feature networks. Experimental results on nine benchmark datasets demonstrate that the proposed ACMEC method can provide the better performances such as 0.979 clustering accuracy on the MNIST dataset and 0.668 on the fashion-MNIST dataset. ACMEC is a comparable competitor to the N2D (not too deep clustering) algorithm that is with 0.979 and 0.672 clustering accuracies respectively. Moreover, it is 16.1% higher than DEC algorithm on the fashion-MNIST dataset.

Neural mixture models with expectation-maximization for end-to-end deep clustering

Any clustering algorithm must synchronously learn to model the clusters and allocate data to those clusters in the absence of labels. Mixture model-based methods model clusters with pre-defined statistical distributions and allocate data to those clusters based on the cluster likelihoods. They iteratively refine those distribution parameters and member assignments following the Expectation-Maximization (EM) algorithm. However, the cluster representability of such hand-designed distributions that employ a limited amount of parameters is not adequate for most real-world clustering tasks. In this paper, we realize mixture model-based clustering with a neural network where the final layer neurons, with the aid of an additional transformation, approximate cluster distribution outputs. The network parameters pose as the parameters of those distributions. The result is an elegant, much-generalized representation of clusters than a restricted mixture of hand-designed distributions. We train the network end-to-end via batch-wise EM iterations where the forward pass acts as the E-step and the backward pass acts as the M-step. In image clustering, the mixture-based EM objective can be used as the clustering objective along with existing representation learning methods. In particular, we show that when mixture-EM optimization is fused with consistency optimization, it improves the sole consistency optimization performance in clustering. Our trained networks outperform single-stage deep clustering methods that still depend on k-means, with unsupervised classification accuracy of 63.8% in STL10, 58% in CIFAR10, 25.9% in CIFAR100, and 98.9% in MNIST.

Active deep image clustering

Deep clustering has attracted increasingly more attention in recent years. However, due to the absence of labels, deep clustering sometimes still provides unreliable clustering results. Although semi-supervised deep clustering can alleviate this problem to some extent by involving few human annotations, we observe that the performance of semi-supervised clustering highly depends on the selection of data for human labeling, but unfortunately, the supervised information selection is still a tough problem as traditional semi-supervised methods pay no attention to it. To tackle this problem, in this paper, we propose a novel deep active clustering method, which can actively select the key data for human labeling and apply the human annotations to improve the deep clustering. Different from conventional semi-supervised deep clustering methods which use fixed pre-given supervised information, we design a simple yet effective strategy to select the informative and uncertain data for querying annotation, which is beneficial to the clustering task. Furthermore, we integrate deep representation learning, clustering, and data selection into a unified framework, so that each task can be boosted by each other. Finally, we conduct extensive experiments on benchmark data sets by comparing it with some state-of-the-art deep clustering methods and semi-supervised clustering methods. The experimental results show that our active clustering methods can outperform both the unsupervised and semi-supervised clustering methods, demonstrating the effectiveness of the proposed method. The codes of this paper are released in https://doctor-nobody.github.io/codes/ADC_codes.zip.

A Generative Clustering Ensemble Model and Its Application in IoT Data Analysis

Data analysis is the foundation of Internet of Things (IoT) based applications, and clustering is an effective technology of data analysis. Clustering ensemble integrates multiple base clustering results to obtain a consensus result and thus improves the clustering performance in stability and robustness. However, it is difficult for existing clustering ensemble algorithms to achieve a satisfying ensemble result, when the base clustering results are unreliable. Concerning this problem, we develop a new clustering ensemble model in this paper, which has several advantages compared with traditional algorithms: (i) structure information about the data is effectively extracted from the base clusterings; (ii) data characteristics and structure information are integrated in an elegant fashion, in the production of the consensus clustering result; and (iii) our model has the generative ability that makes the model achieve outstanding performance when training samples are insufficient. In our model, the structural information is extracted by explicating the coupling relationships between base clusterings and between samples in clustering members. Then, data characteristics and structure information are combined in a generative graph representation learning framework. And the objectives of representation learning and consensus clustering are integrated into a unified optimization model, in which the prior distribution of the data is approximated by a Gaussian mixture model (GMM). Extensive experiments are conducted on multiple IoT datasets; the results prove that our model not only performs better than the conventional clustering ensemble algorithms but also outperforms the state-of-the-art deep clustering methods.

Top-Down Deep Clustering with Multi-Generator GANs

Deep clustering (DC) leverages the representation power of deep architectures to learn embedding spaces that are optimal for cluster analysis. This approach filters out low-level information irrelevant for clustering and has proven remarkably successful for high dimensional data spaces. Some DC methods employ Generative Adversarial Networks (GANs), motivated by the powerful latent representations these models are able to learn implicitly. In this work, we propose HC-MGAN, a new technique based on GANs with multiple generators (MGANs), which have not been explored for clustering. Our method is inspired by the observation that each generator of a MGAN tends to generate data that correlates with a sub-region of the real data distribution. We use this clustered generation to train a classifier for inferring from which generator a given image came from, thus providing a semantically meaningful clustering for the real distribution. Additionally, we design our method so that it is performed in a top-down hierarchical clustering tree, thus proposing the first hierarchical DC method, to the best of our knowledge. We conduct several experiments to evaluate the proposed method against recent DC methods, obtaining competitive results. Last, we perform an exploratory analysis of the hierarchical clustering tree that highlights how accurately it organizes the data in a hierarchy of semantically coherent patterns.

Adaptive Correlation Integration for Deep Image Clustering

Deep clustering methods often estimate data correlations to guide unsupervised learning. However, since different kinds of correlations capture different data characteristics, a strong data correlation may not be appropriate for every case. In this paper, we propose a novel Teacher-Student framework to adaptively estimate correlation and interactively learn a deep clustering model for various data distributions. Specifically, a teacher module mines and integrates various kinds of correlations from different perspectives. To adapt to various cases, we propose a novel Adaptive Integration Gate (ADI-Gate) to selectively and dynamically integrate different data correlations in a teaching-feedback manner. Furthermore, a student module performs unsupervised clustering inference with the estimated correlation and provides a preference for teacher module. We also design a Pairwise-Weighted loss (PW-loss) to enhance high-confident correlation guidance of data pairs during the learning process of student module. In image clustering experiments on four public datasets, our model achieves consistent improvements over state-of-the-art models.

Unsupervised discriminative feature learning via finding a clustering-friendly embedding space

In this paper, we propose an enhanced deep clustering network (EDCN), which is composed of a Feature Extractor, a Conditional Generator, a Discriminator and a Siamese Network. Specifically, we will utilize two kinds of generated data based on adversarial training, as well as the original data, to train the Feature Extractor for learning effective latent representations. In addition, we adopt the Siamese network to find an embedding space, where a better affinity similarity matrix is obtained as the key to success of spectral clustering in providing reliable pseudo-labels. Particularly, the obtained pseudo-labels will be used to generate realistic data by the Generator. Finally, the discriminator is used to model the real joint distribution of data and corresponding latent representations for Feature Extractor enhancement. To evaluate our proposed EDCN, we conduct extensive experiments on multiple data sets including MNIST, USPS, FRGC, CIFAR-10, STL-10, and Fashion-MNIST by comparing our method with a number of state-of-the-art deep clustering methods, and experimental results demonstrate its effectiveness and superiority.