Deep Clustering Research Articles

Joint optimization of an autoencoder for clustering and embedding

Deep embedded clustering has become a dominating approach to unsupervised categorization of objects with deep neural networks. The optimization of the most popular methods alternates between the training of a deep autoencoder and a k-means clustering of the autoencoder’s embedding. The diachronic setting, however, prevents the former to benefit from valuable information acquired by the latter. In this paper, we present an alternative where the autoencoder and the clustering are learned simultaneously. This is achieved by providing novel theoretical insight, where we show that the objective function of a certain class of Gaussian mixture models (GMM’s) can naturally be rephrased as the loss function of a one-hidden layer autoencoder thus inheriting the built-in clustering capabilities of the GMM. That simple neural network, referred to as the clustering module, can be integrated into a deep autoencoder resulting in a deep clustering model able to jointly learn a clustering and an embedding. Experiments confirm the equivalence between the clustering module and Gaussian mixture models. Further evaluations affirm the empirical relevance of our deep architecture as it outperforms related baselines on several data sets.

RETRACTED ARTICLE: Deep clustering based on embedded auto-encoder

RETRACTED ARTICLE: Deep clustering based on embedded auto-encoder

X-DC: Explainable Deep Clustering Based on Learnable Spectrogram Templates

Deep neural networks (DNNs) have achieved substantial predictive performance in various speech processing tasks. Particularly, it has been shown that a monaural speech separation task can be successfully solved with a DNN-based method called deep clustering (DC), which uses a DNN to describe the process of assigning a continuous vector to each time-frequency (TF) bin and measure how likely each pair of TF bins is to be dominated by the same speaker. In DC, the DNN is trained so that the embedding vectors for the TF bins dominated by the same speaker are forced to get close to each other. One concern regarding DC is that the embedding process described by a DNN has a black-box structure, which is usually very hard to interpret. The potential weakness owing to the noninterpretable black box structure is that it lacks the flexibility of addressing the mismatch between training and test conditions (caused by reverberation, for instance). To overcome this limitation, in this letter, we propose the concept of explainable deep clustering (X-DC), whose network architecture can be interpreted as a process of fitting learnable spectrogram templates to an input spectrogram followed by Wiener filtering. During training, the elements of the spectrogram templates and their activations are constrained to be nonnegative, which facilitates the sparsity of their values and thus improves interpretability. The main advantage of this framework is that it naturally allows us to incorporate a model adaptation mechanism into the network thanks to its physically interpretable structure. We experimentally show that the proposed X-DC enables us to visualize and understand the clues for the model to determine the embedding vectors while achieving speech separation performance comparable to that of the original DC models.

Embedding-based Detection and Extraction of Research Topics from Academic Documents Using Deep Clustering

Abstract Purpose Detection of research fields or topics and understanding the dynamics help the scientific community in their decisions regarding the establishment of scientific fields. This also helps in having a better collaboration with governments and businesses. This study aims to investigate the development of research fields over time, translating it into a topic detection problem. Design/methodology/approach To achieve the objectives, we propose a modified deep clustering method to detect research trends from the abstracts and titles of academic documents. Document embedding approaches are utilized to transform documents into vector-based representations. The proposed method is evaluated by comparing it with a combination of different embedding and clustering approaches and the classical topic modeling algorithms (i.e. LDA) against a benchmark dataset. A case study is also conducted exploring the evolution of Artificial Intelligence (AI) detecting the research topics or sub-fields in related AI publications. Findings Evaluating the performance of the proposed method using clustering performance indicators reflects that our proposed method outperforms similar approaches against the benchmark dataset. Using the proposed method, we also show how the topics have evolved in the period of the recent 30 years, taking advantage of a keyword extraction method for cluster tagging and labeling, demonstrating the context of the topics. Research limitations We noticed that it is not possible to generalize one solution for all downstream tasks. Hence, it is required to fine-tune or optimize the solutions for each task and even datasets. In addition, interpretation of cluster labels can be subjective and vary based on the readers’ opinions. It is also very difficult to evaluate the labeling techniques, rendering the explanation of the clusters further limited. Practical implications As demonstrated in the case study, we show that in a real-world example, how the proposed method would enable the researchers and reviewers of the academic research to detect, summarize, analyze, and visualize research topics from decades of academic documents. This helps the scientific community and all related organizations in fast and effective analysis of the fields, by establishing and explaining the topics. Originality/value In this study, we introduce a modified and tuned deep embedding clustering coupled with Doc2Vec representations for topic extraction. We also use a concept extraction method as a labeling approach in this study. The effectiveness of the method has been evaluated in a case study of AI publications, where we analyze the AI topics during the past three decades.

A Deep Clustering Method For Analyzing Uterine Cervix Images Across Imaging Devices.

Visual inspection of the cervix with acetic acid (VIA), though error prone, has long been used for screening women and to guide management for cervical cancer. The automated visual evaluation (AVE) technique, in which deep learning is used to predict precancer based on a digital image of the acetowhitened cervix, has demonstrated its promise as a low-cost method to improve on human performance. However, there are several challenges in moving AVE beyond proof-of-concept and deploying it as a practical adjunct tool in visual screening. One of them is making AVE robust across images captured using different devices. We propose a new deep learning based clustering approach to investigate whether the images taken by three different devices (a common smartphone, a custom smartphone-based handheld device for cervical imaging, and a clinical colposcope equipped with SLR digital camera-based imaging capability) can be well distinguished from each other with respect to the visual appearance/content within their cervix regions. We argue that disparity in visual appearance of a cervix across devices could be a significant confounding factor in training and generalizing AVE performance. Our method consists of four components: cervix region detection, feature extraction, feature encoding, and clustering. Multiple experiments are conducted to demonstrate the effectiveness of each component and compare alternative methods in each component. Our proposed method achieves high clustering accuracy (97%) and significantly outperforms several representative deep clustering methods on our dataset. The high clustering performance indicates the images taken from these three devices are different with respect to visual appearance. Our results and analysis establish a need for developing a method that minimizes such variance among the images acquired from different devices. It also recognizes the need for large number of training images from different sources for robust device-independent AVE performance worldwide.

A Self-Supervised Deep Learning Framework for Unsupervised Few-Shot Learning and Clustering

The need to learn a good representation is a core problem central to AI. We present a self-supervised representation learning framework and demonstrate its use for few-shot classification and clustering. Our framework can be interpreted as repeatedly discovering new categories from learned embeddings and training a new embedding function with self-supervised signals to differentiate the discovered categories. In our framework, we first discover categories from unlabeled data. Next we post-process the previous partition results to remove outliers and derive prototypes of each category. We then construct few-shot learning tasks with previously selected data and augmented virtual data. Lastly, we iterative train the network through previous steps to learn the final representation. Our framework can considerably outperform previous baselines in unsupervised few-shot classification tasks on miniImageNet and Omniglot data sets. We also validate our learned representation on clustering tasks and demonstrate that our framework further improves upon the recent deep clustering methods.

Deep Fusion Clustering Network

Deep clustering is a fundamental yet challenging task for data analysis. Recently we witness a strong tendency of combining autoencoder and graph neural networks to exploit structure information for clustering performance enhancement. However, we observe that existing literature 1) lacks a dynamic fusion mechanism to selectively integrate and refine the information of graph structure and node attributes for consensus representation learning; 2) fails to extract information from both sides for robust target distribution (i.e., “groundtruth” soft labels) generation. To tackle the above issues, we propose a Deep Fusion Clustering Network (DFCN). Specifically, in our network, an interdependency learning-based Structure and Attribute Information Fusion (SAIF) module is proposed to explicitly merge the representations learned by an autoencoder and a graph autoencoder for consensus representation learning. Also, a reliable target distribution generation measure and a triplet self-supervision strategy, which facilitate cross-modality information exploitation, are designed for network training. Extensive experiments on six benchmark datasets have demonstrated that the proposed DFCN consistently outperforms the state-of-the-art deep clustering methods.

Deep semi‐supervised classification based in deep clustering and cross‐entropy

Self-labeled techniques, a semi-supervised classification paradigm (SSC), are highly effective in alleviating the scarcity of labeled data used in classification tasks through an iterative process of self-training. This problem was addressed by several approaches with different assumptions about the features of the input data, examples of these approaches being self-training, co-training, STRED, among others. This paper presents a framework for data self-labeling based on deep autoencoder combined with a self-labeled technique that takes into consideration cross-entropy. The model uses the Encoder to reduce the dimensionality of the input that is submitted to a labeling layer. The weights of this layer are adjusted through the learning from a clustering performed in the Z space, which is the reduced dimensionality space. Results showed that the proposed method obtained competitive performance in relation to classic methods that are found in the literature.

A Multiscale Self-Attention Deep Clustering for Change Detection in SAR Images

Synthetic aperture radar (SAR) image change detection (CD) is an important application in the field of remote sensing. Due to the lack of labeled data especially in the pixelwise task, it is urgent to develop unsupervised techniques to effectively detect changes. In this article, we propose a novel unsupervised representation learning framework for CD in SAR images, called multiscale self-attention (SA) deep clustering based on octave convolution. The main motivation is that a convolutional neural network (CNN) has the ability to extract significant feature hidden in input images, but it relies heavily on annotated data. Clustering is typically free from supervision; however, SAR images always suffer from speckle noise, which is unfriendly for clustering. Thus, we integrate unsupervised clustering with CNN to learn clustering-friendly feature representations. In the unified framework, CNN feature learning and clustering can be optimized end-to-end without supervision. To better suppress speckle noise and boost the joint optimization for distinguishing changes and unchanges, we use the K-means++ algorithm that is robust to noise as the clustering algorithm. In the meanwhile, we introduce the octave convolution and SA mechanism into the network to fully mine important spatial structure information for enhancing noise resistance of the network. Moreover, multiscale fusion modules are proposed to fuse multiscale input into a complementary feature representation that contains more context and semantic information around each pixel so that it refines the difference feature extraction while reducing speckle noise. Experiments on challenging SAR data sets demonstrate the effectiveness and potential of the proposed model compared with the current state-of-the-art algorithms.

Deep Clustering for Improved Inter-Cluster Separability and Intra-Cluster Homogeneity with Cohesive Loss

Deep Clustering for Improved Inter-Cluster Separability and Intra-Cluster Homogeneity with Cohesive Loss

Reducing Objective Function Mismatch in Deep Clustering with the Unsupervised Companion Objective

Preservation of local similarity structure is a key challenge in deep clustering. Many recent deep clustering methods therefore use autoencoders to help guide the model's neural network towards an embedding which is more reflective of the input space geometry. However, recent work has shown that autoencoder-based deep clustering models can suffer from objective function mismatch (OFM). In order to improve the preservation of local similarity structure, while simultaneously having a low OFM, we develop a new auxiliary objective function for deep clustering. Our Unsupervised Companion Objective (UCO) encourages a consistent clustering structure at intermediate layers in the network -- helping the network learn an embedding which is more reflective of the similarity structure in the input space. Since a clustering-based auxiliary objective has the same goal as the main clustering objective, it is less prone to introduce objective function mismatch between itself and the main objective. Our experiments show that attaching the UCO to a deep clustering model improves the performance of the model, and exhibits a lower OFM, compared to an analogous autoencoder-based model.

Adversarial self-supervised clustering with cluster-specificity distribution

As we embark on the big data era, numerous unlabeled data has generated increasingly, and thus deep clustering analysis has become prevalent in artificial intelligence. Yet, most existing deep clustering methods still have the following demerits: 1) They fail to take cluster-specificity distribution into account, resulting in suboptimal latent representation. 2) They suffer from the scale issue of the distributions between the given sample and cluster centers, resulting in unstable clustering performance. 3) They fail to utilize the obtained clustering labels, resulting in suboptimal clustering performances. To fill these gaps, we propose a new deep clustering solution, namely Adversarial Self-supervised Clustering With Cluster-specificity Distribution (ASC2D). Specifically, by imposing the cluster-specificity constraint, which is measured by the ℓ1,2-norm, the learned latent representation well encodes the cluster structure. Meanwhile, by introducing the thought of adversarial learning, ASC2D well eliminates the gaps between distributions. Moreover, ASC2D utilize the clustering label to supervise the learning of representation, where the latter is used in turn to conduct the subsequent clustering. By this way, clustering and representation learning are seamlessly connected, with the aim to achieve better clustering performance. Extensive experimental results show that ASC2D is superior to 14 state-of-the-art baselines on six image datasets in terms of three evaluation metrics, especially on Fashion-MNIST datasets, ASC2D brings about 4.1% and 7.1% improvement over the best baseline in terms of ACC and NMI metrics.

Deep embedded clustering of coral reef bioacoustics.

Deep clustering was applied to unlabeled, automatically detected signals in a coral reef soundscape to distinguish fish pulse calls from segments of whale song. Deep embedded clustering (DEC) learned latent features and formed classification clusters using fixed-length power spectrograms of the signals. Handpicked spectral and temporal features were also extracted and clustered with Gaussian mixture models (GMM) and conventional clustering. DEC, GMM, and conventional clustering were tested on simulated datasets of fish pulse calls (fish) and whale song units (whale) with randomized bandwidth, duration, and SNR. Both GMM and DEC achieved high accuracy and identified clusters with fish, whale, and overlapping fish and whale signals. Conventional clustering methods had low accuracy in scenarios with unequal-sized clusters or overlapping signals. Fish and whale signals recorded near Hawaii in February-March 2020 were clustered with DEC, GMM, and conventional clustering. DEC features demonstrated the highest accuracy of 77.5% on a small, manually labeled dataset for classifying signals into fish and whale clusters.

Deep embedded clustering of coral reef bioacoustics

Deep clustering was applied to unlabeled, automatically detected signals in a coral reef soundscape to distinguish fish pulse calls from segments of whale song. Deep embedded clustering (DEC) learned latent features of the signals and formed clusters using fixed-length power spectrograms of the signals. Handpicked spectral and temporal features were also extracted and clustered with Gaussian mixture models (GMM) and conventional clustering. DEC, GMM, and conventional clustering were tested on simulated datasets of fish pulse calls and whale song units with randomized bandwidth, duration, and SNR. Both GMM and DEC achieved high accuracy and identified clusters with fish calls, whale song units, and simultaneous fish calls and whale song units. Conventional clustering methods K-means and hierarchical agglomerative clustering had low accuracy in scenarios with unequally sized clusters or multiple signals. Experimentally detected fish pulse calls and whale song units recorded near Hawaii in February–March 2020 were clustered with DEC, GMM, and conventional clustering. DEC demonstrated the highest accuracy on a small, manually labeled dataset and successfully separated clusters of fish calls and whale song units. GMM achieved high recall for detecting whale song units but accuracy. Both GMM and DEC overpredicted the number of whale song unit signals.

Unsupervised Human Activity Representation Learning with Multi-task Deep Clustering

Human activity recognition (HAR) based on sensing data from wearable and mobile devices has become an active research area in ubiquitous computing, and it envisions a wide range of application scenarios in mobile social networking, environmental context sensing, health and well-being monitoring, etc. However, activity recognition based on manually annotated sensing data is manpower-expensive, time-consuming, and privacy-sensitive, which prevents HAR systems from being really deployed in scale. In this paper, we address the problem of unsupervised human activity recognition, which infers activities from unlabeled datasets without the need of domain knowledge. We propose an end-to-end multi-task deep clustering framework to solve the problem. Taking the unlabeled multi-dimensional sensing signals as input, we firstly apply a CNN-BiLSTM autoencoder to form a compressed latent feature representation. Then we apply a K-means clustering algorithm based on the extracted features to partition the dataset into different groups, which produces pseudo labels for the instances. We further train a deep neural network (DNN) with the latent features and pseudo labels for activity recognition. The tasks of feature representation, clustering, and classification are integrated into a uniform multi-task learning framework and optimized jointly to achieve unsupervised activity classification. We conduct extensive experiments based on three public datasets. It is shown that the proposed approach outperforms shallow unsupervised learning approaches, and it performs close to the state-of-the-art supervised approaches by fine-tuning with a small number of labeled data. The proposed approach significantly reduces the cost of human-based data annotation and narrows down the gap between unsupervised and supervised human activity recognition.

Dike Inflation Process Beneath Sakurajima Volcano, Japan, During the Earthquake Swarm of August 15, 2015

Magma intrusion usually causes seismicity and deformation in the surrounding rock and often leads to eruptions. A swarm of volcano-tectonic (VT) earthquakes associated with rapid dike intrusion in hours occurred beneath Sakurajima volcano on August 15, 2015. We determined the hypocenters and focal mechanisms of the VT earthquake swarm. The distributions of pressure (P)- and tension (T)-axes of the azimuths of the mechanisms are also obtained. The results indicate spatiotemporal changes of the distributions of the hypocenters and P- and T-axes. The hypocenters are distributed at depths of 0.3–1 km and 7:00–10:30 JST, and are located at depths of 0.3–3 km and 10:30–12:00 during which the seismic activity is the largest. At 12:00–24:00, the hypocenters are distributed in shallow and deep clusters at depths of 0.2–1 km and 1.5–3.5 km, respectively. The dike induced rapid ground deformation and is located between the shallow and deep clusters. The strike and opening directions of the dike are parallel to the NE–SW and NW–SE directions, respectively, corresponding to the regional maximum and minimum compression stress. The T-axes of the shallow cluster are distributed parallel to the opening direction of the dike. The P-axes of the deep cluster exhibit a pattern corresponding to the NE–SW direction and the T-axes are distributed in the NW–SE direction. In contrast, a 90° rotated pattern of strike-slip faulting is also observed at the deep cluster at 12:00–24:00, where the P-axes are distributed in the NW–SE direction and the T-axes are distributed in the NE–SW direction. This reflects the change in the stress field due to the dike inflation during the earthquake generation, and indicates that the alteration of stress in the vicinity of the dike due to the dike inflation and VT earthquakes are induced by the differential stress exceeding the brittle fracture strength of the rock. Future seismic and deformation observations in volcanoes will verify whether the spatiotemporal changes of the hypocenters and focal mechanism shown by this study are unique features of rapid dike intrusion.

RETRACTED ARTICLE: Regularized deep clustering approach for effective categorization of maize diseases

The disease prevalence among different plant species reduces the grain production, significantly affects the growth in terms of quantity and quality. Early diagnosis of the ailment effectively minimizes the plant devastation and improves the condition. Mostly, people involved in the agriculture-related activity identifies the abnormalities in the plants. However, in some cases, the chance of manual error is high during the diagnosis of the plant status. It further leads to economic deprivation, affects the livelihood of the agriculturalists. Effective monitoring and diagnosing plant conditions can be automated with available tools and applications. Such systems serve as virtual assistance to the needy anytime, which fills the gap of the presence of domain expertise. Moreover, automated screening tools contribute to elevated agricultural activities and ensures security upon food production and availability. These applications are powered by advanced, intelligent computational models for an accurate diagnosis. Deep learning algorithms are gaining more attention in the field of real-time monitoring and assistance due to its higher precision over complex applications. However, not all the models fit for applications with different environmental conditions. In this paper, a deep cluster-based plant disease categorization system is proposed to find the discriminative patterns from the images of different categories of maize crop. The proposed regularized deep clustering (RDC) algorithm combines the effectiveness of the convolutional autoencoder model alongside local structure preservation constraints and regularization. The proposed model is compared with state-of-the-art deep clustering algorithms to exhibit the efficacy of the RDC under different modes of clustering performance evaluation. This system acts as an effective tool for targeted users by providing them better assistance on plant disease diagnosis.

Spectral Viterbi Algorithm for Contactless Wide-Range Heart Rate Estimation With Deep Clustering

Objective: The main challenge in contactless heartbeat detection comes from breathing and/or body motion, which typically deteriorate heart rate (HR) measurements, due to incorrect selection of spectral peaks associated with HR. To acquire the reliable peak selection on spectrum within a relatively broad range, this article first proposes a spectral Viterbi algorithm. Second, a nonlinear source separation approach is further proposed to eliminate the noises generated by respiration and movements, suppressing the undesired spectral energy. Proposal: Inspired by the fact that the period of peak-to-peak intervals of heartbeat (RRIs) rarely vary within a short duration, a novel spectral Viterbi algorithm is proposed to estimate HR change, by the path metric (PM) of candidate paths of HR change. Moreover, based on a deep recurrent neural network (RNN), deep clustering (DC) is applied to separate out the targeted heartbeat source from Doppler signal, by dividing its spectrogram. Results: On the premise of wide-range HR measurement, the usage of spectral Viterbi algorithm substantially improved the precision compared with typical methods of HR estimation, both in the statuses of human subjects' sitting still and typewriting. In addition, the combination of DC obtains the smallest average errors. Significance: The proposed spectral Viterbi algorithm with DC is provided with three main strengths: 1) good adaptability to wide-range HR change; 2) robustness to nonlinearly mixed signal and noises; and 3) requirement of only a single-channel sensor.

A Deep Clustering via Automatic Feature Embedded Learning for Human Activity Recognition

Traditional clustering algorithms are widely used for building bag-of-words (BOW) models to aggregate spatio-temporal feature points extracted from a video for human activity recognition problems. Their performances are restricted by the computational complexity which limits the number of feature points being used. In contrast, deep clustering yields good clustering performance without the limit of the number of feature points. Therefore, this work proposes a dual stacked autoencoders features embedded clustering (DSAFEC) and a BOW construction method based on the DSAFEC (B-DSAFEC) to reduce the computational complexity and to remove the selection restriction. The DSAFEC first transforms feature points extracted from a video to a learned feature space and then probabilities of cluster assignment of feature points are predicted to build BOWs for human activity recognition. A soft clustering is used by assigning each feature point to multiple clusters yielding the largest probabilities instead of only one in hard clustering. Experimental results on three benchmark human activity datasets show that the B-DSAFEC yields better performance compared to five reference methods which are developed based on either traditional clustering methods or deep clustering methods.

Region-factorized recurrent attentional network with deep clustering for radar HRRP target recognition

Feature extraction plays an essential role in radar automatic target recognition (RATR) with high-resolution range profiles (HRRPs). Traditional feature extraction algorithms usually ignore that different regions in HRRP contain the information with different importance, resulting in their inadequacy in characterizing HRRP data. In this work, we propose a region factorized recurrent attentional network (RFRAN) for HRRP-RATR by making use of the temporal dependence through recurrent neural network (RNN) and automatically finding the informative regions by a deep clustering mechanism in HRRP samples, which reflects the distribution of scatterers in target along range dimension. Specifically, we represent the temporal RNN hidden state using a region factorized encoder whose parameters are conditioned on the HRRP region cluster centers. Moreover an attention mechanism is used to weight up the different recognition contribution of each time step’s hidden state. The aim of all the above modules is to achieve a more informative and discriminative feature. Crucially, the loss function of RFRAN is differentiable, so all components can be jointly trained with a gradient-based optimization. Compared with traditional methods, besides the competitive recognition performance, RFRAN has a promising interpretability thanks to the sequential region-specific hidden states.