Image Clustering Tasks Research Articles

Image clustering using generated text centroids

In recent years, deep neural networks pretrained on large-scale datasets have been used to address data deficiency and achieve better performance through prior knowledge. Contrastive language–image pretraining (CLIP), a vision-language model pretrained on an extensive dataset, achieves better performance in image recognition. In this study, we harness the power of multimodality in image clustering tasks, shifting from a single modality to a multimodal framework using the describability property of image encoder of the CLIP model. The importance of this shift lies in the ability of multimodality to provide richer feature representations. By generating text centroids corresponding to image features, we effectively create a common descriptive language for each cluster. It generates text centroids assigned by the image features and improves the clustering performance. The text centroids use the results generated by using the standard clustering algorithm as a pseudo-label and learn a common description of each cluster. Finally, only text centroids were added when the image features on the same space were assigned to the text centroids, but the clustering performance improved significantly compared to the standard clustering algorithm, especially on complex datasets. When the proposed method is applied, the normalized mutual information score rises by 32% on the Stanford40 dataset and 64% on ImageNet-Dog compared to the k-means clustering algorithm.

Semi-supervised nonnegative matrix factorization with label propagation and constraint propagation

Semi-supervised nonnegative matrix factorization (SNMF) improves the clustering effect of nonnegative matrix factorization (NMF) by integrating label information. However, as one of the most commonly used semi-supervised learning methods, label propagation algorithm (LP) has some limitations due to its heavy dependence on the predefined similarity matrix. To address this deficiency and effectively use limited supervisory information, a structural NMF algorithm with label propagation and constraint propagation (PNLP-SCNMF) is proposed in this paper, which simultaneously performs data representation and classification in a joint manner. Both positive label information and negative label information are introduced into the proposed method, and the factor matrix is constrained by label information to construct the tri-decomposition form. Constraint propagation algorithm (CPA) is used to correct the constructed similarity matrix, so that the low-dimensional subspace obtained by matrix decomposition can retain the geometric structure characteristics of the original sample space. In addition, an efficient iterative update optimization scheme is proposed to solve this objective function, and its convergence is proved theoretically. A large number of comparative experiments on nine real datasets show that the proposed method can consistently achieve higher clustering accuracy in image clustering tasks.

Multitask Image Clustering via Deep Information Bottleneck.

Multitask image clustering approaches intend to improve the model accuracy on each task by exploring the relationships of multiple related image clustering tasks. However, most existing multitask clustering (MTC) approaches isolate the representation abstraction from the downstream clustering procedure, which makes the MTC models unable to perform unified optimization. In addition, the existing MTC relies on exploring the relevant information of multiple related tasks to discover their latent correlations while ignoring the irrelevant information between partially related tasks, which may also degrade the clustering performance. To tackle these issues, a multitask image clustering method named deep multitask information bottleneck (DMTIB) is devised, which aims at conducting multiple related image clustering by maximizing the relevant information of multiple tasks while minimizing the irrelevant information among them. Specifically, DMTIB consists of a main-net and multiple subnets to characterize the relationships across tasks and the correlations hidden in a single clustering task. Then, an information maximin discriminator is devised to maximize the mutual information (MI) measurement of positive samples and minimize the MI of negative ones, in which the positive and negative sample pairs are constructed by a high-confidence pseudo-graph. Finally, a unified loss function is devised for the optimization of task relatedness discovery and MTC simultaneously. Empirical comparisons on several benchmark datasets, NUS-WIDE, Pascal VOC, Caltech-256, CIFAR-100, and COCO, show that our DMTIB approach outperforms more than 20 single-task clustering and MTC approaches.

Confusion-Based Metric Learning for Regularizing Zero-Shot Image Retrieval and Clustering.

Deep metric learning turns to be attractive in zero-shot image retrieval and clustering (ZSRC) task in which a good embedding/metric is requested such that the unseen classes can be distinguished well. Most existing works deem this "good" embedding just to be the discriminative one and race to devise the powerful metric objectives or the hard-sample mining strategies for learning discriminative deep metrics. However, in this article, we first emphasize that the generalization ability is also a core ingredient of this "good" metric and it largely affects the metric performance in zero-shot settings as a matter of fact. Then, we propose the confusion-based metric learning (CML) framework to explicitly optimize a robust metric. It is mainly achieved by introducing two interesting regularization terms, i.e., the energy confusion (EC) and diversity confusion (DC) terms. These terms daringly break away from the traditional deep metric learning idea of designing discriminative objectives and instead seek to "confuse" the learned model. These two confusion terms focus on local and global feature distribution confusions, respectively. We train these confusion terms together with the conventional deep metric objective in an adversarial manner. Although it seems weird to "confuse" the model learning, we show that our CML indeed serves as an efficient regularization framework for deep metric learning and it is applicable to various conventional metric methods. This article empirically and experimentally demonstrates the importance of learning an embedding/metric with good generalization, achieving the state-of-the-art performances on the popular CUB, CARS, Stanford Online Products, and In-Shop datasets for ZSRC tasks.

Learning adaptive Grassmann neighbors for image-set analysis

Representing image sets as subspaces on Grassmann manifold and leveraging the Riemannian geometry of this space has proven to be highly effective in various visual recognition tasks. However, the computational cost associated with operating on Riemannian manifolds, especially of high-dimensional ones, poses a significant limitation to the applicability of existing techniques. Moreover, the image sets with labels are scarce and valuable in real-world scenarios, where the manual annotation is time-consuming and costly. To tackle these issues, in this paper, we propose an unsupervised Grassmannian dimensionality reduction method for image set classification and clustering, which simultaneously performs local structure learning and dimensionality reduction under the subspace view. Specifically, instead of relying on the pre-defined similarity matrix that obtained from raw data and may contain much noise, we adaptively learn the local structures from the projected low-dimensional Grassmann manifold. The refined structures then provide more discriminative power to learn the accurate manifold-to-manifold mapping. To better preserve the global structure of the data, this mapping is constrained to satisfy the orthogonal condition. In addition, our model can be compatible with F-norm OLPP for Grassmann manifold when the squared F-norm is transformed into projection metric. A re-weighted iterative scheme is carefully developed to optimize the problem. Experimental results on several benchmark datasets for image set classification and clustering tasks clearly demonstrate that our model consistently outperforms baselines, yielding significant improvements.

Semantic-Enhanced Image Clustering

Image clustering is an important and open challenging task in computer vision. Although many methods have been proposed to solve the image clustering task, they only explore images and uncover clusters according to the image features, thus being unable to distinguish visually similar but semantically different images. In this paper, we propose to investigate the task of image clustering with the help of visual-language pre-training model. Different from the zero-shot setting, in which the class names are known, we only know the number of clusters in this setting. Therefore, how to map images to a proper semantic space and how to cluster images from both image and semantic spaces are two key problems. To solve the above problems, we propose a novel image clustering method guided by the visual-language pre-training model CLIP, named Semantic-Enhanced Image Clustering (SIC). In this new method, we propose a method to map the given images to a proper semantic space first and efficient methods to generate pseudo-labels according to the relationships between images and semantics. Finally, we propose to perform clustering with consistency learning in both image space and semantic space, in a self-supervised learning fashion. The theoretical result of convergence analysis shows that our proposed method can converge at a sublinear speed. Theoretical analysis of expectation risk also shows that we can reduce the expectation risk by improving neighborhood consistency, increasing prediction confidence, or reducing neighborhood imbalance. Experimental results on five benchmark datasets clearly show the superiority of our new method.

Mutated Aquila Optimizer for assisting brain tumor segmentation

Image processing is a crucial effort to discover suspicious regions and robust features from magnetic resonance imaging (MRI). Many segmentation algorithms have been proposed to increase the accuracy of brain tumor detection. However, brain image segmentation is considered a complex and challenging step in medical image processing. K-means is an unsupervised learning method for MRI image clustering and other tasks such as image segmentation and pattern identification of interest. However, the K-means algorithm still has many shortcomings, including a low convergence rate, a tendency to get stuck in local minima, and sensitivity to initialization. An improved version of Aquila Optimizer (AO) boosted with an alternative selection scheme and Cauchy mutation, called Mutated Aquila Optimizer (MAO), and the K-Means algorithm were combined to create a new dynamic and intelligent clustering method for brain tumor segmentation. The proposed MAO-Kmeans approach aims to increase the ability of K-Means to automatically extract the correct number and location of cluster centers and the number of pixels in each cluster in abnormal MRI images (multiple sclerosis lesions). The experimental results on the Brain Tumor Segmentation dataset (BraTS 2020) demonstrated the effectiveness of the MAO-Kmeans method in improving the performance of conventional K-means clustering. Furthermore, qualitative and quantitative comparisons between MAO-Kmeans and the other 12 state-of-the-art segmentation techniques (which include machine learning, artificial neural network, and deep learning-based methods) demonstrate the superiority of our proposed algorithm.

Multi-View Diffusion Process for Spectral Clustering and Image Retrieval.

This paper presents a novel approach to multi-view graph learning that combines weight learning and graph learning in an alternating optimization framework. Multi-view graph learning refers to the problem of constructing a unified affinity graph using heterogeneous sources of data representation, which is a popular technique in many learning systems where no prior knowledge of data distribution is available. Our approach is based on a fusion-and-diffusion strategy, in which multiple affinity graphs are fused together via a weight learning scheme based on the unsupervised graph smoothness and utilised as a consensus prior to the diffusion. We propose a novel multi-view diffusion process that learns a manifold-aware affinity graph by propagating affinities on tensor product graphs, leveraging high-order contextual information to enhance pairwise affinities. In contrast to existing multi-view graph learning approaches, our approach is not limited by the quality of initial graphs or the assumption of a latent common subspace among multiple views. Instead, our approach is able to identify the consistency among views and fuse multiple graphs adaptively. We formulate both weight learning and diffusion-based affinity learning in a unified framework and propose an alternating optimization solver that is guaranteed to converge. The proposed approach is applied to image retrieval and clustering tasks on 16 real-world datasets. Extensive experimental results demonstrate that our approach outperforms state-of-the-art methods for both retrieval and clustering on 13 out of 16 datasets.

Deep metric learning via group channel-wise ensemble

Deep metric learning aims at learning the distance metric for data samples by deep neural networks. Essentially, it derives an embedding space where the mappings of semantically related samples are much closer than those of irrelevant ones. When learning a single metric using all samples, it fails to well encode the sample similarity in several aspects, such as color, shape, and appearance. To handle this issue, some works simply divide the linear layer into multiple sub-layers, and the data subsets for learning multiple metrics are randomly generated at sample level. This usually makes the samples in the same class appear in different subspaces. Therefore, we propose a Group Channel-wise Ensemble (GCE) method that devises a group channel-wise convolution block in convolution networks for learning multiple distance metrics. Its core idea is that the channels of convolution layer encode different similarity metrics in groups. Specifically, the embedding space is partitioned into several subspaces, where one base learner captures one distance metric. Moreover, the class-proxy based meta-class update strategy is adopted for generating data subsets. This strategy learns discriminative distance metrics by optimizing the triplet loss on training samples. Empirical studies on four benchmarks demonstrate that GCE achieves promising performances in image retrieval and clustering tasks.

Recursive Sample Scaling Low-Rank Representation

The low-rank representation (LRR) method has recently gained enormous popularity due to its robust approach in solving the subspace segmentation problem, particularly those concerning corrupted data. In this paper, the recursive sample scaling low-rank representation (RSS-LRR) method is proposed. The advantage of RSS-LRR over traditional LRR is that a cosine scaling factor is further introduced, which imposes a penalty on each sample to minimize noise and outlier influence better. Specifically, the cosine scaling factor is a similarity measure learned to extract each sample’s relationship with the low-rank representation’s principal components in the feature space. In order words, the smaller the angle between an individual data sample and the low-rank representation’s principal components, the more likely it is that the data sample is clean. Thus, the proposed method can then effectively obtain a good low-rank representation influenced mainly by clean data. Several experiments are performed with varying levels of corruption on ORL, CMU PIE, COIL20, COIL100, and LFW in order to evaluate RSS-LRR’s effectiveness over state-of-the-art low-rank methods. The experimental results show that RSS-LRR consistently performs better than the compared methods in image clustering and classification tasks.

Cross-Domain Object Representation via Robust Low-Rank Correlation Analysis

Cross-domain data has become very popular recently since various viewpoints and different sensors tend to facilitate better data representation. In this article, we propose a novel cross-domain object representation algorithm (RLRCA) which not only explores the complexity of multiple relationships of variables by canonical correlation analysis (CCA) but also uses a low rank model to decrease the effect of noisy data. To the best of our knowledge, this is the first try to smoothly integrate CCA and a low-rank model to uncover correlated components across different domains and to suppress the effect of noisy or corrupted data. In order to improve the flexibility of the algorithm to address various cross-domain object representation problems, two instantiation methods of RLRCA are proposed from feature and sample space, respectively. In this way, a better cross-domain object representation can be achieved through effectively learning the intrinsic CCA features and taking full advantage of cross-domain object alignment information while pursuing low rank representations. Extensive experimental results on CMU PIE, Office-Caltech, Pascal VOC 2007, and NUS-WIDE-Object datasets, demonstrate that our designed models have superior performance over several state-of-the-art cross-domain low rank methods in image clustering and classification tasks with various corruption levels.

Neighborhood preserving embedding on Grassmann manifold for image-set analysis

Modeling image sets as points on Grassmann manifold has attracted increasing interests in computer vision community and has been applied to many applications. However, such approaches have suffered from the limitation that high computational cost on Grassmann manifold must be involved, especially high-dimensional ones. In this paper, we propose an unsupervised robust dimensionality reduction algorithm for Grassmann manifold based on Neighborhood Preserving Embedding (GNPE). We first introduce two strategies to construct the coefficients-based similarity graph to eliminate the effects of errors. Then, a projection is learned from the high-dimensional Grassmann manifold to the relative low-dimensional one with more discriminative capability, where the local neighborhood structure is well preserved. To address the issue that the estimated similarity graph is unreliable with noise and outliers, we further propose a unified learning framework which performs similarity learning and projection learning simultaneously. By leveraging the interactions between these two essential tasks, we can capture accurate structures and learn discriminative projections. The proposed method can be optimized by an efficient iterative algorithm. Experiments on various image set classification and clustering tasks clearly show that our model achieves consistent improvements in terms of both effectiveness and efficiency.

Soft and self constrained clustering for group-based labeling.

When using deep neural networks in medical image classification tasks, it is mandatory to prepare a large-scale labeled image set, and this often requires significant effort by medical experts. One strategy to reduce the labeling cost is group-based labeling, where image samples are clustered and then a label is attached to each cluster. The efficiency of this strategy depends on the purity of the clusters. Constrained clustering is an effective way to improve the purity of the clusters if we can give appropriate must-links and cannot-links as constraints. However, for medical image clustering, the conventional constrained clustering methods encounter two issues. The first issue is that constraints are not always appropriate due to the gap between semantic and visual similarities. The second issue is that attaching constraints requires extra effort from medical experts. To deal with the first issue, we propose a novel soft-constrained clustering method, which has the ability to ignore inappropriate constraints. To deal with the second issue, we propose a self-constrained clustering method that utilizes prior knowledge about the target images to set the constraints automatically. Experiments with the endoscopic image datasets demonstrated that the proposed methods give clustering results with higher purity.

Representation Learning Based on Autoencoder and Deep Adaptive Clustering for Image Clustering

Image clustering is a complex procedure, which is significantly affected by the choice of image representation. Most of the existing image clustering methods treat representation learning and clustering separately, which usually bring two problems. On the one hand, image representations are difficult to select and the learned representations are not suitable for clustering. On the other hand, they inevitably involve some clustering step, which may bring some error and hurt the clustering results. To tackle these problems, we present a new clustering method that efficiently builds an image representation and precisely discovers cluster assignments. For this purpose, the image clustering task is regarded as a binary pairwise classification problem with local structure preservation. Specifically, we propose here such an approach for image clustering based on a fully convolutional autoencoder and deep adaptive clustering (DAC). To extract the essential representation and maintain the local structure, a fully convolutional autoencoder is applied. To manipulate feature to clustering space and obtain a suitable image representation, the DAC algorithm participates in the training of autoencoder. Our method can learn an image representation that is suitable for clustering and discover the precise clustering label for each image. A series of real-world image clustering experiments verify the effectiveness of the proposed algorithm.

Improving Deep Metric Learning by Divide and Conquer.

Deep metric learning (DML) is a cornerstone of many computer vision applications. It aims at learning a mapping from the input domain to an embedding space, where semantically similar objects are located nearby and dissimilar objects far from another. The target similarity on the training data is defined by user in form of ground-truth class labels. However, while the embedding space learns to mimic the user-provided similarity on the training data, it should also generalize to novel categories not seen during training. Besides user-provided groundtruth training labels, a lot of additional visual factors (such as viewpoint changes or shape peculiarities) exist and imply different notions of similarity between objects, affecting the generalization on the images unseen during training. However, existing approaches usually directly learn a single embedding space on all available training data, struggling to encode all different types of relationships, and do not generalize well. We propose to build a more expressive representation by jointly splitting the embedding space and the data hierarchically into smaller sub-parts. We successively focus on smaller subsets of the training data, reducing its variance and learning a different embedding subspace for each data subset. Moreover, the subspaces are learned jointly to cover not only the intricacies, but the breadth of the data as well. Only after that, we build the final embedding from the subspaces in the conquering stage. The proposed algorithm acts as a transparent wrapper that can be placed around arbitrary existing DML methods. Our approach significantly improves upon the state-of-the-art on image retrieval, clustering, and re-identification tasks evaluated using CUB200-2011, CARS196, Stanford Online Products, In-shop Clothes, and PKU VehicleID datasets.

Combining pretrained CNN feature extractors to enhance clustering of complex natural images

Recently, a common starting point for solving complex unsupervised image classification tasks is to use generic features, extracted with deep Convolutional Neural Networks (CNN) pretrained on a large and versatile dataset (ImageNet). However, in most research, the CNN architecture for feature extraction is chosen arbitrarily, without justification. This paper aims at providing insight on the use of pretrained CNN features for image clustering (IC). First, extensive experiments are conducted and show that, for a given dataset, the choice of the CNN architecture for feature extraction has a huge impact on the final clustering. These experiments also demonstrate that proper extractor selection for a given IC task is difficult. To solve this issue, we propose to rephrase the IC problem as a multi-view clustering (MVC) problem that considers features extracted from different architectures as different “views” of the same data. This approach is based on the assumption that information contained in the different CNN may be complementary, even when pretrained on the same data. We then propose a multi-input neural network architecture that is trained end-to-end to solve the MVC problem effectively. This approach is tested on nine natural image datasets, and produces state-of-the-art results for IC.

Robust semi-supervised nonnegative matrix factorization for image clustering

Nonnegative matrix factorization (NMF) is a powerful dimension reduction method, and has received increasing attention in various practical applications. However, most traditional NMF based algorithms are sensitive to noisy data, or fail to fully utilize the limited supervised information. In this paper, a novel robust semi-supervised NMF method, namely correntropy based semi-supervised NMF (CSNMF), is proposed to solve these issues. Specifically, CSNMF adopts a correntropy based loss function instead of the squared Euclidean distance (SED) in constrained NMF to suppress the influence of non-Gaussian noise or outliers contaminated in real world data, and simultaneously uses two types of supervised information, i.e., the pointwise and pairwise constraints, to obtain the discriminative data representation. The proposed method is analyzed in terms of convergence, robustness and computational complexity. The relationships between CSNMF and several previous NMF based methods are also discussed. Extensive experimental results show the effectiveness and robustness of CSNMF in image clustering tasks, compared with several state-of-the-art methods.

Robust sparse coding via self-paced learning for data representation

Sparse coding (SC), due to its thorough theoretical property and outstanding effectiveness, is attracting more and more attention in various data representation and data mining applications. However, the optimization of most existing sparse coding algorithms are non-convex and thus prone to become stuck into bad local minima under the framework of alternative optimization, especially when there are many outliers and noisy data. To enhance the learning robustness, in this study, we will present an unified framework named Self-Paced Sparse Coding (SPSC), which gradually includes data into the learning process of SC from easy ones to complex ones by incorporating self-paced learning methodology. It implements a soft instance selection accordingly rather than a heuristic hard strategy sample selection. We also generalize the self-paced learning schema into different levels of dynamic selection on instances, features and elements respectively. Further, we show an optimization algorithm to solve it and a theoretical explanation to analyze the effectiveness of it. Extensive experimental results on the real-world clean image datasets and images with two kinds of corruptions demonstrate the remarkable robustness of the proposed method for high dimensional data representation on image clustering and reconstruction tasks over the state-of-the-arts.

High-Rankness Regularized Semi-Supervised Deep Metric Learning for Remote Sensing Imagery

Deep metric learning has recently received special attention in the field of remote sensing (RS) scene characterization, owing to its prominent capabilities for modeling distances among RS images based on their semantic information. Most of the existing deep metric learning methods exploit pairwise and triplet losses to learn the feature embeddings with the preservation of semantic-similarity, which requires the construction of image pairs and triplets based on the supervised information (e.g., class labels). However, generating such semantic annotations becomes a completely unaffordable task in large-scale RS archives, which may eventually constrain the availability of sufficient training data for this kind of models. To address this issue, we reformulate the deep metric learning scheme in a semi-supervised manner to effectively characterize RS scenes. Specifically, we aim at learning metric spaces by utilizing the supervised information from a small number of labeled RS images and exploring the potential decision boundaries for massive sets of unlabeled aerial scenes. In order to reach this goal, a joint loss function, composed of a normalized softmax loss with margin and a high-rankness regularization term, is proposed, as well as its corresponding optimization algorithm. The conducted experiments (including different state-of-the-art methods and two benchmark RS archives) validate the effectiveness of the proposed approach for RS image classification, clustering and retrieval tasks. The codes of this paper are publicly available.

Robust low rank representation via feature and sample scaling

Low-rank representation (LRR) is a very competitive technique in various real-world applications for its powerful capability in discovering latent structure of noisy or corrupted data set. However, traditional low-rank models treat each data point and feature equally so that noisy data cannot be detected and suppressed effectively and have obvious deterioration in performance, especially in heavy noisy scenario. In this paper, to address this problem, we develop a method of feature and sample scaling for low rank representation. The importance of data points and their features in both feature and sample spaces are considered, as such, clean data points and noisy data points and their features can be distinguished. In addition, based on the observation that noisy data points are usually deviated far away from the principal projection of the data set, a cosine similarity metric between data vector and the principal projection vector is developed to measure the importance of each sample. Applying our method into two classical low rank models such as Low Rank Representation (LRR) and Bilinear Factorization (BF), we can learn better low-rank structure of clean data, while the outliers or missing data being suppressed. Extensive experimental results on ORL, COIL20 and video surveillance, demonstrate that our proposed method can outperform state-of-the-art low rank methods in image clustering tasks with various levels of corruptions, especially in a heavy noisy scenario.