Unsupervised Image Clustering Research Articles

Auroral breakup detection in all-sky images by unsupervised learning

Abstract. Due to a large number of automatic auroral camera systems on the ground, image data analysis requires more efficiency than what human expert visual inspection can provide. Furthermore, there is no solid consensus on how many different types or shapes exist in auroral displays. We report the first attempt to classify auroral morphological forms by an unsupervised learning method on an image set that contains both nightside and dayside aurora. We used 6 months of full-colour auroral all-sky images captured at a high-Arctic observatory on Svalbard, Norway, in 2019–2020. The selection of images containing aurora was performed manually. These images were then input into a convolutional neural network called SimCLR for feature extraction. The clustered and fused features resulted in 37 auroral morphological clusters. In the clustering of auroral image data with two different time resolutions, we found that the occurrence of 8 clusters strongly increased when the image cadence was high (24 s), while the occurrence of 14 clusters experienced little or no change with changes in input image cadence. We therefore investigated the temporal evolution of a group of eight “active aurora” clusters. Time periods for which this active aurora persisted for longer than two consecutive images with a maximum cadence of 6 min coincided with ground-magnetic deflections, and their occurrence was found to maximize around magnetic midnight. The active aurora onsets typically included vortical auroral structures and equivalent current patterns typical for substorms. Our findings therefore suggest that our unsupervised image clustering method can be used to detect auroral breakups in ground-based image datasets with a temporal accuracy determined by the image cadence.

A new commercial boundary dataset for metropolitan areas in the USA and Canada, built from open data

The purpose of this study is to define the geographic boundaries of commercial areas by creating a consistent definition, combining various commercial area types, including downtowns, retail centres, financial districts, and other employment subcentres. Our research involved the collection of office, retail and job density data from 69 metropolitan regions across USA and Canada. Using this data, we conducted an unsupervised image segmentation model and clustering methods to identify distinctive commercial geographic boundaries. As a result, we identified 23,751 commercial areas, providing a detailed perspective on the commercial landscape of metropolitan areas in the USA and Canada. In addition, the generated boundaries were successfully validated through comparison with previously established commerce-related boundaries. The output of this study has implications for urban and regional planning and economic development, delivering valuable insights into the overall commercial geography in the region. The commercial boundary and used codes are freely available on the School of Cities Github, and users can reuse, reproduce and modify the boundaries.

Convergence or divergence? A cross-platform analysis of climate change visual content categories, features, and social media engagement on Twitter and Instagram

Advocacy organizations increasingly leverage social media and visuals to communicate complex climate issues. By examining an extensive dataset of visual posts collected from five organization accounts on two multimodal social media platforms, Twitter and Instagram, we conducted a cross-platform comparison of visual content categories and visual features related to climate change. Through deep-learning-based unsupervised image clustering, we discovered that visual content on both platforms could be broadly classified into five categories: infographics/captioned images, nature landscape/wildlife, climate activism, technology, and data visualization. However, these categories were not equally represented on each platform. Instagram featured more nature landscape/wildlife content, while Twitter showed more infographics/captioned images and data visualization. Through computational visual analysis, we found that the two platforms also presented significant differences in overall warm and cool colors, brightness, colorfulness, visual complexity, and presence of faces. Additionally, we identified platform-specific patterns of engagement associated with these categories and features. With the urgency to address climate change, these findings can guide climate advocacy organizations in developing strategies tailored to each platform’s specific characteristics for maximum effectiveness. This study highlights the significance of using computational methods in efficiently uncovering meaningful themes from extensive visual data and quantifying aesthetic features in strategic communication.

The Mobilizing Power of Visual Media Across Stages of Social-Mediated Protests

ABSTRACT The popularity of camera phones, the availability of photo-editing apps, and the rise of visually oriented social media platforms have made it convenient for citizens to produce and circulate visual content in contentious politics. While scholars have increasingly recognized the role of visuals in mobilizing social-mediated protests, how different types of visuals affect message engagement across different stages of protests remains underexplored. For this study, we analyzed approximately ten million tweets from Twitter for three social-mediated protests (Black Lives Matter, Stop Asian Hate, and Women’s March). We found that posts with images and videos generally attracted more audience engagement than their textual counterparts. Unpacking the role of visual media across different modalities and stages of social-mediated protests, we found that the superior effects of visuals were generally more pronounced during the ignition phase of the protest than the periods before and after. By applying unsupervised image clustering on millions of protest visuals, we systematically established four common visual content categories: crowd-based protest photos, non-crowd-protest human photos, non-human photos, and non-photograph visuals. We revealed heterogeneous effects on audience engagement across content categories and protests, and explored these categories through qualitative analysis of most-engaged visuals. These findings enrich our understanding of the mobilizing power of visual media in social movements and shed light on effective communication strategies regarding social inequalities.

Deep expectation-maximization network for unsupervised image segmentation and clustering

Unsupervised learning, such as unsupervised image segmentation and clustering, are fundamental tasks in image representation learning. In this paper, we design a deep expectation-maximization (DEM) network for unsupervised image segmentation and clustering. It is based on the statistical modeling of image in its latent feature space by Gaussian mixture model (GMM), implemented in a novel deep learning framework. Specifically, in the unsupervised setting, we design an auto-encoder network and an EM module over the image latent features, for jointly learning the image latent features and GMM model of the latent features in a single framework. To construct the EM-module, we unfold the iterative operations of EM algorithm and the online EM algorithm in fixed steps to be differentiable network blocks, plugged into the network to estimate the GMM parameters of the image latent features. The proposed network parameters can be end-to-end optimized using losses based on log-likelihood of GMM, entropy of Gaussian component assignment probabilities and image reconstruction error. Extensive experiments confirm that our proposed networks achieve favorable results compared with several state-of-the-art methods in unsupervised image segmentation and clustering.

Automatic detection of thin oil films on water surfaces in ultraviolet imagery

AbstractAmong the various remote sensing technologies that have been applied to monitor oil spills on the sea surface, passive ultraviolet (UV) imaging is a controversial one that has raised some disputes in the community of oil spill remote sensing. As a result, the research and applications of oil spill detection using passive UV imaging have not been as developed as other methods. In order to clarify some existing questions on oil spill detection using passive UV remote sensing technology, this paper discusses the needs of thin oil film detection, examines the feasibility of thin oil film detection using passive UV imaging through field experiments under controlled conditions and validates it with the UV imagery derived from the airborne visible/infrared imaging spectrometer (AVIRIS) observation of the Deepwater Horizon oil spill. Two types of fully automatic models are designed to extract the thin oil films on the water surface: (1) a binary classification model based on an adaptive threshold; (2) an unsupervised image segmentation model based on image clustering and greyscale histogram analysis. The two models are tested on the UV imagery obtained through both field experiments and AVIRIS observations. The results indicate that the binary classification model can extract the thin oil films with reasonable accuracy under stable imaging conditions, while the unsupervised image clustering model can robustly detect the thin oil films at the cost of higher computational complexity. These results infer that passive UV imaging is an effective way to detect thin oil films and could be applied to provide early warning at the beginning stage of oil spills and reduce further damage. It may also be applied as a supplementary method for oil spill detection to achieve comprehensive oil spill monitoring.

A Novel Image Clustering Algorithm Based on Supported Nearest Neighbors

Unsupervised image clustering is a challenging task in computer vision. Recently, various deep clustering algorithms based on contrastive learning have achieved promising performance and some distinguishable features representation were obtained only by taking different augmented views of same image as positive pairs and maximizing their similarities, whereas taking other images’ augmentations in the same batch as negative pairs and minimizing their similarities. However, due to the fact that there is more than one image in a batch belong to the same class, simply pushing the negative instances apart will result in inter-class conflictions and lead to the clustering performance degradation. In order to solve this problem, we propose a deep clustering algorithm based on supported nearest neighbors (SNDC), which constructs positive pairs of current images by maintaining a support set and find its k nearest neighbors from the support set. By going beyond single instance positive, SNDC can learn more generalized features representation with inherent semantic meaning and therefore alleviating inter-class conflictions. Experimental results on multiple benchmark datasets show that the performance of SNDC is superior to the state-of-the-art clustering models, with accuracy improvement of 6.2% and 20.5% on CIFAR-10 and ImageNet-Dogs respectively.

ProFeat: Unsupervised image clustering via progressive feature refinement

Unsupervised image clustering is a chicken-and-egg problem that involves representation learning and clustering. To resolve the inter-dependency between them, many approaches that iteratively perform the two tasks have been proposed, but their accuracy is limited due to inaccurate intermediate representations and clusters. To overcome this, this paper proposes ProFeat, a novel iterative approach to unsupervised image clustering based on progressive feature refinement. To learn discriminative features for clustering while avoiding adversarial influence from inaccurate intermediate clusters, ProFeat rigorously divides representation learning and clustering by modeling a neural network for clustering as a composition of an embedding and a clustering function and introducing an auxiliary embedding function. ProFeat progressively refines representations using confident samples from intermediate clusters using an extended contrastive loss. This paper also proposes ensemble-based feature refinement for more robust clustering. Our experiments demonstrate that ProFeat achieves superior results compared to previous methods.

DNB: A Joint Learning Framework for Deep Bayesian Nonparametric Clustering.

Clustering algorithms based on deep neural networks have been widely studied for image analysis. Most existing methods require partial knowledge of the true labels, namely, the number of clusters, which is usually not available in practice. In this article, we propose a Bayesian nonparametric framework, deep nonparametric Bayes (DNB), for jointly learning image clusters and deep representations in a doubly unsupervised manner. In doubly unsupervised learning, we are dealing with the problem of "unknown unknowns," where we estimate not only the unknown image labels but also the unknown number of labels as well. The proposed algorithm alternates between generating a potentially unbounded number of clusters in the forward pass and learning the deep networks in the backward pass. With the help of the Dirichlet process mixtures, the proposed method is able to partition the latent representations space without specifying the number of clusters a priori. An important feature of this work is that all the estimation is realized with an end-to-end solution, which is very different from the methods that rely on post hoc analysis to select the number of clusters. Another key idea in this article is to provide a principled solution to the problem of "trivial solution" for deep clustering, which has not been much studied in the current literature. With extensive experiments on benchmark datasets, we show that our doubly unsupervised method achieves good clustering performance and outperforms many other unsupervised image clustering methods.

Maturity Grading and Identification of Camellia oleifera Fruit Based on Unsupervised Image Clustering

Maturity grading and identification of Camellia oleifera are prerequisites to determining proper harvest maturity windows and safeguarding the yield and quality of Camellia oil. One problem in Camellia oleifera production and research is the worldwide confusion regarding the grading and identification of Camellia oleifera fruit maturity. To solve this problem, a Camellia oleifera fruit maturity grading and identification model based on the unsupervised image clustering model DeepCluster has been developed in the current study. The proposed model includes the following two branches: a maturity grading branch and a maturity identification branch. The proposed model jointly learns the parameters of the maturity grading branch and maturity identification branch and used the maturity clustering assigned from the maturity grading branch as pseudo-labels to update the parameters of the maturity identification branch. The maturity grading experiment was conducted using a training set consisting of 160 Camellia oleifera fruit samples and 2628 Camellia oleifera fruit digital images collected using a smartphone. The proposed model for grading Camellia oleifera fruit samples and images in training set into the following three maturity levels: unripe (47 samples and 883 images), ripe (62 samples and 1005 images), and overripe (51 samples and 740 images). Results suggest that there was a significant difference among the maturity stages graded by the proposed method with respect to seed oil content, seed soluble protein content, seed soluble sugar content, seed starch content, dry seed weight, and moisture content. The maturity identification experiment was conducted using a testing set consisting of 160 Camellia oleifera fruit digital images (50 unripe, 60 ripe, and 50 overripe) collected using a smartphone. According to the results, the overall accuracy of maturity identification for Camellia oleifera fruit was 91.25%. Moreover, a Gradient-weighted Class Activation Mapping (Grad-CAM) visualization analysis reveals that the peel regions, crack regions, and seed regions were the critical regions for Camellia oleifera fruit maturity identification. Our results corroborate a maturity grading and identification application of unsupervised image clustering techniques and are supported by additional physical and quality properties of maturity. The current findings may facilitate the harvesting process of Camellia oleifera fruits, which is especially critical for the improvement of Camellia oil production and quality.

AutoNovel: Automatically Discovering and Learning Novel Visual Categories.

We tackle the problem of discovering novel classes in an image collection given labelled examples of other classes. We present a new approach called AutoNovel to address this problem by combining three ideas: (1) we suggest that the common approach of bootstrapping an image representation using the labelled data only introduces an unwanted bias, and that this can be avoided by using self-supervised learning to train the representation from scratch on the union of labelled and unlabelled data; (2) we use ranking statistics to transfer the model's knowledge of the labelled classes to the problem of clustering the unlabelled images; and, (3) we train the data representation by optimizing a joint objective function on the labelled and unlabelled subsets of the data, improving both the supervised classification of the labelled data, and the clustering of the unlabelled data. Moreover, we propose a method to estimate the number of classes for the case where the number of new categories is not known a priori. We evaluate AutoNovel on standard classification benchmarks and substantially outperform current methods for novel category discovery. In addition, we also show that AutoNovel can be used for fully unsupervised image clustering, achieving promising results.

Image Clustering: An Unsupervised Approach to Categorize Visual Data in Social Science Research

Automated image analysis has received increasing attention in social scientific research, yet existing scholarship has mostly covered the application of supervised learning to classify images into predefined categories. This study focuses on the task of unsupervised image clustering, which aims to automatically discover categories from unlabelled image data. We first review the steps to perform image clustering and then focus on one key challenge in this task—finding intermediate representations of images. We present several methods of extracting intermediate image representations, including the bag-of-visual-words model, self-supervised learning, and transfer learning (in particular, feature extraction with pretrained models). We compare these methods using various visual datasets, including images related to protests in China from Weibo, images about climate change on Instagram, and profile images of the Russian Internet Research Agency on Twitter. In addition, we propose a systematic way to interpret and validate clustering solutions. Results show that transfer learning significantly outperforms the other methods. The dataset used in the pretrained model critically determines what categories the algorithms can discover.

Improving unsupervised image clustering with spatial consistency

Unsupervised image clustering (UIC) is regularly employed to group images without manual annotation. One significant problem that occurs in the UIC context is that the visual-feature similarity across different semantic classes tends to introduce instance-dependent errors to clustering. The most successful recent approaches aimed at resolving this problem have focused on semisupervised reclustering, which utilizes reliable samples selected from the existing clusters. Despite this, virtually no previous work has considered the spatial consistency of the instance- and class-level representations which is crucial for error disambiguation. This makes it difficult to assess whether the selected reliable sample is reasonable. Accordingly, we propose a spatial consistency-based clustering (SCC) method to retain the alignment of representations learned from the instance and class levels in order to effectively select reliable samples from pre-existing clusters with errors thus leading to better clustering performance. More specifically, we first learn instance- and class-level representations by encouraging the semantic invariants of different instance augmentations and enforcing class alignment across semantically similar instances, respectively. We then assign instances with similar class-level representations to the same cluster to obtain the preliminary clusters. Subsequently we assess sample reliability by utilizing the spatial consistency constraint, which diffuses the class-level representations within the instance-level representation space. Finally, we employ semisupervised baselines combined with refinement techniques to perform reclustering based on the selected reliable samples. Extensive experimental results demonstrate that SCC can effectively obtain credible samples and outperform current SOTA clustering methods on the CIFAR-10 and CIFAR-100-20 benchmarks. The relevant code is available at https://github.com/RyanZhaoIc/SCC.git.

End-to-end robust joint unsupervised image alignment and clustering.

Computing dense pixel-to-pixel image correspondences is a fundamental task of computer vision. Often, the objective is to align image pairs from the same semantic category for manipulation or segmentation purposes. Despite achieving superior performance, existing deep learning alignment methods cannot cluster images; consequently, clustering and pairing images needed to be a separate laborious and expensive step. Given a dataset with diverse semantic categories, we propose a multi-task model, Jim-Net, that can directly learn to cluster and align images without any pixel-level or image-level annotations. We design a pair-matching alignment unsupervised training algorithm that selectively matches and aligns image pairs from the clustering branch. Our unsupervised Jim-Net achieves comparable accuracy with state-of-the-art supervised methods on benchmark 2D image alignment dataset PF-PASCAL. Specifically, we apply Jim-Net to cryo-electron tomography, a revolutionary 3D microscopy imaging technique of native subcellular structures. After extensive evaluation on seven datasets, we demonstrate that Jim-Net enables systematic discovery and recovery of representative macromolecular structures in situ, which is essential for revealing molecular mechanisms underlying cellular functions. To our knowledge, Jim-Net is the first end-to-end model that can simultaneously align and cluster images, which significantly improves the performance as compared to performing each task alone.

Joint DBN and Fuzzy C-Means unsupervised deep clustering for lung cancer patient stratification

Patient stratification has made a great contribution to efficient and personalized medicine. An important task in patient stratification is to discover quite distinct disease subtypes for effective treatment. In this paper, we propose a new deep learning and clustering model which combines Deep Belief Network (DBN) and Fuzzy C-Means(FCM), called Unsupervised Deep Fuzzy C-Means clustering Network(UDFCMN), to cluster lung cancer patients from lung CT images. In our deep clustering network, images after preprocessing are first encoded into multiple layers of hidden variables to extract hierarchical features and feature distribution and form the high-level representations. Here, to solve the problem of feature homogenization in DBN, we introduce the Winner-Take-All (WTA) idea to meliorate the traditional DBN structure, called WTADBN. Then FCM is used to produce the initial cluster labels with the new representations learnt by stacked WTARBM. Therefore, the FCM-generated cluster labels are used for the fine-tuning of the DBN as ground-truth labels. And an unsupervised image clustering and patient stratification process is completed by cross iteration. We tested our deep FCM clustering algorithm to do experiment on both public dataset from the internet and private dataset from cooperate hospital. For the latter one, the clinical and biological verification was also performed. Experimental results reveal outperformance of UDFCMN as compared to the state-of-the-art unsupervised classification methods. These results also indicate that our approach may have practical applications in lung cancer pathogenesis studies and provide useful guidelines for personalized cancer therapy.

DIC: Deep Image Clustering for Unsupervised Image Segmentation

Unsupervised segmentation is an essential pre-processing technique in many computer vision tasks. However, current unsupervised segmentation techniques are sensitive to the parameters such as the segmentation numbers or of high training and inference complexity. Encouraged by neural networks’ flexibility and their ability for modelling intricate patterns, an unsupervised segmentation framework based on a novel deep image clustering (DIC) model is proposed. The DIC consists of a feature transformation subnetwork (FTS) and a trainable deep clustering subnetwork (DCS) for unsupervised image clustering. FTS is built on a simple and capable network architecture. DCS can assign pixels with different cluster numbers by updating cluster associations and cluster centers iteratively. Moreover, a superpixel guided iterative refinement loss is designed to optimize the DIC parameters in an overfitting manner. Extensive experiments have been conducted on the Berkley Segmentation Database. The experimental results show that DCS is more effective in aggregating features during the clustering procedure. DIC has also proven to be less sensitive to varying segmentation parameters and of lower computation costs, and DIC can achieve significantly better segmentation performance compared to the state-of-the-art techniques. The source code is available on https://github.com/zmbhou/DIC .

Developing land-cover driver model for estimating the intensity of surface urban heat islands using landsat 8 satellite imagery

It is of utmost importance to understand and monitor the impact of urban heat islands on ecosystems and overall human health in the context of climate change and global warming. This research was conducted in a tropical city, Hanoi, with a major objective of assessing the quantitative relationships between the composition of the main land-cover types and surface urban heat island phenomenon. In this research, we analyzed the correlation between land-cover composition, percentage coverage of the land cover types, and land surface temperature for different moving window sizes or urban land management units. Landsat 8 OLI (Operational Land Imager) satellite data was utilized for preparing land-cover composition datasets in inner Hanoi by employing the unsupervised image clustering method. High-resolution (30m) land surface temperature maps were generated for different days of the years 2016 and 2017 using Landsat 8 TIRS (Thermal Infrared Sensor) images. High correlations were observed between percentage coverage of the land-cover types and land surface temperature considering different window sizes. A new model for estimating the intensity of surface urban heat islands from Landsat 8 imagery is developed, through recursively analyzing the correlation between land-cover composition and land surface temperature at different moving window sizes. This land-cover composition-driven model could predict land surface temperature efficiently not only in the case of different window sizes but also on different days. The newly developed model in this research provides a wonderful opportunity for urban planners and designers to take measures for adjusting land surface temperature and the associated effects of surface urban heat islands by managing the land cover composition and percentage coverage of the individual land-cover types.

A comparative study of unsupervised image clustering systems

The purpose of clustering algorithms is to give sense and extract value from large sets of structured and unstructured data. Thus, clustering is present in all science areas that use automatic learning. Therefore, we present in this paper a comparative study and an evaluation of different clustering methods proposed in the literature such as prototype based clustering, fuzzy and probabilistic clustering, hierarchical clustering and density based clustering. We present also an analysis of advantages and disadvantages of these clustering methods based essentially on experimentation. Extensive experiments are conducted on three real-world high dimensional datasets to evaluate the potential and the effectiveness of seven well-known methods in terms of accuracy, purity and normalised mutual information.

ConvART: Improving Adaptive Resonance Theory for Unsupervised Image Clustering

While supervised learning techniques have become increasinglyadept at separating images into different classes, these techniquesrequire large amounts of labelled data which may not always beavailable. We propose a novel neuro-dynamic method for unsuper-vised image clustering by combining 2 biologically-motivated mod-els: Adaptive Resonance Theory (ART) and Convolutional Neu-ral Networks (CNN). ART networks are unsupervised clustering al-gorithms that have high stability in preserving learned informationwhile quickly learning new information. Meanwhile, a major prop-erty of CNNs is their translation and distortion invariance, whichhas led to their success in the domain of vision problems. Byembedding convolutional layers into an ART network, the usefulproperties of both networks can be leveraged to identify differentclusters within unlabelled image datasets and classify images intothese clusters. In exploratory experiments, we demonstrate thatthis method greatly increases the performance of unsupervisedART networks on a benchmark image dataset.

Unsupervized Image Clustering With SIFT-Based Soft-Matching Affinity Propagation

It is known that affinity propagation can perform exemplar-based unsupervised image clustering by taking as input similarities between pairs of images and producing a set of exemplars that best represent the images, and then assigning each nonexemplar image to its most appropriate exemplar. However, the clustering performance of affinity propagation is largely limited by the adopted similarity between any pair of images. As the scale invariant feature transform (SIFT) has been widely employed to extract image features, the nonmetric similarity between any pair of images was proposed by “hard” matching of SIFT features (e.g., counting the number of matching SIFT features). In this letter, we notice, however, that the decision of hard matching of SIFT features is binary, which is not necessary for deriving similarities. Hence, we propose a novel measure of similarities by replacing hard matching with the so-called soft matching. Experimental examples show that significant performance gains can be achieved by the resulting affinity propagation algorithm.