Embedding Clustering Research Articles

Automatic assignment of microgenres to movies using a word embedding-based approach

Streaming services are increasingly leveraging Artificial Intelligence (AI) technologies for improved content cataloging, user experiences in content discovery, and personalization. A significant challenge in this domain is the automated assignment of microgenres to movies. This study introduces and evaluates approaches based on clustering, topic modeling, and word embedding to address this task. The evaluation employs a preprocessed dataset containing movie-related data—title tags, synopses, genres, and reviews—alongside a predefined microgenre list. Comparisons of three activation functions (binary step, ramp, and sigmoid) gauge their effectiveness in augmenting microgenre tags. Results demonstrate the superiority of the word embedding approach over clustering and topic modeling in terms of mean accuracy. Even more, the word embedding approach stands as the sole fully automated solution. Analysis indicates that incorporating review-based tags introduces noise and undermines accuracy. Besides, the word embedding approach yields optimal outcomes using the sigmoid function, effectively doubling assigned tags while maintaining matching quality. This sheds light on the potential of word embedding methods within the movie domain.

Neural Network Approaches for Recommender Systems

Recommender systems are special algorithms that allow users to receive personalized recommendations on topics that interest them. Systems of this kind are widely used in various fields, for example, in e-commerce, provider services, social networks, etc. Together with classical approaches, neural networks have also become popular in recommender systems in recent years, which are gradually replacing traditional methods of collaborative filtering and content-based algorithms. However, neural networks require large computing resources, which often raises questions on whether an increase in quality will be justified and whether there be one at all. The neural network approach in recommender systems—the self-attentive sequential recommendation (SASRec) transformer model from Microsoft Recommenders—is studied and compared with the classic algorithm, the LightFM hybrid model. For training and validation, the data taken from a housing search application are used. It is proposed to use the hit rate as the main metric for comparison. The results of the experiments will help to understand which algorithms have higher accuracy in terms of predictions and recommendations. As an additional part, the clustering of user and object embeddings is considered.

A Comparative Study of Perceptions of Destination Image Based on Content Mining: Fengjing Ancient Town and Zhaojialou Ancient Town as Examples

Ancient canal towns in Jiangnan have become important tourist destinations due to their unique water town scenery and historical value. Creating a unique tourist image boosts these ancient towns’ competitive edge in tourism and contributes significantly to their preservation and growth. The vast amount of data from social media has become an essential source for uncovering tourism perceptions. This study takes two ancient towns in Shanghai, Zhaojialou and Fengjing, as case study areas. In order to explore and compare the destination images of the towns, in the perception of tourists and in official publicity, machine learning approaches like word embedding and K-means clustering are adopted to process the comments on Sina Weibo and publicity articles, and statistical analysis and correspondence analysis are used for comparative study. The results reveal the following: (1) Using k-means clustering, destination perceptions were categorized into 16 groups spanning three dimensions, “space, activity, and sentiment”, with the most keywords in “activity” and the fewest in “sentiment”. (2) The perception of tourists often differs significantly from the official promotional materials. Official promotions place a strong emphasis on shaping the image of ancient towns based on their historical resources, presenting a more general picture. Tourist perception, which is fragmented, highlights emerging elements and the experiential activities, along with the corresponding emotional experiences. (3) Comparing the two towns, Fengjing Ancient Town stands out, with more diverse tourist perceptions and richer emotional experiences. This underscores the effectiveness of tourism activities that use space as a media to evoke emotions, surpassing the impact of the spaces themselves.

Graph attention-based deep embedded clustering for speaker diarization

Deep speaker embedding extraction models have recently served as the cornerstone for modular speaker diarization systems. However, in current modular systems, the extracted speaker embeddings (namely, speaker features) do not effectively leverage their intrinsic relationships, and moreover, are not tailored specifically for the clustering task. In this paper, inspired by deep embedded clustering (DEC), we propose a speaker diarization method using the graph attention-based deep embedded clustering (GADEC) to address the aforementioned issues. First, considering the temporal nature of speech signals, when segmenting the speech signal into small segments, the speech in the current segment and its neighboring segments may likely belong to the same speaker. This suggests that embeddings extracted from neighboring segments could help generate a more informative speaker representation for the current segment. To better describe the complex relationships between segments and leverage the local structural information among their embeddings, we construct a graph for the pre-extracted speaker embeddings in a continuous audio signal. On this basis, we introduce a graph attentional encoder (GAE) module to integrate information from neighboring nodes (i.e., neighboring segments) in the graph and learn latent speaker embeddings. Moreover, we further jointly optimize both the latent speaker embeddings and the clustering results within a unified framework, leading to more discriminative speaker embeddings for the clustering task. Experimental results demonstrate that our proposed GADEC-based speaker diarization system significantly outperforms the baseline systems and several other recent speaker diarization systems concerning diarization error rate (DER) on the NIST SRE 2000 CALLHOME, AMI, and VoxConverse datasets.

Formulating the potentials of clustering of event data over multiple entities for decision support: a network embeddings approach

ABSTRACT Event data from business processes evidence their patterns, behaviors, and dysfunctions. Analytics techniques like clustering and sorting can reveal relevant insights, when data are correlated with a single case identifier. However, when multiple entities are involved, unidimensional models are challenged. We introduce a novel method for analyzing business processes involving multiple interacting entity types. Our approach employs embedding representations to capture pairwise similarities among entity types and their interrelationships. An optimization problem encompasses similarity matrices, cross-entity relationship matrices, and embeddings. An iterative algorithm refines this model, yielding embedding representations and cluster assignments for each entity type. Formulating our method across three diverse business scenarios demonstrates its practicality and potential. Our results, through a proof of concept using real-world data, underscore the value of accounting for the multifaceted nature of business processes, showing substantial improvements and qualitative distinctions compared to unidimensional models.

Enhancing cluster analysis via topological manifold learning

AbstractWe discuss topological aspects of cluster analysis and show that inferring the topological structure of a dataset before clustering it can considerably enhance cluster detection: we show that clustering embedding vectors representing the inherent structure of a dataset instead of the observed feature vectors themselves is highly beneficial. To demonstrate, we combine manifold learning method UMAP for inferring the topological structure with density-based clustering method DBSCAN. Synthetic and real data results show that this both simplifies and improves clustering in a diverse set of low- and high-dimensional problems including clusters of varying density and/or entangled shapes. Our approach simplifies clustering because topological pre-processing consistently reduces parameter sensitivity of DBSCAN. Clustering the resulting embeddings with DBSCAN can then even outperform complex methods such as SPECTACL and ClusterGAN. Finally, our investigation suggests that the crucial issue in clustering does not appear to be the nominal dimension of the data or how many irrelevant features it contains, but rather how separable the clusters are in the ambient observation space they are embedded in, which is usually the (high-dimensional) Euclidean space defined by the features of the data. The approach is successful because it performs the cluster analysis after projecting the data into a more suitable space that is optimized for separability, in some sense.

Elastic deep autoencoder for text embedding clustering by an improved graph regularization

Text clustering is a task for grouping extracted information of the text in different clusters, which has many applications in recommender systems, sentiment analysis, and more. Deep learning-based methods have become increasingly popular due to their high accuracy in identifying nonlinear structures. They usually consist of two major parts: dimensionality reduction and clustering. Autoencoders are simple unsupervised neural networks used for better representation of low-dimensional data and have shown good performance in dealing with non-linear features. However, while they utilize the Frobenius norm to deal well with Gaussian noise, they are sensitive to outlier data and Laplacian noise. In this paper, a deep autoencoder with an adapted elastic loss for text embedding clustering (EDA-TEC) is proposed. The elastic loss is a combination of the Frobenius norm and L2,1-norm to consider both types of noises. Additionally, to maintain the high-dimensional data geometric structure, a modified graph regularization term based on the weighted cosine similarity measure is used. EDA-TEC also improves clustering results by considering the sparsity regularization of the manifold representation data. In this jointly end-to-end deep learning model, better representation and text clustering results are achieved with high accuracy on common datasets compared to existing methods.11https://github.com/safinal/text-embedding-clustering.

Health diagnosis of concrete dams with continuous missing data for assessing structural deformation based on tSNE–AHC algorithm and deep transfer learning

To address the issue of health diagnosis for concrete dams with continuous missing data, this study aimed to develop and validate a health diagnosis model based on domain learning. Initially, the target measuring point is precisely matched with a source measuring point via the tSNE–AHC algorithm (by combining t-distributed stochastic neighbor embedding and agglomerative hierarchical clustering). Subsequently, TR-CNN-LSTM models based on convolutional neural networks, long and short-term memory and transfer learning are built to transfer the knowledge of dam deformation learned from the source measurement points to the target measurement point. Finally, the performance is quantified under different degrees of missing data. The case study revealed that the established model offers optimal prediction accuracy and robustness, which conceptualizes a new approach to structural health monitoring.

Ultra-Short-Term Load Forecasting for Customer-Level Integrated Energy Systems Based on Composite VTDS Models

A method is proposed to address the challenging issue of load prediction in user-level integrated energy systems (IESs) using a composite VTDS model. Firstly, an IES multi-dimensional load time series is decomposed into multiple intrinsic mode functions (IMFs) using variational mode decomposition (VMD). Then, each IMF, along with other influential features, is subjected to data dimensionality reduction and clustering denoising using t-distributed stochastic neighbor embedding (t-SNE) and fast density-based spatial clustering of applications with noise (FDBSCAN) to perform major feature selection. Subsequently, the reduced and denoised data are reconstructed, and a time-aware long short-term memory (T-LSTM) artificial neural network is employed to fill in missing data by incorporating time interval information. Finally, the selected multi-factor load time series is used as input into a support vector regression (SVR) model optimized using the quantum particle swarm optimization (QPSO) algorithm for load prediction. Using measured load data from a specific user-level IES at the Tempe campus of Arizona State University, USA, as a case study, a comparative analysis between the VTDS method and other approaches is conducted. The results demonstrate that the method proposed in this study achieved higher accuracy in short-term forecasting of the IES’s multiple loads.

Discovering Causes of Traffic Congestion via Deep Transfer Clustering

Traffic congestion incurs long delay in travel time, which seriously affects our daily travel experiences. Exploring why traffic congestion occurs is significantly important to effectively address the problem of traffic congestion and improve user experience. Traditional approaches to mine the congestion causes depend on human efforts, which is time consuming and cost-intensive. Hence, we aim at discovering the known and unknown causes of traffic congestion in a systematic way. However, to achieve it, there are three challenges: (1) traffic congestion is affected by several factors with complex spatio-temporal relations; (2) there are a few samples of congestion data with known causes due to the limitation of human label; (3) more unknown congestion causes are unexplored since several factors contribute to traffic congestion. To address above challenges, we design a congestion cause discovery system consisting of two modules: (1) congestion feature extraction module, which extracts the important features distinguishing between different causes of congestion; and (2) congestion cause discovery module, which designs a deep semi-supervised learning based framework to discover the causes of traffic congestion with limited labeled data. Specifically, in pre-training stage, it first leverages a few labeled data as prior knowledge to pre-train the model. Then, in clustering stage, we propose two different clustering methods to discover the congestion causes. For the first clustering method, we extend the classic deep embedded clustering model to produce clusters via soft assignment. For the second one, we iteratively use k -means to group the latent features extracted from the pre-trained model, and use the cluster results as pseudo-labels to fine-tune the network. Extensive experiments show that the performance of our methods is superior to the state-of-the-art baselines, which demonstrates the effectiveness of the proposed cause discovery system. Additionally, our system is deployed and used in the practical production environment at Amap.

GPENs: Graph Data Learning With Graph Propagation-Embedding Networks.

Compact representation of graph data is a fundamental problem in pattern recognition and machine learning area. Recently, graph neural networks (GNNs) have been widely studied for graph-structured data representation and learning tasks, such as graph semi-supervised learning, clustering, and low-dimensional embedding. In this article, we present graph propagation-embedding networks (GPENs), a new model for graph-structured data representation and learning problem. GPENs are mainly motivated by 1) revisiting of traditional graph propagation techniques for graph node context-aware feature representation and 2) recent studies on deeply graph embedding and neural network architecture. GPENs integrate both feature propagation on graph and low-dimensional embedding simultaneously into a unified network using a novel propagation-embedding architecture. GPENs have two main advantages. First, GPENs can be well-motivated and explained from feature propagation and deeply learning architecture. Second, the equilibrium representation of the propagation-embedding operation in GPENs has both exact and approximate formulations, both of which have simple closed-form solutions. This guarantees the compactivity and efficiency of GPENs. Third, GPENs can be naturally extended to multiple GPENs (M-GPENs) to address the data with multiple graph structures. Experiments on various semi-supervised learning tasks on several benchmark datasets demonstrate the effectiveness and benefits of the proposed GPENs and M-GPENs.

Two-Dimensional Embedded Fuzzy Data Clustering

Fuzzy <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"><tex-math notation="LaTeX">$k$</tex-math></inline-formula> -means (FKM) is a popular clustering method by assigning data points into respective clusters with uncertainty measured by the membership degree. Usually, FKM performs clustering according to the distance between data points in the original space, which might contain undesirable noises and redundant features; therefore, the underlying data semantic connections cannot be accurately captured. Moreover, the vectorized representation of the two-dimensional data such as image inevitably leads to the loss of structural information. In this paper, we propose a novel FKM method termed two-dimensional embedded fuzzy data clustering (2DEFC) which has two merits. First, 2DEFC directly takes 2D data as input without vectorizing them in order to retain more structural information of data. Second, the two subspace projection matrices are jointly optimized with the data membership degree for better collaborating with each other, which effectively avoids the sub-optimality limitation caused by the conventional mode of sequentially performing dimensionality reduction and clustering. An efficient algorithm is proposed to optimize the 2DEFC objective function. Besides, we provide comprehensive analysis on 2DEFC including its convergence behavior, computational complexity, and the fuzzy weighting exponent. Extensive comparative studies on benchmark 2D data sets demonstrate that the competitive performance of 2DEFC in 2D data clustering.

The LSST AGN Data Challenge: Selection Methods

Development of the Rubin Observatory Legacy Survey of Space and Time (LSST) includes a series of Data Challenges (DCs) arranged by various LSST Scientific Collaborations that are taking place during the project's preoperational phase. The AGN Science Collaboration Data Challenge (AGNSC-DC) is a partial prototype of the expected LSST data on active galactic nuclei (AGNs), aimed at validating machine learning approaches for AGN selection and characterization in large surveys like LSST. The AGNSC-DC took place in 2021, focusing on accuracy, robustness, and scalability. The training and the blinded data sets were constructed to mimic the future LSST release catalogs using the data from the Sloan Digital Sky Survey Stripe 82 region and the XMM-Newton Large Scale Structure Survey region. Data features were divided into astrometry, photometry, color, morphology, redshift, and class label with the addition of variability features and images. We present the results of four submitted solutions to DCs using both classical and machine learning methods. We systematically test the performance of supervised models (support vector machine, random forest, extreme gradient boosting, artificial neural network, convolutional neural network) and unsupervised ones (deep embedding clustering) when applied to the problem of classifying/clustering sources as stars, galaxies, or AGNs. We obtained classification accuracy of 97.5% for supervised models and clustering accuracy of 96.0% for unsupervised ones and 95.0% with a classic approach for a blinded data set. We find that variability features significantly improve the accuracy of the trained models, and correlation analysis among different bands enables a fast and inexpensive first-order selection of quasar candidates.

A unified density matrix functional construction of quantum baths in density matrix embedding theory beyond the mean-field approximation.

The equivalence in one-electron quantum baths between the practical implementation of density matrix embedding theory (DMET) and the more recent Householder-transformed density matrix functional embedding theory has been shown previously in the standard but special case where the reference full-size (one-electron reduced) density matrix, from which the bath is constructed, is idempotent [S. Yalouz et al., J. Chem. Phys. 157, 214112 (2022)]. We prove mathematically that the equivalence remains valid when the density matrix is not idempotent anymore, thus allowing for the construction of correlated (one-electron) quantum baths. A density-matrix functional exactification of DMET is derived within the present unified quantum embedding formalism. Numerical examples reveal that the embedding cluster can be quite sensitive to the level of density-matrix functional approximation used for computing the reference density matrix.

Identifying typical pre-crash scenarios based on in-depth crash data with deep embedded clustering for autonomous vehicle safety testing

Choosing appropriate scenarios is critical for autonomous vehicles (AVs) safety testing. Real-world crash scenarios can be used as critical scenarios to test the safety performance of AVs. As one of the dominant types of traffic crashes, the car to powered-two-wheelers (PTWs) crash results in a higher possibility of fatality than ordinary car-to-car crashes. Generally, typical testing scenarios are chosen according to the subjective understanding of the safety experts with limited static features of crashes (e.g., geometric features, weather). This study introduced a novel method to identify typical car-to-PTWs crash scenarios based on real-world crashes with dynamic pre-crash features investigated from the China In-depth Mobility Safety Study-Traffic Accident (CIMSS-TA) database. First, we present crash data collection and construction methods of the CIMSS-TA database to construct testing scenarios. Second, the stacked autoencoder methods are used to learn and obtain embedded features from the high-dimensional data. Third, the extracted features are clustered using k-means clustering algorithm, and then the clustering results are interpreted. Six typical car-to-PTWs scenarios are obtained with the proposed processes. This study introduces a typical high-risk scenario construction method based on deep embedded clustering. Unlike existing researches, the proposed method eliminates the negative impacts of manually selecting clustering variables and provides a more detailed scenario description. As a result, the typical scenarios obtained from AV testing are more robust.

RDEIC-LFW-DSS: ResNet-based deep embedded image clustering using local feature weighting and dynamic sample selection mechanism

In existing deep clustering methods, as the model gets deeper, extracted representations can be deteriorated due to a vanishing gradient, leading to reduced performance. Also, existing deep clustering frameworks may distort the hidden space without true labels by learning from unreliable estimated pseudo-labels. Therefore, the model learns non-discriminative features from the early epochs, which results in worse estimated pseudo-labels. Moreover, in these models, it is assumed that all learned representations in the bottleneck layer have equal significance throughout the clustering procedure.To address the shortcomings, this paper introduces RDEIC-LFW-DSS, a ResNet-based deep embedded clustering framework. The advantage of residual connection is that it enables the user to add residual connections for increased model capacity without incurring the cost of degradation for unsupervised feature learning compared to standard autoencoders. Furthermore, an efficient local feature weighting mechanism is used to weight each cluster's representations correctly. In addition, we introduce a mechanism in which suitable samples with highly reliable estimated labels are selected to train the deep clustering framework. Extensive experiments conducted on benchmark datasets and comparisons with state-of-the-art approaches confirm the high performance of RDEIC-LFW-DSS. The implementation- source code- of RDEIC-LFW-DSS is made publicly accessible at https://github.com/Amin-Golzari-Oskouei/RDEIC-LFW-DSS.

Guided deep embedded clustering regularization for multifeature medical signal classification

Medical signal classification often focuses on one representation (raw signal or time frequency). In that context, recent works have shown the value of exploiting different representations simultaneously. We propose a regularized end-to-end trained model for classification in a medical context exploiting both the raw signal and a time-frequency representation (TFR). First, a 2D convolutional neural network (CNN) encoder and a 1D CNN-transformer encoder start by extracting embedded representations from the TFR and the raw signal, respectively. Then, the obtained embeddings are fused to form a common latent space that is used for classification. We propose to guide the training of each encoder by applying two iterated losses. Moreover, we propose to regularize the fused common latent space using deep embedded clustering. Extensive experiments on three medical datasets and ablation studies show the adaptability and good performance of our method for medical signal classification. Our method makes it possible to improve the classification performance from 4% to 12% MCC on a transcranial Doppler dataset, when compared with single-feature counterparts, while giving more stable models. The code is available at: https://github.com/gdec-submission/gdec/.

Identifying low-performing regions in Moroccan education: A deep learning approach using the PISA dataset

This study highlights the ongoing nature of the school reform movement, emphasizing the need for continuous attention and action. Despite this effort, academic performance has exhibited relative stability in recent years, while significant regional performance disparities persist. Addressing these inequalities requires novel approaches to enhance educational quality. Past research has explored clustering algorithms in developed countries, providing insights into personalized teaching strategies based on students' learning style preferences. In response, our research aims to identify underperforming regions in Morocco, necessitating attention and intervention. We employ an unsupervised deep learning method called "deep embedding clustering" to group Moroccan students based on their performance. The results are subsequently visualized on a choropleth map, revealing intricate patterns and trends in educational performance that might not be immediately apparent. The analysis employs the comprehensive program for international student assessment (PISA) dataset, encompassing individual students' responses and plausible values reflecting cognitive abilities. The findings indicate that the "Guelmim-Oued Noun" region exhibits the highest performance level among all regions, while "Dakhla-Oued Eddahab," "Béni Mellal-Khénifra," and "Oriental" regions display lower performance levels. As a result, this study urges policymakers to incorporate tailored measures into regional policies to improve students' educational outcomes.

Exploring the Interaction between Local and Global Latent Configurations for Clustering Single-Cell RNA-Seq: A Unified Perspective

The most recent approaches for clustering single-cell RNA-sequencing data rely on deep auto-encoders. However, three major challenges remain unaddressed. First, current models overlook the impact of the cumulative errors induced by the pseudo-supervised embedding clustering task (Feature Randomness). Second, existing methods neglect the effect of the strong competition between embedding clustering and reconstruction (Feature Drift). Third, the previous deep clustering models regularly fail to consider the topological information of the latent data, even though the local and global latent configurations can bring complementary views to the clustering task. To address these challenges, we propose a novel approach that explores the interaction between local and global latent configurations to progressively adjust the reconstruction and embedding clustering tasks. We elaborate a topological and probabilistic filter to mitigate Feature Randomness and a cell-cell graph structure and content correction mechanism to counteract Feature Drift. The Zero-Inflated Negative Binomial model is also integrated to capture the characteristics of gene expression profiles. We conduct detailed experiments on real-world datasets from multiple representative genome sequencing platforms. Our approach outperforms the state-of-the-art clustering methods in various evaluation metrics.

Self-Supervised Graph Attention Networks for Deep Weighted Multi-View Clustering

As one of the most important research topics in the unsupervised learning field, Multi-View Clustering (MVC) has been widely studied in the past decade and numerous MVC methods have been developed. Among these methods, the recently emerged Graph Neural Networks (GNN) shine a light on modeling both topological structure and node attributes in the form of graphs, to guide unified embedding learning and clustering. However, the effectiveness of existing GNN-based MVC methods is still limited due to the insufficient consideration in utilizing the self-supervised information and graph information, which can be reflected from the following two aspects: 1) most of these models merely use the self-supervised information to guide the feature learning and fail to realize that such information can be also applied in graph learning and sample weighting; 2) the usage of graph information is generally limited to the feature aggregation in these models, yet it also provides valuable evidence in detecting noisy samples. To this end, in this paper we propose Self-Supervised Graph Attention Networks for Deep Weighted Multi-View Clustering (SGDMC), which promotes the performance of GNN-based deep MVC models by making full use of the self-supervised information and graph information. Specifically, a novel attention-allocating approach that considers both the similarity of node attributes and the self-supervised information is developed to comprehensively evaluate the relevance among different nodes. Meanwhile, to alleviate the negative impact caused by noisy samples and the discrepancy of cluster structures, we further design a sample-weighting strategy based on the attention graph as well as the discrepancy between the global pseudo-labels and the local cluster assignment. Experimental results on multiple real-world datasets demonstrate the effectiveness of our method over existing approaches.