Cluster Quality Metrics Research Articles

Robust Parameter Optimisation of Noise-Tolerant Clustering for DENCLUE Using Differential Evolution

Clustering samples based on similarity remains a significant challenge, especially when the goal is to accurately capture the underlying data clusters of complex arbitrary shapes. Existing density-based clustering techniques are known to be best suited for capturing arbitrarily shaped clusters. However, a key limitation of these methods is the difficulty in automatically finding the optimal set of parameters adapted to dataset characteristics, which becomes even more challenging when the data contain inherent noise. In our recent work, we proposed a Differential Evolution-based DENsity CLUstEring (DE-DENCLUE) to optimise DENCLUE parameters. This study evaluates DE-DENCLUE for its robustness in finding accurate clusters in the presence of noise in the data. DE-DENCLUE performance is compared against three other density-based clustering algorithms—DPC based on weighted local density sequence and nearest neighbour assignment (DPCSA), Density-Based Spatial Clustering of Applications with Noise (DBSCAN), and Variable Kernel Density Estimation–based DENCLUE (VDENCLUE)—across several datasets (i.e., synthetic and real). The study has consistently shown superior results for DE-DENCLUE compared to other models for most datasets with different noise levels. Clustering quality metrics such as the Silhouette Index (SI), Davies–Bouldin Index (DBI), Adjusted Rand Index (ARI), and Adjusted Mutual Information (AMI) consistently show superior SI, ARI, and AMI values across most datasets at different noise levels. However, in some cases regarding DBI, the DPCSA performed better. In conclusion, the proposed method offers a reliable and noise-resilient clustering solution for complex datasets.

Deep multimodal saliency parcellation of cerebellar pathways: Linking microstructure and individual function through explainable multitask learning.

Parcellation of human cerebellar pathways is essential for advancing our understanding of the human brain. Existing diffusion magnetic resonance imaging tractography parcellation methods have been successful in defining major cerebellar fibre tracts, while relying solely on fibre tract structure. However, each fibre tract may relay information related to multiple cognitive and motor functions of the cerebellum. Hence, it may be beneficial for parcellation to consider the potential importance of the fibre tracts for individual motor and cognitive functional performance measures. In this work, we propose a multimodal data-driven method for cerebellar pathway parcellation, which incorporates both measures of microstructure and connectivity, and measures of individual functional performance. Our method involves first training a multitask deep network to predict various cognitive and motor measures from a set of fibre tract structural features. The importance of each structural feature for predicting each functional measure is then computed, resulting in a set of structure-function saliency values that are clustered to parcellate cerebellar pathways. We refer to our method as Deep Multimodal Saliency Parcellation (DeepMSP), as it computes the saliency of structural measures for predicting cognitive and motor functional performance, with these saliencies being applied to the task of parcellation. Applying DeepMSP to a large-scale dataset from the Human Connectome Project Young Adult study (n = 1065), we found that it was feasible to identify multiple cerebellar pathway parcels with unique structure-function saliency patterns that were stable across training folds. We thoroughly experimented with all stages of the DeepMSP pipeline, including network selection, structure-function saliency representation, clustering algorithm, and cluster count. We found that a 1D convolutional neural network architecture and a transformer network architecture both performed comparably for the multitask prediction of endurance, strength, reading decoding, and vocabulary comprehension, with both architectures outperforming a fully connected network architecture. Quantitative experiments demonstrated that a proposed low-dimensional saliency representation with an explicit measure of motor versus cognitive category bias achieved the best parcellation results, while a parcel count of four was most successful according to standard cluster quality metrics. Our results suggested that motor and cognitive saliencies are distributed across the cerebellar white matter pathways. Inspection of the final k = 4 parcellation revealed that the highest-saliency parcel was most salient for the prediction of both motor and cognitive performance scores and included parts of the middle and superior cerebellar peduncles. Our proposed saliency-based parcellation framework, DeepMSP, enables multimodal, data-driven tractography parcellation. Through utilising both structural features and functional performance measures, this parcellation strategy may have the potential to enhance the study of structure-function relationships of the cerebellar pathways.

Optimizing Patient Stratification in Healthcare: A Comparative Analysis of Clustering Algorithms for EHR Data

Advanced data analytics are increasingly being employed in healthcare research to improve patient classification and personalize medicinal therapies. In this paper, we focus on the critical problem of clustering electronic health record (EHR) data to enable appropriate patient categorization. In the era of personalized medicine, optimizing patient classification is critical to healthcare analytics. This research presents a comparative assessment of different clustering algorithms for Electronic Health Record (EHR) data, with the goal of improving the efficacy and productivity of patient clustering methods. Our study focuses on Fuzzy Technique for Order of Preference by Similarity to Ideal Solution (Fuzzy TOPSIS) as a Multi-Criteria Decision-Making (MCDM) strategy, includes an in-depth assessment of eight clustering algorithms: K-Means, DBSCAN, Hierarchical Clustering, Mean Shift, Affinity Propagation, Spectral Clustering, Gaussian Mixture Models (GMM), as well as Self-Organizing Maps. The evaluation factors used for evaluation in this research are Cluster Quality Metrics, Scalability, Robustness to Noise, Cluster Shape and Density, Interpretability, Cluster Number, Dimensionality, and Consistency and Stability. These criteria and alternatives were chosen after conducting a thorough assessment of the literature and consulting with domain experts. All participated specialists actively engaged in the decision-making process, bringing unique insights into the best clustering algorithms for healthcare data. The results of this study illustrate each algorithm’s strengths and weaknesses in the setting of patient stratification, providing insight into their performance across multiple dimensions. The fuzzy TOPSIS MCDM strategy is a reliable instrument for synthesizing expert opinions and methodically evaluating the found clustering alternatives. This study advances healthcare analytics by giving practitioners and researchers with informative perspectives on the selection of clustering algorithms designed to address the unique problems of patient stratification utilizing EHR data.

Automated Trace Clustering Pipeline Synthesis in Process Mining

Business processes have undergone a significant transformation with the advent of the process-oriented view in organizations. The increasing complexity of business processes and the abundance of event data have driven the development and widespread adoption of process mining techniques. However, the size and noise of event logs pose challenges that require careful analysis. The inclusion of different sets of behaviors within the same business process further complicates data representation, highlighting the continued need for innovative solutions in the evolving field of process mining. Trace clustering is emerging as a solution to improve the interpretation of underlying business processes. Trace clustering offers benefits such as mitigating the impact of outliers, providing valuable insights, reducing data dimensionality, and serving as a preprocessing step in robust pipelines. However, designing an appropriate clustering pipeline can be challenging for non-experts due to the complexity of the process and the number of steps involved. For experts, it can be time-consuming and costly, requiring careful consideration of trade-offs. To address the challenge of pipeline creation, the paper proposes a genetic programming solution for trace clustering pipeline synthesis that optimizes a multi-objective function matching clustering and process quality metrics. The solution is applied to real event logs, and the results demonstrate improved performance in downstream tasks through the identification of sub-logs.

DGCLUSTER: A Neural Framework for Attributed Graph Clustering via Modularity Maximization

Graph clustering is a fundamental and challenging task in the field of graph mining where the objective is to group the nodes into clusters taking into consideration the topology of the graph. It has several applications in diverse domains spanning social network analysis, recommender systems, computer vision, and bioinformatics. In this work, we propose a novel method, DGCluster, which primarily optimizes the modularity objective using graph neural networks and scales linearly with the graph size. Our method does not require the number of clusters to be specified as a part of the input and can also leverage the availability of auxiliary node level information. We extensively test DGCluster on several real-world datasets of varying sizes, across multiple popular cluster quality metrics. Our approach consistently outperforms the state-of-the-art methods, demonstrating significant performance gains in almost all settings.

Identifying bias in network clustering quality metrics.

We study potential biases of popular network clustering quality metrics, such as those based on the dichotomy between internal and external connectivity. We propose a method that uses both stochastic and preferential attachment block models construction to generate networks with preset community structures, and Poisson or scale-free degree distribution, to which quality metrics will be applied. These models also allow us to generate multi-level structures of varying strength, which will show if metrics favour partitions into a larger or smaller number of clusters. Additionally, we propose another quality metric, the density ratio. We observed that most of the studied metrics tend to favour partitions into a smaller number of big clusters, even when their relative internal and external connectivity are the same. The metrics found to be less biased are modularity and density ratio.

SeQual: an unsupervised feature selection method for cloud workload traces

One challenge of studying cloud workload traces is the lack of available users’ identities. Therefore, clustering methods were used to address this challenge through extracting these identities from workload traces. For better extraction, it is beneficial to select attributes (columns in the traces) for clustering by using feature selection methods. However, the use of general selection methods requires details that are not available for workload traces (e.g. predefined number of clusters). Therefore, in this paper, we present an unsupervised feature selection method for cloud workload traces to identify good candidate attributes for clustering. This method uses Silhouette coefficients to rank attributes that are best for users’ extraction through clustering. The performance of our SeQual method is evaluated in comparison with commonly used (supervised and unsupervised) feature selection methods with the help of clustering quality metrics (i.e. adjusted rand index, entropy and precision). The results show that the SeQual method can compete with the supervised methods and perform better than unsupervised ones, with an average accuracy between 90% and 99%.

On the Best Way to Cluster NCI-60 Molecules

Machine learning-based models have been widely used in the early drug-design pipeline. To validate these models, cross-validation strategies have been employed, including those using clustering of molecules in terms of their chemical structures. However, the poor clustering of compounds will compromise such validation, especially on test molecules dissimilar to those in the training set. This study aims at finding the best way to cluster the molecules screened by the National Cancer Institute (NCI)-60 project by comparing hierarchical, Taylor-Butina, and uniform manifold approximation and projection (UMAP) clustering methods. The best-performing algorithm can then be used to generate clusters for model validation strategies. This study also aims at measuring the impact of removing outlier molecules prior to the clustering step. Clustering results are evaluated using three well-known clustering quality metrics. In addition, we compute an average similarity matrix to assess the quality of each cluster. The results show variation in clustering quality from method to method. The clusters obtained by the hierarchical and Taylor-Butina methods are more computationally expensive to use in cross-validation strategies, and both cluster the molecules poorly. In contrast, the UMAP method provides the best quality, and therefore we recommend it to analyze this highly valuable dataset.

Robust Representation and Efficient Feature Selection Allows for Effective Clustering of SARS-CoV-2 Variants

The widespread availability of large amounts of genomic data on the SARS-CoV-2 virus, as a result of the COVID-19 pandemic, has created an opportunity for researchers to analyze the disease at a level of detail, unlike any virus before it. On the one hand, this will help biologists, policymakers, and other authorities to make timely and appropriate decisions to control the spread of the coronavirus. On the other hand, such studies will help to more effectively deal with any possible future pandemic. Since the SARS-CoV-2 virus contains different variants, each of them having different mutations, performing any analysis on such data becomes a difficult task, given the size of the data. It is well known that much of the variation in the SARS-CoV-2 genome happens disproportionately in the spike region of the genome sequence—the relatively short region which codes for the spike protein(s). In this paper, we propose a robust feature-vector representation of biological sequences that, when combined with the appropriate feature selection method, allows different downstream clustering approaches to perform well on a variety of different measures. We use such proposed approach with an array of clustering techniques to cluster spike protein sequences in order to study the behavior of different known variants that are increasing at a very high rate throughout the world. We use a k-mers based approach first to generate a fixed-length feature vector representation of the spike sequences. We then show that we can efficiently and effectively cluster the spike sequences based on the different variants with the appropriate feature selection. Using a publicly available set of SARS-CoV-2 spike sequences, we perform clustering of these sequences using both hard and soft clustering methods and show that, with our feature selection methods, we can achieve higher F1 scores for the clusters and also better clustering quality metrics compared to baselines.

An elitism based self-adaptive multi-population Poor and Rich optimization algorithm for grouping similar documents

In this digital era, grouping similar documents from the archives on the web is a difficult and computationally expensive task. In this paper, we propose an elitism based self-adaptive multi-population Poor and Rich optimization algorithm for grouping the similar documents, referred to as ESAMPRO. The objective function of the proposed work maximizes the accuracy and minimize the intra cluster distance. The proposed algorithm is evaluated using the various extrinsic cluster quality metrics. An in-depth analysis of the experimental results on four supervised benchmark datasets confirms that the proposed ESAMPRO algorithm outperformed the five well-known document clustering algorithms such as K-means, particle swarm optimization, whale optimization, dragonfly and grey wolf optimization algorithm.

Analyzing Constellation Performance for the Radio Occultation Tomography of Internal Gravity Waves

Radio occultation (RO) is a powerful tool for remotely sensing the atmosphere, producing globally distributed soundings with both high vertical resolution and high temperature retrieval accuracy, especially in the stratosphere. The spatial distribution of the soundings typically prevents the use of these measurements for studying atmospheric effects with small horizontal or temporal scales. However, careful arrangement of a dedicated RO constellation can yield sounding clusters useful for the tomographic reconstruction of internal gravity waves with horizontal wavelengths in the tens of kilometers. This article presents occultation cluster quality metrics predictive of internal gravity wave tomographic reconstruction error and uses these metrics to compare the performance of two alternative RO constellation geometries: mutual orbit groups (MOG) and a spread in right ascension of the ascending node (RAAN). MOG constellations have better overall performance and yield more consistent cluster quality across all sampled latitudes, while RAAN-spread constellations have improved equatorial quality and a trend toward reduced quality at the edges of the latitude range. Additionally, analysis of clusters by latitude, ray azimuth, and quality is performed in order to examine the trends in the outlier best- and worst-performing clusters for each constellation type.

Clustering quality metrics for subspace clustering

We study the problem of clustering validation, i.e., clustering evaluation without knowledge of ground-truth labels, for the increasingly-popular framework known as subspace clustering. Existing clustering quality metrics (CQMs) rely heavily on a notion of distance between points, but common metrics fail to capture the geometry of subspace clustering. We propose a novel point-to-point pseudometric for points lying on a union of subspaces and show how this allows for the application of existing CQMs to the subspace clustering problem. We provide theoretical and empirical justification for the proposed point-to-point distance, and then demonstrate on a number of common benchmark datasets that our proposed methods generally outperform existing graph-based CQMs in terms of choosing the best clustering and the number of clusters.

Deep learning-based clustering approaches for bioinformatics

Clustering is central to many data-driven bioinformatics research and serves a powerful computational method. In particular, clustering helps at analyzing unstructured and high-dimensional data in the form of sequences, expressions, texts and images. Further, clustering is used to gain insights into biological processes in the genomics level, e.g. clustering of gene expressions provides insights on the natural structure inherent in the data, understanding gene functions, cellular processes, subtypes of cells and understanding gene regulations. Subsequently, clustering approaches, including hierarchical, centroid-based, distribution-based, density-based and self-organizing maps, have long been studied and used in classical machine learning settings. In contrast, deep learning (DL)-based representation and feature learning for clustering have not been reviewed and employed extensively. Since the quality of clustering is not only dependent on the distribution of data points but also on the learned representation, deep neural networks can be effective means to transform mappings from a high-dimensional data space into a lower-dimensional feature space, leading to improved clustering results. In this paper, we review state-of-the-art DL-based approaches for cluster analysis that are based on representation learning, which we hope to be useful, particularly for bioinformatics research. Further, we explore in detail the training procedures of DL-based clustering algorithms, point out different clustering quality metrics and evaluate several DL-based approaches on three bioinformatics use cases, including bioimaging, cancer genomics and biomedical text mining. We believe this review and the evaluation results will provide valuable insights and serve a starting point for researchers wanting to apply DL-based unsupervised methods to solve emerging bioinformatics research problems.

A novel clustering algorithm for attributed graphs based on K-medoid algorithm

Articulateness and plasticity are two essential attributes that make a graph as an efficient model to real life problems. Nowadays, the attributed graph is received lots of attentions because of usability and effectiveness. In this study, a novel k-Medoid based clustering algorithm, which focuses simultaneously on both structural and contextual aspects using Signal and the weighted Jaccard similarities, are introduced. Two real life data-sets, Political Blogs and DBLP bibliography, are employed in order to evaluate and compare the proposed algorithm with state-of-the-art clustering algorithms. The results show the superiorities of the proposed algorithm in terms of cluster quality metrics.

MP04-01 LONGITUDINAL SUBPHENOTYPES OF URINARY SYMPTOMS IN TYPE I DIABETES

You have accessJournal of UrologyBenign Prostatic Hyperplasia: Epidemiology & Evaluation1 Apr 2018MP04-01 LONGITUDINAL SUBPHENOTYPES OF URINARY SYMPTOMS IN TYPE I DIABETES Karandeep Singh, Hunter Wessells, J. Quentin Clemens, Rodney L. Dunn, Sarah K. Holt, James Hotaling, Rodica Pop-Busui, William Herman, Alan M. Jacobson, Barbara H. Braffett, Aruna V. Sarma, and for the DCCT/EDIC Research Group Karandeep SinghKarandeep Singh More articles by this author , Hunter WessellsHunter Wessells More articles by this author , J. Quentin ClemensJ. Quentin Clemens More articles by this author , Rodney L. DunnRodney L. Dunn More articles by this author , Sarah K. HoltSarah K. Holt More articles by this author , James HotalingJames Hotaling More articles by this author , Rodica Pop-BusuiRodica Pop-Busui More articles by this author , William HermanWilliam Herman More articles by this author , Alan M. JacobsonAlan M. Jacobson More articles by this author , Barbara H. BraffettBarbara H. Braffett More articles by this author , Aruna V. SarmaAruna V. Sarma More articles by this author , and for the DCCT/EDIC Research Groupfor the DCCT/EDIC Research Group More articles by this author View All Author Informationhttps://doi.org/10.1016/j.juro.2018.02.151AboutPDF ToolsAdd to favoritesDownload CitationsTrack CitationsPermissionsReprints ShareFacebookTwitterLinked InEmail INTRODUCTION AND OBJECTIVES Lower urinary tract symptoms (LUTS) in men and urinary incontinence (UI) in women are dynamic conditions with numerous factors contributing to risk and progression. Understanding of natural history of urinary symptoms in diabetes is limited. We describe factors influencing the long-term trajectory of LUTS in men and UI in women with type 1 diabetes (T1DM) using non-parametric statistical methods. METHODS Longitudinal data from 565 men and 523 women in the Epidemiology of Diabetes Interventions and Complications (EDIC) study were used to define LUTS/UI annually over 5 years. Trajectories were identified using a k-means clustering approach on the American Urological Association Symptom Index (AUASI) in men and Sandvik Severity Index in women. Subphenotypes were selected based on cluster quality metrics. A multinomial random forest model was fit to predict subphenotype assignment using information from baseline. Factors associated with LUTS/UI trajectory were identified by permuting each predictor and assessing change in accuracy as compared to the non-permuted data using the fitted model. RESULTS At baseline, mean age was 51.0 (SD 6.4) and 50.1 (SD 7.0) in men and women, respectively. 27.5% of men reported moderate/severe LUTS and 28.7% of women moderate/severe UI. We identified 3 subphenotypes for UI among women and 4 for LUTS among men (Figure 1A, 1B). Subphenotypes D in men and C in women demonstrate a deterioration of function over time in contradistinction to stable patterns and low/moderate severity in other clusters. The classification model achieved a multiclass area-under-the-curve (AUC) of 0.65 for AUASI and 0.61 for Sandvik using 10-fold cross-validation. Based on permutation importance (Figure 1C, 1D), key predictors were age (p = 0.002) and autonomic dysfunction (p < 0.001) for LUTS trajectory and BMI (p = 0.02) for UI. CONCLUSIONS LUTS/UI are prevalent complications in men and women with T1DM. A limited number of factors were identified in delineating subphenotypes of LUTS/UI. These data may be relevant to future investigations of mechanisms for distinct endotypes of LUTS/UI, as well as candidates for interventions targeting metabolic factors contributing to variation in symptoms. © 2018FiguresReferencesRelatedDetails Volume 199Issue 4SApril 2018Page: e31-e32 Advertisement Copyright & Permissions© 2018MetricsAuthor Information Karandeep Singh More articles by this author Hunter Wessells More articles by this author J. Quentin Clemens More articles by this author Rodney L. Dunn More articles by this author Sarah K. Holt More articles by this author James Hotaling More articles by this author Rodica Pop-Busui More articles by this author William Herman More articles by this author Alan M. Jacobson More articles by this author Barbara H. Braffett More articles by this author Aruna V. Sarma More articles by this author for the DCCT/EDIC Research Group More articles by this author Expand All Advertisement Advertisement PDF downloadLoading ...

A Fully Automated Approach to Spike Sorting

Understanding the detailed dynamics of neuronal networks will require the simultaneous measurement of spike trains from hundreds of neurons (or more). Currently, approaches to extracting spike times and labels from raw data are time consuming, lack standardization, and involve manual intervention, making it difficult to maintain data provenance and assess the quality of scientific results. Here, we describe an automated clustering approach and associated software package that addresses these problems and provides novel cluster quality metrics. We show that our approach has accuracy comparable to or exceeding that achieved using manual or semi-manual techniques with desktop central processing unit (CPU) runtimes faster than acquisition time for up to hundreds of electrodes. Moreover, asingle choice of parameters in the algorithm is effective for a variety of electrode geometries and across multiple brain regions. This algorithm has the potential to enable reproducible and automated spike sorting of larger scale recordings than is currently possible.

A segmental framework for fully-unsupervised large-vocabulary speech recognition

Zero-resource speech technology is a growing research area that aims to develop methods for speech processing in the absence of transcriptions, lexicons, or language modelling text. Early term discovery systems focused on identifying isolated recurring patterns in a corpus, while more recent full-coverage systems attempt to completely segment and cluster the audio into word-like units—effectively performing unsupervised speech recognition. This article presents the first attempt we are aware of to apply such a system to large-vocabulary multi-speaker data. Our system uses a Bayesian modelling framework with segmental word representations: each word segment is represented as a fixed-dimensional acoustic embedding obtained by mapping the sequence of feature frames to a single embedding vector. We compare our system on English and Xitsonga datasets to state-of-the-art baselines, using a variety of measures including word error rate (obtained by mapping the unsupervised output to ground truth transcriptions). Very high word error rates are reported—in the order of 70–80% for speaker-dependent and 80–95% for speaker-independent systems—highlighting the difficulty of this task. Nevertheless, in terms of cluster quality and word segmentation metrics, we show that by imposing a consistent top-down segmentation while also using bottom-up knowledge from detected syllable boundaries, both single-speaker and multi-speaker versions of our system outperform a purely bottom-up single-speaker syllable-based approach. We also show that the discovered clusters can be made less speaker- and gender-specific by using an unsupervised autoencoder-like feature extractor to learn better frame-level features (prior to embedding). Our system’s discovered clusters are still less pure than those of unsupervised term discovery systems, but provide far greater coverage.

Two-Stage Load Pattern Clustering Using Fast Wavelet Transformation

Smart grids collect large volumes of smart meter data in the form of time series, or so-called load patterns. We outline the applications that benefit from analyzing this data (ranging from customer segmentation to operational system planning), and propose two-stage load pattern clustering. The first stage is performed per individual user and identifies the various typical daily power usage patterns (s)he exhibits. The second stage takes those typical user patterns as input to group users that are similar. To improve scalability, we use fast wavelet transformation (FWT) of the time series data, which reduces the dimensionality of the feature space where the clustering algorithm operates (i.e., from ${N}$ data points in the time domain to log ${N}$ ). Another qualitative benefit of FWT is that patterns that are identical in shape, but just differ in a (typically small) time shift still end up in the same cluster. Furthermore, we use ${g}$ -means instead of ${k}$ -means as the clustering algorithm. Our comprehensive set of experiments analyzes the impact of using FWT versus time-domain features, and ${g}$ - versus ${k}$ -means, to conclude that in terms of cluster quality metrics our system is comparable to state-of-the-art methods, while being more scalable (because of the dimensionality reduction).

Analysis of Network Clustering Algorithms and Cluster Quality Metrics at Scale.

OverviewNotions of community quality underlie the clustering of networks. While studies surrounding network clustering are increasingly common, a precise understanding of the realtionship between different cluster quality metrics is unknown. In this paper, we examine the relationship between stand-alone cluster quality metrics and information recovery metrics through a rigorous analysis of four widely-used network clustering algorithms—Louvain, Infomap, label propagation, and smart local moving. We consider the stand-alone quality metrics of modularity, conductance, and coverage, and we consider the information recovery metrics of adjusted Rand score, normalized mutual information, and a variant of normalized mutual information used in previous work. Our study includes both synthetic graphs and empirical data sets of sizes varying from 1,000 to 1,000,000 nodes.Cluster Quality MetricsWe find significant differences among the results of the different cluster quality metrics. For example, clustering algorithms can return a value of 0.4 out of 1 on modularity but score 0 out of 1 on information recovery. We find conductance, though imperfect, to be the stand-alone quality metric that best indicates performance on the information recovery metrics. Additionally, our study shows that the variant of normalized mutual information used in previous work cannot be assumed to differ only slightly from traditional normalized mutual information.Network Clustering AlgorithmsSmart local moving is the overall best performing algorithm in our study, but discrepancies between cluster evaluation metrics prevent us from declaring it an absolutely superior algorithm. Interestingly, Louvain performed better than Infomap in nearly all the tests in our study, contradicting the results of previous work in which Infomap was superior to Louvain. We find that although label propagation performs poorly when clusters are less clearly defined, it scales efficiently and accurately to large graphs with well-defined clusters.

Multilevel hierarchical kernel spectral clustering for real-life large scale complex networks.

Kernel spectral clustering corresponds to a weighted kernel principal component analysis problem in a constrained optimization framework. The primal formulation leads to an eigen-decomposition of a centered Laplacian matrix at the dual level. The dual formulation allows to build a model on a representative subgraph of the large scale network in the training phase and the model parameters are estimated in the validation stage. The KSC model has a powerful out-of-sample extension property which allows cluster affiliation for the unseen nodes of the big data network. In this paper we exploit the structure of the projections in the eigenspace during the validation stage to automatically determine a set of increasing distance thresholds. We use these distance thresholds in the test phase to obtain multiple levels of hierarchy for the large scale network. The hierarchical structure in the network is determined in a bottom-up fashion. We empirically showcase that real-world networks have multilevel hierarchical organization which cannot be detected efficiently by several state-of-the-art large scale hierarchical community detection techniques like the Louvain, OSLOM and Infomap methods. We show that a major advantage of our proposed approach is the ability to locate good quality clusters at both the finer and coarser levels of hierarchy using internal cluster quality metrics on 7 real-life networks.