Cluster Validity Index Research Articles

Determining the Correct Number of Clusters in the CT Image Segmentation

Clustering algorithm plays an essential role in CT image segmentation, and cluster validity index is an essential component in clustering analysis. There are a lot of validity indices used for assessing clustering results, that is, determine the optimal cluster number. But the existing validity indices are often ineffective for the datasets with irregular-shaped clusters and corrupted by noise. This study aims to define a novel validity index which cannot be affected by the shapes of clusters and corrupted by noise of the investigated datasets. Chain-based distance different from original Euclidean distance is defined first, then by a multidimensional scaling (MDS) transformation, all points are mapped into a new data space. After evaluation of compactness and separation twice in datasets, a novel validity index is proposed. A lot of synthetic datasets and several typical CT images were used for validating the proposed validity index. Experimental results validate the proposed index and this index is applicable to the datasets with arbitrary-shaped clusters and corrupted by noise, which is helpful in clustering analysis and computer-aided detection system.

Time-series clustering approaches for subsurface zonation and hydrofacies detection using a real time-lapse electrical resistivity dataset

One main application of electrical resistivity tomography (ERT) is the non-invasive detection of geological or hydrological structures in the shallow subsurface. This paper investigates the capability of time-series clustering to retrieve such features on real time-lapse ERT datasets considering three aspects: (1) the comparison between three clustering algorithms k-means, hierarchical agglomerative clustering (HAC), and Gaussian Mixture Model (GMM), including the question of the optimal choice of cluster number and the identification of resistivity series whose classification is uncertain, (2) the effect of adding a spatial constraint in clustering, and (3) the robustness of the approaches to various representations of resistivity values and the number of time-steps involved in the clustering. The real time-lapse ERT dataset is obtained from dipole-dipole arrays on a 48 electrodes profile installed on the top of the Rochefort cave in Belgium. It consists of resistivity time-series defined over 465 days and associated with 1558 cells of the 2D ERT models derived from a time-lapse inversion. The clustering results are appreciated using clustering validation indices and further confronted with the expert-based structural model of the site. Results show that the three clustering algorithms provide similar spatial patterns on the standardized data and reveal correlated resistivity time-series. Some clusters are, however, spatially split and regroup time-series with a wide range of mean resistivity, suggesting different geological units within these groups. Clustering on the raw resistivity time-series may also appear inconsistent as the averaged resistivity series per cluster are highly correlated, thus missing the hydrological and functional traits of the subsurface elements. Applying a spatial constraint to the clustering of standardized data increases the number of clusters in order to retrieve spatially tied clusters. The grouped series are more homogeneous in terms of mean resistivity due to their spatial proximity, but some inconsistencies may remain due to synchronous hydrological forcing. Applying the clustering to various time-series representations allows us to gain confidence about the redundant spatial patterns. However, the patterns obtained from the clustering of the full standardized dataset cannot be reproduced from continuous sub-samples up to 100 days, but well from less than 20 samples picked randomly over the 465 days. Accordingly, our study highlights the importance of time-variable parameters in the identification of structural facies and hydrofacies with ERT while demonstrating the strength of long-term monitoring.

Personalized Web-Based Cognitive Rehabilitation Treatments for Patients with Traumatic Brain Injury: Cluster Analysis.

BackgroundTraumatic brain injury (TBI) is a leading cause of disability worldwide. TBI is a highly heterogeneous disease, which makes it complex for effective therapeutic interventions. Cluster analysis has been extensively applied in previous research studies to identify homogeneous subgroups based on performance in neuropsychological baseline tests. Nevertheless, most analyzed samples are rarely larger than a size of 100, and different cluster analysis approaches and cluster validity indices have been scarcely compared or applied in web-based rehabilitation treatments.ObjectiveThe aims of our study were as follows: (1) to apply state-of-the-art cluster validity indices to different cluster strategies: hierarchical, partitional, and model-based, (2) to apply combined strategies of dimensionality reduction by using principal component analysis and random forests and perform stability assessment of the final profiles, (3) to characterize the identified profiles by using demographic and clinically relevant variables, and (4) to study the external validity of the obtained clusters by considering 3 relevant aspects of TBI rehabilitation: Glasgow Coma Scale, functional independence measure, and execution of web-based cognitive tasks.MethodsThis study was performed from August 2008 to July 2019. Different cluster strategies were executed with Mclust, factoextra, and cluster R packages. For combined strategies, we used the FactoMineR and random forest R packages. Stability analysis was performed with the fpc R package. Between-group comparisons for external validation were performed using 2-tailed t test, chi-square test, or Mann-Whitney U test, as appropriate.ResultsWe analyzed 574 adult patients with TBI (mostly severe) who were undergoing web-based rehabilitation. We identified and characterized 3 clusters with strong internal validation: (1) moderate attentional impairment and moderate dysexecutive syndrome with mild memory impairment and normal spatiotemporal perception, with almost 66% (111/170) of the patients being highly educated (P<.05); (2) severe dysexecutive syndrome with severe attentional and memory impairments and normal spatiotemporal perception, with 49.2% (153/311) of the patients being highly educated (P<.05); (3) very severe cognitive impairment, with 45.2% (42/93) of the patients being highly educated (P<.05). We externally validated them with severity of injury (P=.006) and functional independence assessments: cognitive (P<.001), motor (P<.001), and total (P<.001). We mapped 151,763 web-based cognitive rehabilitation tasks during the whole period to the 3 obtained clusters (P<.001) and confirmed the identified patterns. Stability analysis indicated that clusters 1 and 2 were respectively rated as 0.60 and 0.75; therefore, they were measuring a pattern and cluster 3 was rated as highly stable.ConclusionsCluster analysis in web-based cognitive rehabilitation treatments enables the identification and characterization of strong response patterns to neuropsychological tests, external validation of the obtained clusters, tailoring of cognitive web-based tasks executed in the web platform to the identified profiles, thereby providing clinicians a tool for treatment personalization, and the extension of a similar approach to other medical conditions.

An automatic clustering technique for query plan recommendation

The query optimizer is responsible for identifying the most efficient Query Execution Plans (QEP’s). The distributed database relations may be kept in several places. These results in a dramatic increase in the number of alternative query’ plans. The query optimizer cannot exhaustively explore the alternative query plans in a vast search space at reasonable computational costs. Henceforth, reusing the previously generated plans instead of generating new plans for new queries is an efficient technique for query processing. To improve the accuracy of clustering, we’ve rewritten the queries to standardize their structures. Furthermore, TF representation schema has been used to convert the queries into vectors. In this paper, we’ve introduced a multi-objective automatic query plan recommendation method, a combination of incremental DBSCAN and NSGA-II. The quality of the results of incremental DBSCAN has been influenced by Minpts (minimum points) and Eps (epsilon). Two cluster validity indices, Dunn index and Davies–Bouldin index, have simultaneously been optimized to calculate the goodness of an answer. Comparative results have been shown against the incremental DBSCAN and K-means regarding an external cluster validity index, namely, the ARI. By comparing different types of query workloads, we’ve found that the introduced method outperforms the other well-known approaches.

Development of ‘speed ratio’ based level of service criteria on undivided urban streets in mixed traffic context

Developing countries are facing challenges in sustaining urban traffic congestion due to rapid urbanization. To manage urban traffic, a traffic engineer first needs to assess the current operational condition on urban roads. Level of service (LOS) is used to define the operational traffic condition within a traffic stream in terms of service quality that a facility is providing to its user. This paper proposes a novel approach to estimate ‘speed ratio’ by considering individual free-flow speed (FFS) of different vehicle categories. This study also provides a comparison between FFS estimated using the methods given in Highway Capacity Manual 2010 and Indian-Highway Capacity Manual 2018. LOS criteria were developed using five clustering technique: K-means, K-medoids, Clustering Large Applications, Fuzzy-C Means, and Hierarchical Agglomerative Clustering. Both internal and external cluster validation indices were used to find the optimal number of clusters and suitable clustering algorithms.

A fast automatic extraction method for rock mass discontinuity orientation using fast k-means++ and fast silhouette based on 3D point cloud

In this paper, a fast automatic extraction method for rock discontinuity orientation using fast k-means++ and fast silhouette based on three-dimensional (3D) point cloud is proposed. The main contributions of the proposed method include: (1) a concept of base vector is proposed, (2) a fast k-means++ algorithm and a fast clustering validity index of silhouette are proposed and combined to automatically achieve a fast optimal grouping of discontinuities, and (3) the generation and optimization of independent discontinuity based on a double-clustering algorithm. The proposed method consists of three steps : (1) point cloud preprocessing, (2) optimal grouping of discontinuities, and (3) discontinuity generation and optimization. Several cases, including two polyhedral models under laboratory conditions and two rock slope models, are adopted to verify the accuracy and efficiency of the proposed method. A sensitivity analysis is performed to determine the optimal parameters. The results show that the proposed method can automatically achieve a fast and effective extraction of discontinuity orientations compared with the classical method, Slob (2010), Riquelme et al. (2014) and Li et al. (2016), and the efficiency is significantly improved compared with Rousseeuw (1987), Arthur and vassilvitskii (2007) and Li et al. (2016).

Controlling 2D Artificial Data Mixtures Overlap

Clustering methods are used for identifying groups of similar objects considered as homogenous set. Unfortunately, analytic performance evaluation of clustering methods is a difficult task because of their ad-hoc nature. In this paper, we propose a new test case generator of artificial data for 2 dimensional Gaussian mixtures. The proposed generator has two interesting advantages: the first one is its ability to produce simulated mixture for any number of components, while the second one resides in the fact that it formally quantifies the overlap rate which allows us to add some complexity to the data. Clustering algorithms and validity indices behavior is also analyzed by changing the overlap rate between clusters.

Mining event logs for knowledge discovery based on adaptive efficient fuzzy Kohonen clustering network

As a digital representation of building projects, Building Information Modeling (BIM) can accumulate large volumes of log data containing hidden knowledge for deep exploration. However, such ever-increasing logs are likely to suffer from high complexity, inaccuracy, and uncertainty, which will inevitably raise challenges in uncovering latent and meaningful patterns. In order to yield satisfactory clustering quality and efficiency for better design process management, a novel clustering-based BIM event log mining approach is put forward in this paper. For one thing, a hybrid clustering algorithm named adaptive efficient fuzzy Kohonen clustering network (AEFKCN) is developed with a modified learning rate to accelerate the convergence. For another, a new clustering validity index (CVI) only relying on boundary points is designed to reduce computational complexity. An experiment is conducted in a 4 GB realistic BIM design event log dataset to validate the effectiveness of the proposed method. It begins from extracting a set of features associated with designers’ engagement and efficiency and ends up retrieving inherent insights into the person’s design behavioral patterns. Moreover, the cluster analysis can significantly distinguish the design productivity at different time periods into the high, medium, and low level, which presents a unique opportunity in understanding and assessing design productivity objectively. Practically, our method can support data-driven decision making for managers to strategically schedule personalized work for different designers, aiming to boost design efficiency and smooth the design process.

A genetic algorithm using Calinski-Harabasz index for automatic clustering problem

Data clustering is a technique that aims to represent a dataset in clusters according to their similarities. In clustering algorithms, it is usually assumed that the number of clusters is known. Unfortunately, the optimal number of clusters is unknown for many applications. This kind of problem is called Automatic Clustering. There are several cluster validity indexes for evaluating solutions, it is known that the quality of a result is influenced by the chosen function. From this, a genetic algorithm is described in this article for the resolution of the automatic clustering using the Calinski-Harabasz Index as a form of evaluation. Comparisons of the results with other algorithms in the literature are also presented. In a first analysis, fitness values equivalent or higher are found in at least 58% of cases for each comparison. Our algorithm can also find the correct number of clusters or close values in 33 cases out of 48. In another comparison, some fitness values are lower, even with the correct number of clusters, but graphically the partitioning are adequate. Thus, it is observed that our proposal is justified and improvements can be studied for cases where the correct number of clusters is not found.

An ensemble approach to outlier detection using some conventional clustering algorithms

Outlier detection is an important requirement in data mining and machine learning. When data mining and machine learning algorithms are applied on the datasets with outliers, it leads to erroneous conclusion about the data. Therefore, researchers have been working in this field to remove outliers from dataset so that meaningful information from the datasets can be retrieved. In this paper, we take a cluster based ensemble approach for outlier detection, the backbone of which are some conventional clustering algorithms. Keeping in mind the drawbacks of supervised and semi supervised learning, we have relied on unsupervised learning algorithms. For our cluster based ensemble approach, we use three clustering algorithms, namely K-means, K-means++, and Fuzzy C-means. Our model intelligently combines results from individual clustering algorithms, assigning probabilities to each data point in order to decide its belongingness to a certain cluster. We have proposed a technique to assign a membership value to a data point in case of hard clustering algorithms, as we want to keep the flexibility of combining hard and soft clustering algorithms. From the probabilities assigned by the ensemble model, we then identify the outliers from the dataset. After removing these data points from the dataset, we obtain better values of cluster validity indices, thus reaffirming that removal of outliers has resulted in more stringent clusters of data. We have used five different cluster validity indices in our work to measure the goodness of the clusters formed, considering eight widely used datasets for evaluation of the proposed model amongst which three are large datasets. We have noticed a significant improvement in the cluster validity indices after applying our outlier detection algorithm. The experimental results prove that the proposed method is empirically sound.

A consensus multi-view multi-objective gene selection approach for improved sample classification

BackgroundIn the field of computational biology, analyzing complex data helps to extract relevant biological information. Sample classification of gene expression data is one such popular bio-data analysis technique. However, the presence of a large number of irrelevant/redundant genes in expression data makes a sample classification algorithm working inefficiently. Feature selection is one such high-dimensionality reduction technique that helps to maximize the effectiveness of any sample classification algorithm. Recent advances in biotechnology have improved the biological data to include multi-modal or multiple views. Different ‘omics’ resources capture various equally important biological properties of entities. However, most of the existing feature selection methodologies are biased towards considering only one out of multiple biological resources. Consequently, some crucial aspects of available biological knowledge may get ignored, which could further improve feature selection efficiency.ResultsIn this present work, we have proposed a Consensus Multi-View Multi-objective Clustering-based feature selection algorithm called CMVMC. Three controlled genomic and proteomic resources like gene expression, Gene Ontology (GO), and protein-protein interaction network (PPIN) are utilized to build two independent views. The concept of multi-objective consensus clustering has been applied within our proposed gene selection method to satisfy both incorporated views. Gene expression data sets of Multiple tissues and Yeast from two different organisms (Homo Sapiens and Saccharomyces cerevisiae, respectively) are chosen for experimental purposes. As the end-product of CMVMC, a reduced set of relevant and non-redundant genes are found for each chosen data set. These genes finally participate in an effective sample classification.ConclusionsThe experimental study on chosen data sets shows that our proposed feature-selection method improves the sample classification accuracy and reduces the gene-space up to a significant level. In the case of Multiple Tissues data set, CMVMC reduces the number of genes (features) from 5565 to 41, with 92.73% of sample classification accuracy. For Yeast data set, the number of genes got reduced to 10 from 2884, with 95.84% sample classification accuracy. Two internal cluster validity indices - Silhouette and Davies-Bouldin (DB) and one external validity index Classification Accuracy (CA) are chosen for comparative study. Reported results are further validated through well-known biological significance test and visualization tool.

특징공간에서의 군집화 유효성 지수의 개발

In clustering problems, it is difficult to know the optimum number of clusters for a given dataset a priori. Hence, cluster validity indices (CVIs) measuring the fitness of partitions produced by clustering algorithms are important criteria to evaluate the goodness of clustering results. However, many CVIs suffer from asymmetric, arbitrary, noise, and sub-cluster shape of clusters, especially for high-dimensional dataset. Therefore, this paper proposes new CVIs in feature space in which the proposed CVIs transform arbitrary shape of clusters into elliptical or circular clusters by using kernel functions. The experimental results show that the proposed CVIs have a good performance to estimate the optimum number of clusters for asymmetric, noise, and arbitrary shapes of clusters.

Clustering of Time Series Water Quality Data Using Dynamic Time Warping: A Case Study from the Bukhan River Water Quality Monitoring Network

It is essential to monitor water quality for river water management because river water is used for various purposes and is directly related to the health and safety of a population. Proper network installation and removal is an important part of water quality monitoring and network operation efficiency. To do this, cluster analysis based on calculated similarity between measuring stations can be used. In this study, we measured the similarities between 12 water quality monitoring stations of the Bukhan River. River water quality data always have a station-dependent time lag because water flows from upstream to downstream; therefore, we proposed a Dynamic Time Warping (DTW) algorithm that searches for the minimum distance by changing and comparing time-points, rather than using the Euclidean algorithm, which compares the same time-point. Both Euclidean and DTW algorithms were applied to nine water quality variables to identify similarities between stations, and K-medoids cluster analysis were performed based on the similarity. The Clustering Validation Index (CVI) was used to select the optimal number of clusters. Our results show that the Euclidean algorithm formed clusters by mixing mainstream and tributary stations; the mainstream stations were largely divided into three different clusters. In contrast, the DTW algorithm formed clear clusters by reflecting the characteristics of water quality and watershed. Furthermore, because the Euclidean algorithm requires the lengths of the time series to be the same, data loss was inevitable. As a result, even where clusters were the same as those obtained by DTW, the characteristics of the water quality variables in the cluster differed. The DTW analysis in this study provides useful information for understanding the similarity or difference in water parameter values between different locations. Thus, the number and location of required monitoring stations can be adjusted to improve the efficiency of field monitoring network management.

A methodology for automatic parameter-tuning and center selection in density-peak clustering methods

The density-peak clustering algorithm, which we refer to as DPC, is a novel and efficient density-based clustering approach. The method has the advantage of allowing non-convex clusters, and clusters of variable size and density, to be grouped together, but it also has some limitations, such as the visual location of centers and the parameter tuning. This paper describes an optimization-based methodology for automatic parameter/center selection applicable both to the DPC and to other algorithms derived from it. The objective function is an internal/external cluster validity index, and the decisions are the parameterization of the algorithm and the choice of centers. The internal validation measures lead to an automatic parameter-tuning process, and the external validation measures lead to the so-called optimal rules, which are a tool to bound the performance of a given algorithm from above on the set of parameterizations. A numerical experiment with real data was performed for the DPC and for the fuzzy weighted k-nearest neighbor (FKNN-DPC) which validates the automatic parameter-tuning methodology and demonstrates its efficiency compared to the state of the art.

A new robust fuzzy clustering validity index for imbalanced data sets

Determining the number of clusters of a data set, which is usually evaluated by a clustering validity index (CVI), is a significant issue in clustering analysis. While several CVIs have been proposed, the imperfect clustering results of the fuzzy c-means (FCM) clustering algorithm on imbalanced data sets may affect their decisions. To address this problem, the impact of imperfect clustering results on the traditional CVI is first analyzed, and it is found that the distance between two imbalanced clusters becomes closer, which will subsequently impact the separation metric. Inspired by this, a new fuzzy CVI called the imbalanced index (IMI) is proposed in this paper. IMI is the ratio of the fuzzy compactness and separation metrics. The main characteristic of IMI is the new definition of the separation metric, in which the imbalance ratio of two clusters is used to enlarge the distance between their centers. IMI is then employed to evaluate the clustering results of FCM on a variety of data sets, and is compared with several well-known CVIs. The experimental results demonstrate that IMI is robust to the imperfect clustering results of FCM caused by imbalanced data distributions and achieves superior performance as compared to other CVIs.

Advanced Machine Learning on Cognitive Computing for Human Behavior Analysis

With the increasing size of data, massive amounts of data are being generated continuously. It is hoped to find a cognitive computing technology that can effectively learn and process large-scale data. The decision tree algorithm is optimized from the perspective of machine learning. A cognitive computing model based on context-aware data flow is constructed. Classification and regression tree (CART) algorithm is used in the data computing layer of the cognitive model. In addition, the clustering effectiveness index based on frequent patterns optimizes the K-means clustering method. The performance of the algorithm is analyzed through simulation experiments. The results show that the CART algorithm requires fewer training data sets while guaranteeing classification accuracy. Also, the algorithm has obvious advantages under large-scale data. In the application of actual data set, on Over, Over+Noise, and Bridge, only the clustering validity index based on frequent pattern (FPCVI) index proposed finds the correct number of clusters. The application of DPCK-K-means clustering algorithm can ensure the accuracy and stability of behavior classification, which is greatly significant for operators to analyze user behavior and develop personalized services.

Cluster validity index for irregular clustering results

Different clustering algorithms with different parameter settings can produce various partitions on the input data. Without the priori knowledge, it is difficult for users to select the proper clustering algorithm and the parameters for the specific data in advance. Therefore, the cluster validity index (CVI) is crucial to help select the best partition that fits the underlying structure of the data. However, most existing CVIs (including some recent ones that designed for complex partitions) have strong assumptions. They only work well for partitions where clusters are spherically distributed, with similar sizes and densities, and with large separation distances. In complicated situations where irregular clustering results (i.e., clustering results having clusters in arbitrary shapes, different sizes and densities, and with small separation distances) exist, they usually fail to find the best fitting partition. Focusing on the insufficiencies of the existing CVIs (designed for hard clustering), this paper presents a new index that helps to find the best partition produced by hard clustering algorithms when irregular clustering results exist. The proposed index uses the density changes inside a cluster, and the density changes from the inner to the inter-cluster regions of the cluster to evaluate its quality. Experiments are implemented on both real and synthetic datasets. 11 other CVIs including some well known as well as recently proposed ones are also tested for comparison. Experimental results are given to demonstrate the effectiveness of the new index.

Leveraging clustering validation index for detecting ‘stops’ in spatial trajectory data: a semi-automatic approach

ABSTRACT Spatial trajectory data, interestingly attracting organizations to obtain mobility-based activity patterns of smartphone users. One of the basic objective in this regard is the determination of ‘stops’ or technically high-density points in the trajectory data. Most works carried out in this area uses variants of density-based clustering algorithms for determining stop points. One of the notable challenges in this area is the determination of the parameters for the clustering algorithm, which highly affects the accuracy of detecting the ‘stops’.In this paper a semi-automatic approach is proposed based on particle swarm optimization, DBSCAN, and S_Dbw internal validity index for determining appropriate parameter values for the clustering algorithm and fast convergence.

OsamorSoft: clustering index for comparison and quality validation in high throughput dataset

The existence of some differences in the results obtained from varying clustering k-means algorithms necessitated the need for a simplified approach in validation of cluster quality obtained. This is partly because of differences in the way the algorithms select their first seed or centroid either randomly, sequentially or some other principles influences which tend to influence the final result outcome. Popular external cluster quality validation and comparison models require the computation of varying clustering indexes such as Rand, Jaccard, Fowlkes and Mallows, Morey and Agresti Adjusted Rand Index (ARIMA) and Hubert and Arabie Adjusted Rand Index (ARIHA). In literature, Hubert and Arabie Adjusted Rand Index (ARIHA) has been adjudged as a good measure of cluster validity. Based on ARIHA as a popular clustering quality index, we developed OsamorSoft which constitutes DNA_Omatrix and OsamorSpreadSheet as a tool for cluster quality validation in high throughput analysis. The proposed method will help to bridge the yawning gap created by lesser number of friendly tools available to externally evaluate the ever-increasing number of clustering algorithms. Our implementation was tested alongside with clusters created with four k-means algorithms using malaria microarray data. Furthermore, our results evolved a compact 4-stage OsamorSpreadSheet statistics that our easy-to-use GUI java and spreadsheet-based tool of OsamorSoft uses for cluster quality comparison. It is recommended that a framework be evolved to facilitate the simplified integration and automation of several other cluster validity indexes for comparative analysis of big data problems.

Models for Internal Clustering Validation Indexes Based on Hadoop-MapReduce

Cluster validation aims to both evaluate the results of clustering algorithms and predict the number of clusters. It is usually achieved using several indexes. Traditional internal clustering validation indexes (CVIs) are mainly based in computing pairwise distances which results in a quadratic complexity of the related algorithms. The existing CVIs cannot handle large data sets properly and need to be revisited to take account of the ever-increasing data set volume. Therefore, design of parallel and distributed solutions to implement these indexes is required. To cope with this issue, the authors propose two parallel and distributed models for internal CVIs namely for Silhouette and Dunn indexes using MapReduce framework under Hadoop. The proposed models termed as MR_Silhouette and MR_Dunn have been tested to solve both the issue of evaluating the clustering results and identifying the optimal number of clusters. The results of experimental study are very promising and show that the proposed parallel and distributed models achieve the expected tasks successfully.