Dunn Index Research Articles

Models for Internal Clustering Validation Indexes Based on Hadoop-MapReduce

Cluster validation aims to both evaluate the results of clustering algorithms and predict the number of clusters. It is usually achieved using several indexes. Traditional internal clustering validation indexes (CVIs) are mainly based in computing pairwise distances which results in a quadratic complexity of the related algorithms. The existing CVIs cannot handle large data sets properly and need to be revisited to take account of the ever-increasing data set volume. Therefore, design of parallel and distributed solutions to implement these indexes is required. To cope with this issue, the authors propose two parallel and distributed models for internal CVIs namely for Silhouette and Dunn indexes using MapReduce framework under Hadoop. The proposed models termed as MR_Silhouette and MR_Dunn have been tested to solve both the issue of evaluating the clustering results and identifying the optimal number of clusters. The results of experimental study are very promising and show that the proposed parallel and distributed models achieve the expected tasks successfully.

Wage Level as One of the Most Important Indicators of the Quantitative Aspect of the Standard of Living of the Population and Selected Indicators of Economic Maturity in OECD Member Countries

The present paper focuses on the comparison of wage levels across OECD countries, the research data coming from an official OECD website. The following eight variables are employed in this study – the average wage, minimum wage, GDP per capita, tertiary education attainment, employment ratio, trade unions, labour productivity and inflation rate. The average wage represents the main explained variable in regression and correlation analysis, the remaining seven variables being used as potential explanatory ones. In order to compare living standards in different countries, average and minimum wages as well as per capita GDP data were adjusted to relative purchasing power parity. The principal objective was to identify which explanatory variables statistically significantly affect the average wage. The analysis showed that only three of them – namely the employment ratio, GDP per capita and labour productivity – have a significant effect at a 5% statistical level. The regression hyperplane with a forward stepwise selection was applied. Nine clusters of OECD countries were created based on both all the eight variables and four of them selected in regression analysis (the average wage and three explanatory ones) with the aim to identify the countries that coexist in the same cluster. Ward's method and Euclidean distance are utilized in cluster analysis, the number of clusters being determined with the use of the Dunn index. The study also aims at the prediction of the average wage by 2022, which was made via exponential smoothing of time series. (The greatest purchasing power is reported by Luxembourg, Switzerland, Iceland, the U.S., the Netherlands, Denmark, Norway and Austria, the highest average wage growth rate by 2022 being expected in the Baltic and some other post-communist countries.)

Development of the WEEE grouping system in South Korea using the hierarchical and non-hierarchical clustering algorithms

South Korea has been operating an extended producer responsibility system (EPR) since 2003 to collect, transport, and dispose of e-waste. Until 2019, the EPR system was operated with a total number of 27 electronic products classified into five categories based on weight and volume, but 23 items will be added in 2020 along with a change to five categories based on the function of the products. In this study that used actual operational data related to the collection, transport, and recycling steps from recycling plants in South Korea, we have analyzed how well the new five-category grouping appropriately reflected actual recycling industrial conditions and have provided optimal classification alternatives. The results showed that clustering accuracy was the best for the classification that used the hierarchical method. In particular, the evaluation index, silhouettes, showed the best accuracy with three clusters (0.4155), and the Dunn index indicated the best performance with four clusters (0.2333). Based these results, ANOVA tests were implemented, and showed that the three clusters in the relevant models were significantly different with regard to takt-time, weight, volume, and no. of recycling processes (p ≤ 0.01) and to both recycling cost and value of material (p ≤ 0.05). In contrast, with regard to the grouping suggested by the South Korean government, the overall results of the clustering accuracy using silhouettes and Dunn indices were –0.2028 and 0.058, respectively. In conclusion, the new grouping suggested by the hierarchical method with four clusters can be utilized as a political decision-making tool.

Clustering of unknown protocol messages based on format comparison

As a solution to detect and analyse unknown or proprietary protocols, Protocol Reverse Engineering(PRE) has been developed swiftly in recent years. In this field, message clustering aimed at protocol format serves as a fundamental solution for differentiating of unknown protocol messages. This paper works on the problem of format-oriented message clustering of unknown protocols, including messages from proprietary or non-cooperative network environments with their specifications unknown. By introducing basic rules of ABNF, we define Token Format Distance (TFD) and Message Format Distance (MFD) to represent format similarity of tokens and messages, and introduce Jaccard Distance and an optimized sequence alignment algorithm (MFD measurement) to compute them. Then, a distance matrix is built by MFD and we feed it to DBSCAN algorithm to cluster unknown protocol messages into classes with different formats. In this process, we design an unsupervised clustering strategy with Silhouette Coefficient and Dunn Index applied to parameter selecting of DBSCAN. In experiment on two datasets, the harmonic average v-measures of homogeneity and completeness on result clusters are both above 0.91, with fmis and coverages no less than 0.97. Together with iqr of v-measure and fmi bellow 0.1 and 0.03 separately in boxplot analyses, this method is proved to have remarkable validity and stability. Comprehensive analyses and comparisons on these indexes also show considerable advantages of our method over previous work.

Bayesian neural network–based thermal error modeling of feed drive system of CNC machine tool

It is well known that thermal error has a significant impact on the accuracy of CNC machine tools. In order to decrease the thermally induced positioning error of machine tools, a novel thermal error modeling approach based on Bayesian neural network is proposed in this paper. The relationship between the temperature rise and positioning error of the feed drive system is investigated by simultaneously measuring the thermal characteristics that include the temperature field and positioning error of the CNC machine tool. Fuzzy c-means (FCM) clustering and correlation analysis are used to select temperature-sensitive points, and the Dunn index is introduced to determine the optimal number of clustering groups, which can inhibit the multicollinearity problem among temperature measuring points effectively. The least-square linear fitting is applied to explore the feature of the positioning error data. The results show that compared with the BP neural network and multiple linear regression model, the Bayesian neural network not only has higher prediction accuracy but also can guarantee excellent prediction performance under different working conditions. The prediction results obtained under different operating conditions indicate that the maximum thermal error can be reduced from around 18.2 to 5.14 μm by using the Bayesian neural network, which represents a 71% reduction in the thermally induced error of the feed drive system of machine tool.

Partial discharges and noise classification under HVDC using unsupervised and semi-supervised learning

This paper tackles the problem of the classification of partial discharge (PD) and noise signals by applying unsupervised and semi-supervised learning methods. The first step in the proposed methodology is to prepare a set of classification features from the statistical moments of the distribution of the Wavelet detail coefficients extracted from a dataset of signals acquired from a test cell under 40 kVDC. In a second step, an unsupervised learning framework that implements the k–means algorithm is applied to reduce the dimensionality of this initial feature set. The Silhouette index is used to evaluate the number of natural clusters in the dataset while the Dunn index is used to determine which subset of features produces the best clustering quality. Since the unsupervised learning does not provide any method for result validation, then the third step in the methodology of this paper consists of applying a semi-supervised learning framework that implements Transductive Support-Vector Machines. The labeling of the test set that is required in this framework for the result validation is carried out by visual checking of the signal waveforms assisted by GUI tools such as the software PDflex. The results using this methodology showed a high classification accuracy and proved that both learning frameworks can be combined to optimize the selection of classification features.

Cluster Analysis of Earthquake's Data Clustering in Indonesia using Fuzzy K-means Clustering

The earthquake is shocks or vibrations in the earth's surface because of shifting layers of rock at the base of the earth's surface. This natural phenomenon is common in Indonesia because it lies between Australian, Eurasian, Pacific plates, and it location surrounded by a ring of fire precisely. Therefore, this study aims to cluster earthquake events in Indonesia and describe the characteristics of each group based on clustering results. The method used is the Fuzzy K-Means Clustering. The clustering results obtained from clustering based on the depth, longitude, and latitude. In this study, the data used is the earthquake's data, which has a magnitude greater than or equal to 5 SR and only clumped by depth. Based on the Davies-Bouldin and Dunn index, the best clustering is 2 clusters which researchers cluster earthquake data in Indonesia into deep and shallow clusters.

Testing the static and dynamic performance of statistical methods for the detection of national industrial clusters

AbstractThis paper proposes a new framework to test the static and dynamic performance of clustering techniques for the detection of national industrial clusters. Principal component analysis (PCA), latent profile analysis (LPA), Louvain community detection algorithm (Louvain), and fuzzy analysis clustering (Fuzzy) are compared. It also proposes a Penalized Dunn Index as a novel metric for assessing the quality of industrial cluster dynamics. The findings indicate that PCA results in large variations in the total number of clusters among countries, but industrial clusters are not consistent over time. LPA and Fuzzy algorithms perform well in the case of static datasets, whereas Louvain offers a good balance between cluster diversity and dynamic consistency.

Abstract P131: Optimal Phenomapping Strategy to Identify Novel Subgroups of Patients With Type 2 Diabetes and High Cardiovascular Disease Risk

Introduction: Phenomapping can be used to identify subgroups of patients with similar risk factor profiles; however, there is no widely accepted clustering method. We compared 3 phenomapping strategies to identify groups of individuals with diabetes mellitus (DM) and high cardiovascular disease (CVD) risk. Methods: Participants with DM and no baseline CVD from the ACCORD trial (N = 6591) were included in this study. Among 45 mixed data variables, we used an informed variable selection method with optimization of the Wald index to select the top 20 covariates associated with survival. Gaussian mixture models (GMM), Latent Class Analysis (LCA), and finite mixture model-based clustering (FMM) were performed to identify mutually exclusive phenogroups. Bayesian information criterion (BIC) and Dunn index were calculated for each method and compared. The primary composite endpoint was cardiovascular death, non-fatal myocardial infarction, and non-fatal stroke. Results: FMM clustering with 3 phenogroups was the optimal clustering strategy as determined by the lowest BIC and highest Dunn index (Figs A-B). Phenogroup 1 had a higher burden of comorbidities and DM complications; phenogroup 2 were older with intermediate CV comorbidity burden, but lowest complications from DM; and phenogroup 3, the largest group, had a lower prevalence of CV-related comorbidities. In adjusted Cox models, over 4.9 years median follow-up, phenogroup 1 and 2 participants (vs. phenogroup 3) had a significantly higher risk of primary composite endpoint (Fig C). In Cox analysis, when group membership was added to 10 optimal predictors of primary composite endpoint, the C -index improved from 0.68 to 0.70 ( P = 0.02). Similar risk patterns were also observed in an external cohort (Look AHEAD trial, N = 4211) with the highest risk of adverse outcome seen in phenogroup 1 (Fig D). Conclusion: Semi-supervised clustering using FMM efficiently identifies phenogroups of patients with DM with distinct clinical characteristics and CVD risk.

A hybrid modified step Whale Optimization Algorithm with Tabu Search for data clustering

Clustering is a popular data analysis tool that groups similar data objects into disjoint clusters. Clustering algorithms have major drawbacks, such as: sensitivity to initialization and getting trapped in local optima. Swarm Intelligence (SI) have been combined with clustering techniques to enhance their performance. Despite their improvements, some SI methods, such as Particle Swarm Optimization (PSO) and its rivals, had their drawbacks, such as low convergence rates and generating low quality solutions. This is due to storing a single best solution about solution space, which can lead swarm members to local optima. Whale Optimization Algorithm (WOA) is a recent SI method, which has been proved as a global optimizer over multiple optimization problems. Unlikely, WOA uses the same concept as PSO, it uses the best solution to reposition swarm members. To overcome this problem, WOA needs to store multiple best solutions about solution space. Tabu Search (TS) is a meta-heuristic method, which uses memory components to explore and exploit search space. In this paper, we propose combining WOA with TS (WOATS) for data clustering. To assess the efficiency of WOATS, it has been applied in data clustering. WOATS uses an objective function inspired by partitional clustering to maintain the quality of clustering solutions. WOATS was tested over multiple real life datasets, which their sizes vary from small to large-sized datasets. WOATS was able to find centers with high quality in a small number of iterations regardless of the size of the dataset, which clarifies its ability to cover the solution space efficiently. The generated clusters were evaluated according to different outlier criteria: Davies-Bouldin Index and Dunn Index. According to these criteria, WOATS presented its superiority over multiple recent original and hybrid SI methods for different datasets. Results obtained by WOATS were better than the results of the other methods with big differences regarding to the different outlier criteria, which ensured the efficiency of WOATS.

Development of new seed with modified validity measures for k-means clustering

Conventional k-means clustering is the widely used partitional method, mainly adapted to machine learning and pattern recognition problems. This algorithm is highly sensitive to initial centroid points, but it cannot guarantee to arrive at a better solution because initial centroids are computed randomly for the given cluster. In this paper, we have developed a new initialization method for k-means clustering. We have also made an effort to improve the Dunn Index and introduced a new validity ratio based on the silhouette index. The sum of squared error, Dunn Index, silhouette index, modified Dunn Index, and silhouette validity ratio were used as criteria to evaluate the performance of the initialization algorithm. Various benchmark datasets have been used to assess the effectiveness of the proposed initialization algorithm, and we compared the results with conventional k-means and k-means++ algorithms. The results have shown that the sum of squared error and number of iterations obtained by our proposed initialization algorithm are minimum. A precision chart is used to test the consistency of the initialization algorithm. The comparative analysis, based on the modified Dunn Index, and silhouette validity ratio have proved that the proposed initialization algorithm has performed better than the other initialization algorithms.

Adaptation of Common Spatial Patterns based on mental fatigue for motor-imagery BCI

Common Spatial Pattern (CSP) is the most popular method in motor imagery (MI) based Brain–Computer Interfaces (BCI) for extracting features from electroencephalogram (EEG) signals. Due to the non-stationary nature of EEG signals, the CSP computed on the training data may not be optimal for the evaluation data. One of the major causes of such non-stationarity is the change in user's cognitive state due to fatigue, frustration, low arousal level, etc. This paper proposes an adaptive scheme for the CSP based on the mental fatigue of the user. The proposed method uses Linear Discriminant Analysis (LDA) active learning to adapt the CSP. Breaking ties criterion is used for selecting samples from the evaluation data. The separability of MI EEG features extracted with the proposed adaptive CSP has been compared with that of conventional CSP in terms of three separability metrics: Davies Bouldin Index (DBI), Fisher's Score (FS) and Dunn's Index (DI). Experimental results show significantly higher separability of features extracted with adaptive CSP as compared to that with conventional CSP.

Comparative Study of K-Means, Partitioning Around Medoids, Agglomerative Hierarchical, and DIANA Clustering Algorithms by Using Cancer Datasets

Clustering plays a particularly fundamental role in exploring data, creating predictions and to overcome the anomalies in the data. Clusters that contain parallel, identical characteristics in a dataset are grouped using reiterative algorithms. As the data in real world is rising day by day so the challenges of perceiving and interpreting the consequential mass of data, which often consists of millions of measurements are increased by the intricacy of a huge number of genes of biological networks. To addressing this challenge, we use clustering algorithms. In this study, we provided a comparative study of the four most popular clustering algorithms: K-Means, PAM, Agglomerative Hierarchical and DIANA and these are evaluated on eight real cancer (four Affymetrix and four cDNA) gene data and simulated data set. The comparative results based upon seven popular cluster validity indices: Average Silhouette Index, Corrected rand Index, Variation of Information, Dunn Index, Calinski-Harabasz Index, Separation Index, and Pearson Gamma. We determine that PAM is best for Affymetrix data set and DIANA is best for cDNA dataset among these four clustering algorithms. This study provides practical evaluation frameworks for accessing clustering results on gene expression cancer datasets.

Multi-dimensional data visualisation method based on convex-corrected Radviz

Radviz is one of the most commonly used multi-dimensional data visualisation methods. Because the projection points overlap a lot in Radviz, this paper puts forward a new Radviz optimisation method to correct the position of the projected data points. Firstly, the new method introduces the Prim algorithm to realise the optimal ordering of the dimension anchors on Radviz. Secondly, the convex hull mapping and the second Radviz mapping are used to correct the position of the projected data points. Finally, the data points are visualised. In addition, in order to verify the effectiveness of the algorithm, the Dunn index was used to do a quantitative evaluation of visualisation. By comparing multiple sets of data set experiments, the results show that the new method is beneficial to obtain the better visualisation effect of multi-dimensional data in Radviz projection.

Multi-dimensional data visualisation method based on convex-corrected Radviz

Radviz is one of the most commonly used multi-dimensional data visualisation methods. Because the projection points overlap a lot in Radviz, this paper puts forward a new Radviz optimisation method to correct the position of the projected data points. Firstly, the new method introduces the Prim algorithm to realise the optimal ordering of the dimension anchors on Radviz. Secondly, the convex hull mapping and the second Radviz mapping are used to correct the position of the projected data points. Finally, the data points are visualised. In addition, in order to verify the effectiveness of the algorithm, the Dunn index was used to do a quantitative evaluation of visualisation. By comparing multiple sets of data set experiments, the results show that the new method is beneficial to obtain the better visualisation effect of multi-dimensional data in Radviz projection.

Graph Similarity-based Hierarchical Clustering of Trajectory Data

Abstract Trajectory is the path traversed by any moving object like animals, human, vehicles and natural phenomenon. Trajectory analysis and clustering are essential to learn the dynamics of movement and pattern of moving objects. This paper proposes to cluster and identify similar trajectories based on paths traversed by moving object. The proposed algorithm has two phases graph generation and clustering. Moving objects generate a trace of GPS points which are converted into a graph, representing spatial regions. Graph generation methodology transforms trajectories into a series of spatial grid numbers and clustering algorithm group trajectories based on similarity measure which are calculated using edge and vertex similarity. Hierarchical clustering is done using graph based similar measures and the resulting clusters are validated using three measures namely Cophenetic Correlation Coefficient, Davies Bouldin Index and Dunn Index. Experimental analysis demonstrates the effectiveness in representation and clustering of trajectories based on graph model.

Comparação de métodos de agrupamentos hierárquicos aglomerativos em indicadores de sustentabilidade em municípios do estado do Pará

Este estudo teve como objetivo comparar o desempenho dos métodos de agrupamento hierárquico aglomerativo utilizando um conjunto de dados composto por diversos indicadores de sustentabilidade referentes aos municípios do estado do Pará. Assim como, definir a quantidade de agrupamentos iniciais a serem constituídos pela utilização dos índices de validade. Para seleção dos indicadores, foi feito um check-list de estudos científicos de abrangência nacional, regional e local que abordam a temática da sustentabilidade. Posteriormente, foi realizado a padronização dos indicadores, devido às unidades e escalas de medidas diferentes, não interferindo no resultado e possuindo pesos semelhantes no cômputo do coeficiente de similaridade. A medida de dissimilaridade empregada foi a distância euclidiana, e para determinar o método de agrupamento hierárquico foi utilizado o coeficiente aglomerativo (CA). Para estabelecer o número de agrupamento inicial foram empregados índices de validação. O método aglomerativo com melhor desempenho quanto ao (CA) foi de Ward com 0,94, indicando uma melhor força e qualidade entre as técnicas aglomerativos. Os índices de validação Davies Bouldin (DB), Dunn (D) e Silhouette (SIL), indicaram que a quantidade ideal de agrupamentos iniciais a ser formado são 2, todavia o índice PBM constatou que a formação ideal é com 4 grupos. Com relação aos municípios maior homogeneidade, verificou-se que na composição com 2 grupos, as observações mais similares foram m105(Salinópolis) e m109(Santa Izabel do Pará), seguido das observações m102 (Rio Maria) e m144 (Xinguara), todas inseridas no grupo 1.

Autoantibody clustering of lupus-associated pulmonary hypertension

ObjectiveTo define the SLE phenotype associated with pulmonary hypertension using multiple autoantibodies.Methods207 (8%) patients with SLE with pulmonary hypertension, defined as a right ventricular systolic pressure greater than 40 mm...

ANALISIS KLASTER METODE WARD DAN AVERAGE LINKAGE DENGAN VALIDASI DUNN INDEX DAN KOEFISIEN KORELASI COPHENETIC (Studi Kasus: Kecelakaan Lalu Lintas Berdasarkan Jenis Kendaraan Tiap Kabupaten/Kota di Jawa Tengah Tahun 2018)

Based on Central Java Regional Police data, traffic accidents from 2017 to 2018 increased from 17.522 to 19.016 or 8,54 percent. To reduce the number of traffic accidents in Central Java, the initial step was carried out by grouping districts/cities that had the same accident level characteristics based on vehicle type with cluster analysis. The ward and average linkage method is a hierarchical cluster analysis method. ward method can maximize cluster homogeneity. While the average linkage method can generate clusters with small cluster variants. In this study using a measure of squared euclidean distance to measure the similarity between pairs of objects. To determine the quality of clustering results, the validation dunn index and cophenetic coefficients corelation are used. Based on the results of the clustering, the optimal number of clusters is obtained at q = 5 for the average linkage method with the results of validation dunn index = 0,08571196 and the rcoph = 0,687458. Keywords: Accidents, Cluster Analysis, Ward Method, Average linkage, Squared Euclidean Distance, Dunn Index, Cophenetic Correlation Coefficient

A Novel Bio-Inspired Method for Early Diagnosis of Breast Cancer through Mammographic Image Analysis

Breast cancer is a current problem that causes the death of many women. In this work, we test meta-heuristics applied to the segmentation of mammographic images. Traditionally, the application of these algorithms has a direct relationship with optimization problems; however, in this study, its implementation is oriented to the segmentation of mammograms using the Dunn index as an optimization function, and the grey levels to represent each individual. The update of grey levels during the process results in the maximization of the Dunn’s index function; the higher the index, the better the segmentation will be. The results showed a lower error rate using these meta-heuristics for segmentation compared to a well-adopted classical approach known as the Otsu method.