An ensemble agglomerative hierarchical clustering algorithm based on clusters clustering technique and the novel similarity measurement

Abstract

The advent of architectures such as the Internet of Things (IoT) has led to the dramatic growth of data and the production of big data. Managing this often-unlabeled data is a big challenge for the real world. Hierarchical Clustering (HC) is recognized as an efficient unsupervised approach to unlabeled data analysis. In data mining, HC is a mechanism for grouping data at different scales by creating a dendrogram. One of the most common HC methods is Agglomerative Hierarchical Clustering (AHC) in which clusters are created bottom-up. In addition, ensemble clustering approaches are used today in complex problems due to the weakness of individual clustering methods. Accordingly, we propose a clustering framework using AHC methods based on ensemble approaches, which includes the clusters clustering technique and a novel similarity measurement. The proposed algorithm is a Meta-Clustering Ensemble scheme based on Model Selection (MCEMS). MCEMS uses the bi-weighting policy to solve the model selection associated problem to improve ensemble clustering. Specifically, multiple AHC individual methods cluster the data from different aspects to form the primary clusters. According to the results of different methods, the similarity between the instances is calculated using a novel similarity measurement. The MCEMS scheme involves the creation of meta-clusters by re-clustering of primary clusters. After clusters clustering, the number of optimal clusters is determined by merging similar clusters and considering a threshold. Finally, the similarity of the instances to the meta-clusters is calculated and each instance is assigned to the meta-cluster with the highest similarity to form the final clusters. Simulations have been performed on some datasets from the UCI repository to evaluate MCEMS scheme compared to state-of-the-art algorithms. Extensive experiments clearly prove the superiority of MCEMS over HMM, DSPA and WHAC algorithms based on Wilcoxon test and Cophenetic correlation coefficient.