Comparison of Hierarchic Clustering Methods with Cophenetic Correlation Coefficient in Big Data

Abstract

The aim of this study is to compare hierarchical clustering methods by Cophenetic Correlation Coefficient (CCC) when there is a big data. For this purpose, after giving information about big data, clustering methods and CCC, analyzes are carried out for the related data set. The 2015 air travel consumer report, which was used in the application part of the study and published by the US Ministry of Transport, was used as big data. Libraries of the Python programming language installed on the Amazon cloud server, which includes open-source big data technologies, were used for data analysis. Since there is big data in the study, in order to save time and economy, the variables used in the study were first reduced by feature selection method, standardized and analyzed over the final 4 different data sets. As a result of the clustering analysis, it was observed that the highest CCC was obtained with the Average clustering method for all of these four different data sets.