A Robust Distributed Big Data Clustering-based on Adaptive Density Partitioning using Apache Spark

Behrooz Hosseini,Kourosh Kiani

doi:10.3390/sym10080342

Abstract

Unsupervised machine learning and knowledge discovery from large-scale datasets have recently attracted a lot of research interest. The present paper proposes a distributed big data clustering approach-based on adaptive density estimation. The proposed method is developed-based on Apache Spark framework and tested on some of the prevalent datasets. In the first step of this algorithm, the input data is divided into partitions using a Bayesian type of Locality Sensitive Hashing (LSH). Partitioning makes the processing fully parallel and much simpler by avoiding unneeded calculations. Each of the proposed algorithm steps is completely independent of the others and no serial bottleneck exists all over the clustering procedure. Locality preservation also filters out the outliers and enhances the robustness of the proposed approach. Density is defined on the basis of Ordered Weighted Averaging (OWA) distance which makes clusters more homogenous. According to the density of each node, the local density peaks will be detected adaptively. By merging the local peaks, final cluster centers will be obtained and other data points will be a member of the cluster with the nearest center. The proposed method has been implemented and compared with similar recently published researches. Cluster validity indexes achieved from the proposed method shows its superiorities in precision and noise robustness in comparison with recent researches. Comparison with similar approaches also shows superiorities of the proposed method in scalability, high performance, and low computation cost. The proposed method is a general clustering approach and it has been used in gene expression clustering as a sample of its application.

Highlights

Clustering is an unsupervised learning procedure that has a substantial usage in machine learning, data mining, and pattern recognition
The proposed method is a general clustering approach and it has been used in gene expression clustering as a sample of its application
In order to evaluate the advantages of the proposed method (DCDPS), it has been completely

Summary

Introduction

Clustering is an unsupervised learning procedure that has a substantial usage in machine learning, data mining, and pattern recognition. During the recent two decades, a large amount of literature was created by clustering methods [1,2]. Big data is the generation of computation which has opened a new horizon and obtained a hot trend of research and development in the recent years [4,5]. The conventional machine learning algorithms, including data clustering, cannot handle such tremendous volume and complexity of big data with their simple methodologies [6]. A new generation of scalable and distributed clustering algorithms are seriously needed. These algorithms should comply with the latest big data processing

Methods

Results

Conclusion