Sampling-based consensus fuzzy clustering on Big Data

Abstract

Many companies spend vast amounts of resources to collect, transform and store the massive amounts of data that flows through their business processes. When it comes to doing analysis and machine learning such as clustering on this data, time and compute speed gate determine how much data can be analyzed. Moreover, most Big Data clustering algorithms do not look at a complete, large dataset. Instead, they look at a subsample and work on approximations. However, work on samples can spread useful data that can be sources of value. In this paper, we use sampling combined with consensus strategy to dissemble the whole Big Data into small subsets, then basic partitions are locally generated from them using parallel processing. For the sampling part, we propose a partial data clustering (PDC) according to different nodes to classify the current sub-samples of partial data access (PDA) merged together with optimal prototypes generated from the last PDC and condensed into weighted points. For the consensus part, we apply a split-and-merge fuzzy clustering to equivalently transfer the consensus clustering problem into an optimization clustering one. Extensive experiments on several datasets demonstrate the ability to handle massive data and the consensus computing make the proposed classifier promising candidate for Big Data clustering.