Efficient Parallel Clustering Spectral Algorithm Based on Hadoop

Abstract

This paper presents an efficient parallel spectral clustering algorithm. First, the \( kd \)-tree technique was introduced to sparse the similarity matrix. Then, when calculating the eigenvectors, the Laplacian matrix was stored in the Hadoop file system, and the eigenvectors were obtained by the distributed Lanczos operation. Finally, efficient parallel k-means clustering was used to process the transposed matrix of eigenvectors to obtain clustering results. By using different parallel strategies for each step of the algorithm, the entire algorithm achieves a linear increase in speed. Experiments show that the clustering speed increases almost linearly with the expansion of the processing data size. The proposed parallel spectral clustering algorithm is suitable for massive data mining.