LI-MR: A Local Iteration Map/Reduce Model and Its Application to Mine Community Structure in Large-Scale Networks

Abstract

The analysis of large-scale networks requires the parallel techniques of graph processing. Hadoop as an open-source version of Map/Reduce implementation gains its popularity by high efficiency, scalability and fault tolerance. However, Map/Redeuce as a simplified programming model tends to be used in applications with massive datasets and simple processing. In this paper, we aim to adapt Map/Reduce programming in more complex applications such as community detection in large-scale networks. We present a new model, LI-MR (Local Iteration Map/Reduce), to resolve the Map/Reduce model's problems in respect of multi-iteration and random data access. A new system called LI-Hadoop is built to implement LI-MR model based on Hadoop. Furthermore, we propose a new algorithm MR-LPA, which parallelizes LPA in order to mine community structure in large-scale networks. We evaluate the performance of LI-Hadoop by executing MR-LPA on real-world datasets. The experimental results show that our approach is both effective and efficient.