Distributed Multi-Objective Community Detection in Large-Scale and Complex Networks

Abstract

Community detection has arisen as an important topic of many different research areas such as sociology, biology and computer science. However, with the appearance of big data and the rapid increasing size of real-world networks, a traditional evolutionary algorithm is unable or inefficient to solve the community detection problem in large-scale networks. In this paper, a distributed multi-objective evolutionary algorithm (DMOCD) for community detection is proposed. DMOCD is implemented based on Apache Spark and Resilient Distributed Datasets. The proposed distributed framework maintains a set of evolving populations (sub-populations) which evolve separately with different crossover and mutation parameters and an external repository as an elite archive to store the non-dominated individuals. A label propagation-based initialization method, a segmented crossover and mutation tactic for large-scale network are introduced. Experiments on both artificial and real-world networks prove that the proposed method is effective for community detection problems in small-scale networks and is able to process the large-scale networks which the stand-alone multi-objective evolutionary algorithms cannot deal with.