Listing Dense Subgraphs in Small Memory

Abstract

Listing relevant patterns from graphs is becoming increasingly challenging as Web and social graphs are growing in size at a great rate. This scenario requires to process information more efficiently, including the need of processing data that cannot fit in main memory. Typical approaches for processing data using limited main memory include the streaming and external memory models. This paper addresses the problem of listing dense sub graphs from Web and social graphs using little memory. We propose an external memory algorithm based on K-way merge-sort for clustering and reordering input graphs. We also propose mining heuristics that work well with different stream orders such as URL, BFS, and cluster-based. Our experimental evaluation shows that on Web graphs, in comparison with the in-memory algorithm, the streaming mining heuristic is able to find between 70 and 96% of edges participating in dense sub graphs, uses only between 17 and 25% of the memory, and running times are between 34 and 65%. We further consider an application that uses these dense sub graphs for compressing Web graphs with a representation that enables querying the collection of sub graphs for pattern recovery and basic statistics without decompression.