Large Graph Processing Based on Remote Memory System

Abstract

This paper focuses on large graph processing based on the remote memory system. Using our remote memory system enables applications to deal with large data sets, especially graph data, which do not fit into the machines main memory. Although recent dramatic increases in DRAM capacity now allow us to build inexpensive computers with very large amounts of main memory, the rise in brand-new Internet services has resulted in rapid increases in data size. This is especially true for on-line social network services that generate various data sets that can be represented as graphs. On the other hand, high-speed networking technologies such as Infini Band, Myrinet and 10G Ethernet now enable us to transfer data with low latency and high throughput. The advanced networking technologies reduce the latency/bandwidth gap between main memory and remote memory. Thus, remote memory based processing could now be helpful in accelerating large-scale graph process when main memory space is insufficient to store application data. In this paper, we present our design and implementation of remote memory system that efficiently processes large graph data. We also evaluate a breadth-first search of various types of graphs using our system and show that our approach is good for large graph data processing.