USING PREGEL TO CALCULATE PAGE RANK OF A WEBPAGE A BULK SYNCHRONOUS PARALLEL PROGRAMMING APPROACH

Abstract

Although using graphs to represent networks and relationship is not new; the size of network has been dramatically increasing in the past decade so storing the whole graph in one place is almost impossible. Problems arise when processing very large graphs, when visiting billions of highly connected vertices. In such cases a graph can’t fit on a single machine, and the implementation resorts to a big batch distributed over a cluster of machines. The graph needs to be broken into multiple partitions and stored at various locations. This resulted in the need for a framework that can work in a Distributed Environment. Also, by breaking the graph into different partitions, we can manipulate the graph in parallel to speed up the processing. Google Pregel provides a simple straightforward solution to the large-scale graph processing problems. While it sounds similar to MapReduce, Pregel is optimized for graph operations by reducing I/O, ensuring data locality, but also preserving processing state between phases. The paper will give an insight of the Pregel approach for large scale graph processing. The paper will give an overview of PREGEL’s architecture and then will explore use of Pregel to solve real time applications such as finding PageRank of a Webpage. The paper will also give an insight on Bulk Synchronous Programming and will showcase how it increases computation speed with just few simple lines of code.