A feasible MapReduce peer-to-peer framework for distributed computing applications

Abstract

This article presents a MapReduce Peer-to-Peer (P2P) framework that enables a MapReduce implementation on P2P networks to support a class of MapReduce-based computing applications. This framework can be useful for researchers who cannot afford expensive and dedicated clusters for infrequent demands of solving distributed computing problems. The framework also allows Internet users from social and P2P network communities to perform large data processing experiments on distributed environment. The article describes the architecture and prototyping implementation of a MapReduce P2P system. The main features of this system include exploiting leisure computing resources on P2P networks efficiently for computation, providing MapReduce operations using task management for various distributed computing problems, and supporting peer failure management for improving fault tolerance on Internet environment. We have evaluated and compared the MapReduce P2P implementation with a Hadoop MapReduce implementation on local and global-scale networks. The article also includes a discussion of applying the framework to a realistic distributed case-based reasoning system.