Towards efficient data distribution on computational desktop grids with BitTorrent

Abstract

Datacentric applications are still a challenging issue for large-scale distributed computing systems. The emergence of new protocols and software for collaborative content distribution over the Internet offers a new opportunity for efficient and fast delivery of a high volume of data. This paper presents an evaluation of the BitTorrent protocol for computational desktop grids. We first present a prototype of a generic subsystem dedicated to data management and designed to serve as a building block for any desktop grid system. Based on this prototype we conduct experiments to evaluate the potential of BitTorrent compared to a classical approach based on FTP data server. The preliminary results obtained with a 65-node cluster measure the basic characteristics of BitTorrent in terms of latency and bandwidth and evaluate the scalability of BitTorrent for the delivery of large input files. Moreover, we show that BitTorrent has a considerable latency overhead compared to FTP but clearly outperforms FTP when distributing large files or files to a high number of nodes. Tests on a synthetic application show that BitTorrent significantly increases the communication/computation ratio of the applications eligible to run on a desktop grid system.