Service-Oriented Collaborative Framework for High-Performance Data Transfer in Grids

Abstract

As the number of data-intensive applications increases in various domains, scientists need to save, retrieve, and analyze increasingly large datasets. The huge volume of data and the long latency of data transfer on the Internet make it very difficult to ensure high-performance access to Data Grids. Thus, data replication techniques have been widely adopted to solve the latency problem. In this paper, we propose an efficient data replication algorithm for multi-source data transfer, whereby a data replica can be assembled in parallel from multiple distributed data sources and adapted to the fluctuation of network bandwidths. The experimental results show that the proposed algorithm can obtain more aggregated bandwidth, reduce connection overheads, and achieve superior load balance. We also develop a Service-oriented Collaborative Framework that not only implements the proposed data replication algorithm in a real Grid system, but also provides a virtual data management environment that integrates distributed and heterogeneous storage systems and enables users to share their data with group members for collaborative work.