RACAM: design and implementation of a recursively adjusting co‐allocation method with efficient replica selection in Data Grids

Abstract

AbstractData Grids enable the sharing, selection, and connection of a wide variety of geographically distributed computational and storage resources for addressing large‐scale data‐intensive scientific application needs in, for instance, high‐energy physics, bioinformatics, and virtual astrophysical observatories. Data sets are replicated in Data Grids and distributed among multiple sites. Unfortunately, data sets of interest sometimes are significantly large in size, and may cause access efficiency overhead. A co‐allocation architecture was developed in order to enable parallel downloading of data sets from multiple servers. Several co‐allocation strategies have been coupled and used to exploit download rate by specifying among various client–server divides files into multiple blocks of equal sizes to link and address dynamic rate fluctuations. However, one major obstacle, the idle time of faster servers having to wait for the slowest server to deliver the final block, makes it important to reduce differences in finishing time among replica servers. In this paper, we propose a dynamic co‐allocation method, called Recursively Adjusting Co‐Allocation Method (RACAM), to improve the performance of parallel data file transfer. Our approach reduces the idle time spent waiting for the slowest server and decreases data transfer completion time. We also provide an effective scheme for reducing the cost of reassembling data blocks. Copyright © 2010 John Wiley & Sons, Ltd.