Data Transfer Completion Time Research Articles

QoS provisioning for various types of deadline-constrained bulk data transfers between data centers

An increasing number of applications in scientific and other domains have moved or are in active transition to clouds, and the demand for big data transfers between geographically distributed cloud-based data centers is rapidly growing. Many modern backbone networks leverage logically centralized controllers based on software-defined networking (SDN) to provide advance bandwidth reservation for data transfer requests. How to fully utilize the bandwidth resources of the links connecting data centers with guaranteed quality of service for each user request is an important problem for cloud service providers. Most existing work focuses on bandwidth scheduling for a single request for data transfer or multiple requests using the same service model. In this work, we construct rigorous cost models to quantify user satisfaction degree, and formulate a generic problem of bandwidth scheduling for multiple deadline-constrained data transfer requests of different types to maximize the request scheduling success ratio while minimizing the data transfer completion time of each request. We prove this problem to be not only NP-complete but also non-approximable, and hence design a heuristic algorithm. For performance evaluation, we establish a proof-of-concept emulated SDN testbed and also generate large-scale simulation networks. Both experimental and simulation results show that the proposed scheduling scheme significantly outperforms existing methods in terms of user satisfaction degree and scheduling success ratio.

Optimizing Trade-Off Between Cost and Performance of Data Transfers Using Bandwidth Reservation in Dedicated Networks

Because of the solid performance of providing quality of service for various applications for decades, bandwidth reservation has been increasingly used in recent years for large amounts of data transfer to achieve guaranteed performance. However, effective scheduling strategy to achieve the trade-off between data transfer cost and data transfer performance still remains to be investigated. In this paper, we focus on the trade-off between cost and the most common performance parameter, i.e., completion time, of data transfers using bandwidth reservation in dedicated networks. We consider the scheduling of two types of bandwidth reservation requests regarding such trade-off: (1) to achieve the minimum data transfer cost given the data transfer deadline, and (2) to achieve the earliest data transfer completion time given the maximum data transfer cost. We propose two bandwidth reservation algorithms with rigorous optimality proofs to optimize the scheduling of these two types of bandwidth reservation requests. We then compare the proposed algorithms with two scheduling algorithms originating from one widely used scheduling algorithm in production networks, and the efficacy of the proposed optimal algorithms is verified through extensive simulations.

Fault-tolerant bandwidth reservation strategies for data transfers in high-performance networks

Many next-generation e-science applications require fast and reliable transfer of large volumes of data with guaranteed performance, which is typically enabled by the bandwidth reservation service in high-performance networks. One prominent issue in such network environments with large footprints is that node and link failures are inevitable, hence potentially degrading the quality of data transfer. We consider two generic types of bandwidth reservation requests (BRRs) concerning data transfer reliability: (i) to achieve the highest data transfer reliability under a given data transfer deadline, and (ii) to achieve the earliest data transfer completion time while satisfying a given data transfer reliability requirement. We propose two periodic bandwidth reservation algorithms with rigorous optimality proofs to optimize the scheduling of individual BRRs within BRR batches. The efficacy of the proposed algorithms is illustrated through extensive simulations in comparison with scheduling algorithms widely adopted in production networks in terms of various performance metrics.

Managing Flow Transfers in Enterprise Datacenter Networks with Flow Chasing

In this paper, we study how to optimize the data shuffle phase by leveraging the flow relationship in datacenter networks (DCNs). In most of the clustering computer frameworks, the completion of a transfer (a group of flows that can enable a computation stage to start or complete) is determined by the flow completing last, so that limiting the rate of other flows (not the last one) appropriately can save bandwidth without impacting the performance of any transfer. Furthermore, for the flows enter network late, more bandwidth can be assigned to them to accelerate the completion of the entire transfer. Based on these characteristics, we propose the flow chasing algorithm (FCA) to optimize the completion of the entire transfer. We implement FCA on a real testbed. By evaluation, we find that FCA can not only reduce the completion time of data transfer by 6.24% on average, but also accelerate the completion of data shuffle phase and entire job.

A fuzzy-based intelligent vertical handoff decision strategy with maximised user satisfaction for next generation communication networks

A vast diversity of services over the available radio access networks is assured by the next generation wireless systems for consumers with multi-homed cellular devices. Many consumer centric economic surplus-based vertical handoff schemes were already proposed intending to profit the end user. Few of these schemes estimate consumer surplus value with respect to the data transfer completion time assessed against the time constraints agreed by the user. In this paper, we suggested a vertical handoff decision strategy which promises maximised user satisfaction for consumer surplus value-based pricing schemes for non real time mobile-based multi-media services. A fuzzy rule-based system supported by multi criteria decision model is proposed to find optimum network between consumer surplus value and file download culmination time with the objective of maximising user satisfaction. Simulation is carried out using OM Net++ and MATLAB and the results are acknowledged with discussion and analysis.

RACAM: design and implementation of a recursively adjusting co‐allocation method with efficient replica selection in Data Grids

AbstractData Grids enable the sharing, selection, and connection of a wide variety of geographically distributed computational and storage resources for addressing large‐scale data‐intensive scientific application needs in, for instance, high‐energy physics, bioinformatics, and virtual astrophysical observatories. Data sets are replicated in Data Grids and distributed among multiple sites. Unfortunately, data sets of interest sometimes are significantly large in size, and may cause access efficiency overhead. A co‐allocation architecture was developed in order to enable parallel downloading of data sets from multiple servers. Several co‐allocation strategies have been coupled and used to exploit download rate by specifying among various client–server divides files into multiple blocks of equal sizes to link and address dynamic rate fluctuations. However, one major obstacle, the idle time of faster servers having to wait for the slowest server to deliver the final block, makes it important to reduce differences in finishing time among replica servers. In this paper, we propose a dynamic co‐allocation method, called Recursively Adjusting Co‐Allocation Method (RACAM), to improve the performance of parallel data file transfer. Our approach reduces the idle time spent waiting for the slowest server and decreases data transfer completion time. We also provide an effective scheme for reducing the cost of reassembling data blocks. Copyright © 2010 John Wiley & Sons, Ltd.

A Framework for an Integrated Co-allocator for Data Grid in Multi-Sender Environment

Data grid consists of scattered computing and storage resources located dispersedly in the grid network. These large sized data sets are replicated in more than one site for the better availability to the other nodes in a grid. Downloading the dataset from these replicated locations have practical difficulties and we find interest in a co-allocated download framework, which enables parallel download of replicated data from multiple servers. In this paper, we proposed a dynamic co-allocation scheme for parallel data transfer in grid environment, which copes up with highly inconsistent network and server performance. The model comprises of co-allocator, monitor and control mechanisms. The scheme initially obtains the bandwidth parameter from the monitor module to fix the partition size and the data transfer tasks are allocated onto the servers in duplication. In this way, the process of data transfer can neither be interrupted nor paralyzed, even when the network link is broken or server crash. We used Globus toolkit for our framework by making use of grid information and GridFTP services. We compared our scheme with the existing schemes and the results show notable improvement in overall completion time of data transfer.

Improvements on dynamic adjustment mechanism in co-allocation data grid environments

Several co-allocation strategies have been coupled and used to exploit rate differences among various client-server links and to address dynamic rate fluctuations by dividing files into multiple blocks of equal sizes. However, a major obstacle, the idle time of faster servers having to wait for the slowest server to deliver the final block, makes it important to reduce differences in finishing time among replica servers. In this paper, we propose a dynamic co-allocation scheme, namely Recursive-Adjustment Co-Allocation scheme, to improve the performance of data transfer in Data Grids. Our approach reduces the idle time spent waiting for the slowest server and decreases data transfer completion time.