Wide-area Data Transfers Research Articles

Reliable Wide-Area Data Transfers for Streaming Workflows

Many large science projects rely on remote clusters for (near) real-time data processing, thus they demand reliable wide-area data transfer performance for smooth end-to-end workflow executions. However, data transfers are often exposed to performance variations due to the changing network (e.g., background traffic) and dataset (e.g., average file size) conditions, necessitating adaptive solutions to meet stringent performance requirements of delay-sensitive streaming workflows. In this article, we propose <i>FStream++</i> to provide reliable transfer performance for large streaming science applications by dynamically adjusting transfer settings to adapt to changing transfer conditions. <i>FStream++</i> combines three optimization methods as <i>dynamic tuning</i> , <i>online profiling</i> , and <i>historical analysis</i> to swiftly and accurately discover optimal transfer settings that can meet workflow requirements. Dynamic tuning uses a heuristic model to predict the values of transfer parameters based on dataset characteristics and network settings. Since heuristic models fall short to incorporate many important factors such as I/O throughput and resource interference, we complement it with online profiling to execute a real-time search for a subset of transfer settings. Finally, historical analysis takes advantage of the long-running nature of streaming workflows by storing and analyzing previous performance observations to shorten the execution time of online profiling. We evaluate the performance of <i>FStream++</i> by transferring several synthetic and real-world workloads in high-performance production networks and show that it offers up to <inline-formula><tex-math notation="LaTeX">$3.6x$</tex-math></inline-formula> performance improvement over legacy transfer applications and up to 24% over our previous work <i>FStream</i> .

DLS: a cloud-hosted data caching and prefetching service for distributed metadata access

Due to the diverse approaches and characteristics of scientific research, valuable resources within most important scientific disciplines exist in heterogeneous storage systems in distributed environments as they are often created and maintained by different information providers. Considering more than half of all data access operations in traditional storage systems are metadata access operations, it is important to design a framework that can effectively query and access valuable metadata information in distributed environments. In this paper, we present a highly efficient caching and prefetching mechanism tailored to reduce metadata access latency and improve responsiveness in wide-area data transfers. We designed dynamically provisioned parallel TCP streams and non-blocking concurrent in-memory cache for the system performance. We have implemented these mechanisms in a cloud-hosted metadata retrieval, caching, and prefetching system called directory listing service (DLS) and have evaluated its performance on both local area and wide area settings.

Prediction of Optimal Parallelism Level in Wide Area Data Transfers

Wide area data transfer may be a major bottleneck for the end-to-end performance of distributed applications. A practical way of increasing the wide area throughput at the application layer is using multiple parallel streams. Although increased number of parallel streams may yield much better performance than using a single stream, overwhelming the network by opening too many streams may have an inverse effect. The congestion created by excess number of streams may cause a drop down in the throughput achieved. Hence, it is important to decide on the optimal number of streams without congesting the network. Predicting this "optimum” number is not straightforward, since it depends on many parameters specific to each individual transfer. Generic models that try to predict this number either rely too much on historical information or fail to achieve accurate predictions. In this paper, we present a set of new models which aim to approximate the optimal number with least history information and lowest prediction overhead. An algorithm is introduced to select the best combination of historic information to do the prediction for evaluation purposes as well as optimizing prediction by reducing error rate. We measure the feasibility and accuracy of the proposed prediction models by comparing to actual GridFTP data transfer by using little historical information and have seen that we could predict the throughput of parallel streams accurately and find a very close approximation of the optimal stream number.