Bulk Data Transfer Research Articles

Integrated availability model based on performance of computer networks

An efficient computer network can realize bulk data transfer and real time control with high reliability and precision. Its availability is dependent on its topological structure and network services performed simultaneously. To explore the dynamic interactions between software and hardware failures, generalized stochastic petri net (GSPN) is adopted to model the availability of nodes in the network, and then the study is developed through path selection and service performance evaluation. Network performance indices such as throughput and time delay can be obtained by modeling the network service with queueing theory. With Power formula, this paper presents an integrated availability model. This model considers both topological connectivity and operational performance, developing the availability analysis from the viewpoint of a user.

Rethinking end-to-end failover with transport layer multihoming

Using the application of bulk data transfer, we investigate end-to-end failover mechanisms and thresholds for transport protocols that support multihoming (e.g.,Sctp). First, we evaluate temporary failovers, and measure the tradeoff between aggressive (i.e., lower) thresholds and spurious failovers. We surprisingly find that spurious failovers do not degrade performance, and often actually improve goodput regardless of the paths ’ characteristics (bandwidth, delay, and loss rate). A permanent failover mechanism tries to avoid throttling the sending rate by not returning to a primary path when it recovers. We demonstrate that such a mechanism can be beneficial if the sender can estimate each path’s rtt and loss rate. We advocate a new approach to end-to-end failover that temporarily redirects traffic to an alternate path on the first sign of a potential failure (i.e., a timeout) on the primary path, but conservatively proceeds with failure detection of the primary path in the background.

Adaptive data dissemination schemes for location-aware mobile services

Broadcasting is the natural method of propagating information in wireless links, which guarantees scalability in the case of bulk data transfers. It is particularly attractive for resource limited mobile clients in asymmetric communications. To facilitate power saving via wireless data broadcast, index information is typically broadcast along with the data. By first accessing the broadcast index, the mobile client is able to predict the arrival time of the desired data. However, it suffers from the drawback that the client has to wait and tune for an index segment, in order to conserve battery power consumption. In location-aware mobile services (LAMSs), it is important to reduce the query response time, since a late query response may contain out-of-date information. This paper proposes a new broadcast-based spatial query processing method, called BBS designed to support NN query processing. In the BBS, broadcasted data objects are sorted sequentially based on their locations, and the server broadcasts the location dependent data along with an index segment. In this method, since the data objects broadcasted by the server are sequentially ordered based on their location, it is not necessary for the client to wait for an index segment, if it has already identified the desired data items before the associated index segment has arrived. The performance of this scheme is investigated in relation to various environmental variables, such as the distributions of the data objects, the average speed of the clients and the size of the service area.

TCP in wireless environments: problems and solutions

The Internet provides a platform for rapid and timely information exchange among a disparate array of clients and servers. TCP and IP are separately designed and closely tied protocols that define the rules of communication between end hosts, and are the most commonly used protocol suite for data transfer in the Internet. The combination of TCP/IP dominates today's communication in various networks from the wired backbone to the heterogeneous network due to its remarkable simplicity and reliability, TCP has become the de facto standard used in most applications ranging from interactive sessions such as Telnet and HTTP, to bulk data transfer like FTP. TCP was originally designed primarily for wired networks. In a wired network, random bit error rate, a characteristic usually more pronounced in the wireless network, is negligible, and congestion is the main cause of packet loss. The emerging wireless applications, especially high-speed multimedia services and the advent of wireless IP communications carried by the Internet, call for calibration and sophisticated enhancement or modifications of this protocol suite for improved performance. Based on the assumption that packet losses are signals of network congestion, the additive increase multiplicative decrease congestion control of the standard TCP protocol reaches the steady state, which reflects the protocol's efficiency in terms of throughput and link utilization. However, this assumption does not hold when the end-to-end path also includes wireless links. Factors such as high BER, unstable channel characteristics, and user mobility may all contribute to packet losses. Many studies have shown that the unmodified standard TCP performs poorly in a wireless environment due to its inability to distinguish packet losses caused by network congestion from those attributed to transmission errors. In this article, following a brief introduction to TCP, we analyze the problems TCP exhibits in the wireless IP communication environment, and illustrate viable solutions by detailed examples.

Native 10 Gigabit Ethernet experiments over long distances

The current solutions for transmitting data over Wide Area Networks (WANs) are expensive and require protocol translation at layer 1. The IEEE recently standardized the 10 Gigabit Ethernet (10 GE) WAN PHY as a native gateway from the Local Area Networks (LAN) to the WAN. This opened a debate as to whether Ethernet is now a valid alternative to Synchronous Optical Network/Synchronous Digital Hierarchy (SONET/SDH) for WANs. In this article, we report on the experience gathered while building the first trans-European native 10 Gigabit Ethernet testbed based on WAN PHY. We describe and analyze network tests with a 1700 km Ethernet network. Our work validates this approach and indicates that Ethernet can offer a large bandwidth to long-distance bulk data transfers at a trans-European level.

Efficient rate-controlled bulk data transfer using multiple multicast groups

Controlling the rate of bulk data multicast to a large number of receivers is difficult, due to the heterogeneity among the end systems' capabilities and their available network bandwidth. If the data transfer rate is too high, some receivers will lose data, and retransmissions will be required. If the data transfer rate is too slow, an inordinate amount of time will be required to transfer the data. In this paper, we examine an approach toward rate-controlled multicast of bulk data in which the sender uses multiple multicast groups to transmit data at different rates to different subgroups of receivers. We present simple algorithms for determining the transmission rate associated with each multicast channel, based on static resource constraints, e.g., network bandwidth bottlenecks. Transmission rates are chosen so as to minimize the average time needed to transfer data to all receivers. Analysis and simulation are used to show that our policies for rate selection perform well for large and diverse receiver groups and make efficient use of network bandwidth. Moreover, we find that only a small number of multicast groups are needed to reap most of the possible performance benefits.

Scalable Bulk Data Transfer in Wide Area Networks

Bulk data transfer in wide area networks (WAN) requires scalable and high network bandwidth. In this paper, we identify a number of the scalability limitations that affect the full utilization of peak theoretical network bandwidth. In addition, we study and classify different offered approaches to overcome some of the identified limitations and increase network bandwidth among Grid components in WAN. With these limitations in mind, we study and evaluate the scalability and flexibility of a UDP-based multiple-network-interface socket (MuniSocket) model in WAN. The MuniSocket model is a middleware layer between the distributed applications and the multiple networks and system resources available. MuniSocket utilizes existing system resources, network interface cards, and network links to provide a scalable, reliable and high bandwidth network solution for data-intensive distributed applications, thus eliminating most of the identified limitations.

ATLAS Canada lightpath data transfer trial

Emerging grids will play a significant role in the computational, data, storage, and network requirements of High Energy Physics experiments coming online in the next few years. One such requirement, the bulk transfer of data over advanced high speed optical networks is necessary as such experiments are highly distributed with resources and participants from research laboratories and institutions spanning the globe. This trial at the iGrid 2002 conference attempts to stress the feasibility of high speed bulk data transfer over an end-to-end lightpath, a dedicated point-to-point optical link. Specifically, the objective was to transfer 1 TB of Monte Carlo data from TRIUMF in Vancouver, Canada, to CERN in Geneva, Switzerland. A rate equivalent to transferring a full CD of data every 8 s was achieved.

High‐speed bulk data transfer architecture for Gbit/s throughput

AbstractWhen bulk data is transferred between end hosts over an IP network, a known problem is the limited throughput obtained at the application level due to the memory copy processing load between the kernel buffer and the user buffer depending on the operating system. To solve this problem, in this paper we propose a high‐speed bulk data transfer architecture called Network Direct Memory Access (Network DMA) and describe its design, implementation, and test results. The purpose of Network DMA is to be a transport function installed on a network interface card (NIC) and to achieve high‐speed and reliable data transfers over a network between the user buffers of the end hosts without requiring operating system‐specific processing. An application will use the data transfer function provided by a new interface capable of directly specifying the send destination host and the target transfer buffer. Based on the proposed architecture, we developed a 622‐Mbit/s ATM NIC and Gigabit Ethernet NIC incorporating the Network DMA. The ATM NIC we developed achieved a continuous transfer throughput of 540 Mbit/s (99% bandwidth usage efficiency at the application level) in data transfer tests between general‐purpose PCs. The Gigabit Ethernet NIC achieved an 800 Mbit/s continuous transfer throughput. © 2003 Wiley Periodicals, Inc. Electron Comm Jpn Pt 1, 86(11): 57–65, 2003; Published online in Wiley InterScience (www.interscience.wiley.com). DOI 10.1002/ecja.10107

Microprocessor adapter for ATM networks

This paper presents a microprocessor adapter for ATM networks. Its transmission functions, which are related to the upstream direction, include cell buffering, header error control, cell assembling, rate coupling, and information insertion. The reception functions, which are related to the downstream direction, include information extraction, rate decoupling, cell buffering, header error detection and correction, cell delineation, connection identity fields extraction and identification, cell disassembling and classification, and idle cell discarding. The transmission direction can be supported optionally by a traffic shaper, which is responsible for adapting emitted traffic to ATM network. The microprocessor adapter, which can be used in terminals or in interworking units and switches, implements basic functions of the lower layers of the ATM protocol reference model. It uses three applications specific integrated circuit (ASIC) chips. The three chip-set and other logic can been used to develop a personal computer (PC) adapter for ATM networks. This PC adapter offers bulk data transfer across the ATM network, internet applications over TCP/IP, interactive applications, LAN–ATM interworking facilities using a PC as a router, multimedia and other services.

On multicast flow control for heterogeneous receivers

In this paper, we study the impact of heterogeneous receivers on the throughput of multicast flow control and propose a new multicast flow control algorithm to optimally partition group members into multiple subgroups. Our main contributions are as follows. First, we cast the multicast flow control problem in the Internet as the list partition problem and then prove that the list partition problem is equivalent to the optimal paging problem in cellular networks. The result is not only interesting in itself but also essential to derive the first known analytical bounds for the throughput of multicast flow control. Furthermore, we propose an algorithm to solve not only the list partition problem but also the optimal paging problem and the problem of bulk data transfer using multiple multicast groups. The complexity of our algorithm is one order less than the best known algorithm designed only for the problem of bulk data transfer using multiple multicast groups in the literature. While earlier work uses simulations to justify the usage of multiple subgroups to deliver information to a large amount of receivers in heterogeneous networks, we provide the first analytical support.

Integrating Bulk-Data Transfer into the Aurora Distributed Shared Data System

The Aurora distributed shared data system implements a shared-data abstraction on distributed-memory platforms, such as clusters, using abstract data types. Aurora programs are written in C++ and instantiate shared-data objects whose data-sharing behaviour can be optimized using a novel technique called scoped behaviour. Each object and each phase of the computation (i.e., use-context) can be independently optimized with per-object and per-context flexibility. Within the scoped behaviour framework, optimizations such as bulk-data transfer can be implemented and made available to the application programmer. Scoped behaviour carries semantic information regarding the specific data-sharing pattern through various layers of software. We describe how the optimizations are integrated from the uppermost application-programmer layers down to the lowest UDP-based layers of the Aurora system. A bulk-data transfer network protocol bypasses some bottlenecks associated with TCP/IP and achieves higher performance on an ATM network than either TreadMarks (distributed shared memory) or MPICH (message passing) for matrix multiplication and parallel sorting.

ABE: providing a low-delay service within best effort

We propose alternative best effort (ABE), a novel service for IP networks, which idea of providing low-delay at the expense of maybe less throughput. The objective is to retain the simplicity of the original Internet single-class best-effort service while providing low-delay to interactive adaptive applications. With ABE, every best effort packet is marked as either green or blue. Green packets are guaranteed a low bounded delay in every router. In exchange, green packets are more likely to be dropped (or marked using congestion notification) during periods of congestion than blue packets. For every packet, the choice of color is made by the application based on the nature of its traffic and on global traffic conditions. Typically, an interactive application with real-time deadlines, such as audio, will mark most at its packets as green, as long as the network conditions offer large enough throughput. In contrast, an application that transfers binary data such as bulk data transfer will seek to minimize overall transfer time and send blue traffic. We propose router requirements that aim at enforcing benefits for all types of traffic, namely that green traffic achieves low-delay and blue traffic receives at least as much throughput as it would in a flat (legacy) best effort network. ABE is different from differentiated or integrated services in that neither packet color can be said to receive better treatment; thus, flat rate pricing may be maintained, and there is no need for reservations or profiles. We define the ABE service, its requirements, properties, and usage. We discuss the implications of replacing the existing IP best effort service by the ABE service. We propose and analyze an implementation based on a new scheduling method called duplicate scheduling with deadlines. It supports any mixture of TCP, TCP-friendly, and non-TCP-friendly traffic.

Energy efficiency of TCP in a local wireless environment

The focus of this paper is to analyze the energy consumption performance of various versions of TCP, namely, Tahoe, Reno and NewReno, for bulk data transfer in an environment where channel errors are correlated. We investigate the performance of a single wireless TCP connection by modeling the correlated packet loss/error process (e.g., as induced by a multipath fading channel) as a first-order Markov chain. Based on a unified analytical approach, we compute the throughput and energy performance of various versions of TCP. The main findings of this study are the (1) error correlations significantly affect the energy performance of TCP (consistent with analogous conclusions for throughput), and in particular they result in considerably better performance for Tahoe and NewReno than iid errors, and (2) the congestion control mechanism implemented by TCP does a good job at saving energy as well, by backing off and idiling during error bursts. An interesting conclusion is that, unlike throughput, the energy efficiency metric may be very sensitive to the TCP version used and to the choice of the protocol parameters, so that large gains appear possible.

The Eifel retransmission timer

We analyze two alternative retransmission timers for the Transmission Control Protocol (TCP). We first study the retransmission timer of TCP-Lite which is considered to be the current de facto standard for TCP implementations. After revealing four major problems of TCP-Lite's retransmission timer, we propose a new timer, named the Eifel retransmission timer , that eliminates these. The strength of our work lies in its hybrid analysis methodology. We develop models of both retransmission timers for the class of network-limited TCP bulk data transfers in steady state. Using those models, we predict the problems of TCP-Lite's retransmission timer and develop the Eifel retransmission timer. We then validate our model-based analysis through measurements in a real network that yield the same results.

Scheduling and admission control for integrated-services networks: the Priority Token Bank

This paper proposes a new mechanism called the Priority Token Bank for admission control, scheduling and policing in integrated-services networks. In such networks, both arrival processes and performance objectives can vary greatly from one packet stream to another. There are two principal components to the Priority Token Bank: accepting or rejecting requests to admit entire packet streams, where acceptance means guaranteeing that the packet stream's performance objectives will be met, and scheduling the transmission of packets such that performance objectives are met, even under heavy loads. To the extent possible, the performance of traffic is also optimized beyond the requirements. The performance achieved with the Priority Token Bank is compared to that of other typical algorithms. It is shown that, when operating under the constraint that the performance objectives of applications such as packet voice, video and bulk data transfer must be met in an ATM network, the mean delay experienced by other traffic is much better with the Priority Token Bank. Furthermore, the admission control algorithm can guarantee requirements will be met, and admit more traffic than the common alternatives.

The implementation of a high-performance ORB over multiple network transports

This paper describes the implementation of a high-performance Object Request Broker (ORB) - omniORB2. The discussion focuses on the experience in achieving high performance by exploiting the protocol and other characteristics of the CORBA 2.0 specification. The design is also highly adaptable to a variety of network transports. The results of running the ORB over TCP/IP, shared memory, Scalable Coherent Interface (SCI) and ATM Adaptation Layer 5 (AAL5) are presented. In both null calls and bulk data transfers, the performance of omniORB2 is significantly better than other commercial ORBs.

Performance-based path determination for interprocessor communication in distributed computing systems

The different types of messages used by a parallel application program executing in a distributed computing system can each have unique characteristics so that no single communication network can produce the lowest latency for all messages. For instance, short control messages may be sent with the lowest overhead on one type of network, such as Ethernet, while bulk data transfers may be better suited to a different type of network, such as Fibre Channel or HIPPI. This work investigates how to exploit multiple heterogeneous communication networks that interconnect the same set of processing nodes using a set of techniques we call performance-based path determination (PBPD). The performance-based path selection (PBPS) technique selects the best (lowest latency) network among several for each individual message to reduce the communication overhead of parallel programs. The performance-based path aggregation (PBPA) technique, on the other hand, aggregates multiple networks into a single virtual network to increase the available bandwidth. We test the PBPD techniques on a cluster of SGI multiprocessors interconnected with Ethernet, Fibre Channel, and HiPPI networks using a custom communication library built on top of the TCP/IP protocol layers. We find that PBPS can reduce communication overhead in applications compared to using either network alone, while aggregating networks into a single virtual network can reduce communication latency for bandwidth-limited applications. The performance of the PBPD techniques depends on the mix of message sizes in the application program and the relative overheads of the networks, as demonstrated in our analytical models.

Combining compile-time and run-time support for efficient software distributed shared memory

We describe an integrated compile time and run time system for efficient shared memory parallel computing on distributed memory machines. The combined system presents the user with a shared memory programming model. The run time system implements a consistent shared memory abstraction using memory access detection and automatic data caching. The compiler improves the efficiency of the shared memory implementation by directing the run time system to exploit the message passing capabilities of the underlying hardware. To do so, the compiler analyzes shared memory accesses and transforms the code to insert calls to the run time system that provide it with the access information computed by the compiler. The run time system is augmented with the appropriate entry points to use this information to implement bulk data transfer and to reduce the overhead of run time consistency maintenance. In those cases where the compiler analysis succeeds for the entire program, we demonstrate that the combined system achieves performance comparable to that produced by compilers that directly target message passing. If the compiler analysis is successful only for parts of the program, for instance, because of irregular accesses to some of the arrays, the resulting optimizations can be applied to those parts for which the analysis succeeds. If the compiler analysis fails entirely, we rely on the run time maintenance of shared memory and thereby avoid the complexity and the limitations of compilers that directly target message passing. The result is a single system that combines efficient support for both regular and irregular memory access patterns.

Comparative performance analysis of versions of TCP in a local network with a mobile radio link

The scenario is that a bulk data transfer is being performed over a TCP connection, from a host on a local area network (LAN) to a mobile host attached to the LAN by a radio link. In an earlier work we had assumed that packet losses in a TCP connection over a radio link are statistically independent. In this paper, we extend this analysis to a Rayleigh fading link, which we model by a two-state Markov model. The bulk throughputs of TCP-OldTahoe and TCP-Tahoe are compared with and without fading, for various average signal-to-noise ratios. We also study the performance with a link protocol on the wireless link, and study the effect of varying the link packet size, the number of link packet attempts, and the vehicle speed. For the parameters of the BSD UNIX implementation, over a 1.5 Mbps wireless link, we find that, with fading, a signal-to-noise ratio of at least 30 dB is required to get reasonable throughput with TCP Tahoe or OldTahoe; this corresponds to at least 100 times more power than is needed without fading.For fixed signal-to-noise ratio, as the vehicle speed varies there are roughly 3 regions of performance: at very low speeds (pedestrian speeds) the throughput is very good; at low vehicular speeds the throughput deteriorates, and again becomes very good at higher vehicle speeds. The speeds corresponding to the various regions depend on the parameters of the link protocol.