TCP Stack Research Articles

Efficient in-memory data management

This paper analyzes the performance of three systems for in-memory data management: Memcached, Redis and the Resilient Distributed Datasets (RDD) implemented by Spark. By performing a thorough performance analysis of both analytics operations and fine-grained object operations such as set/get , we show that neither system handles efficiently both types of workloads. For Memcached and Redis the CPU and I/O performance of the TCP stack are the bottlenecks -- even when serving in-memory objects within a single server node. RDD does not support efficient get operation for random objects, due to a large startup cost of the get job. Our analysis reveals a set of features that a system must support in order to achieve efficient in-memory data management.

Empowering PlanetLab: A Method for TCP Evaluation Over the Planetary-scale Testbed

PlanetLab, the planetary-scale overlay network, has become the de facto platform for conducting network experiments under real-life realistic conditions, scale, and geographic distribution. Many tools have been developed to manage the PlanetLab resources. However, they do not provide extra experimental capabilities to this network. Currently, PlanetLab only allows experiments at the application level, as lower levels (e.g., TCP level) are shared among all users and modifications to, for example, the TCP stack will affect all users’ experiments. In this paper, we present the design of a methodology that allows researchers to implement their new TCP stack, transport algorithms, and protocols on top of PlanetLab. The approach works much like a plug-in that interacts with the PlanetLab nodes without interfering with the operating system or any other running processes. We present a research case study as an example that can make use of such a method to allow for evaluation over PlanetLab. The case study is about testing a new TCP stack, and comparing it to an existing TCP implementation under specific deployment scenarios.

Cross-Layer Design of Congestion Control and Power Control in Fast-Fading Wireless Networks

We study the cross-layer design of congestion control and power allocation with outage constraint in an interference-limited multihop wireless networks. Using a complete-convexification method, we first propose a message-passing distributed algorithm that can attain the global optimal source rate and link power allocation. Despite the attractiveness of its optimality, this algorithm requires larger message size than that of the conventional scheme, which increases network overheads. Using the bounds on outage probability, we map the outage constraint to an SIR constraint and continue developing a practical near-optimal distributed algorithm requiring only local SIR measurement at link receivers to limit the size of the message. Due to the complicated complete-convexification method, however the congestion control of both algorithms no longer preserves the existing TCP stack. To take into account the TCP stack preserving property, we propose the third algorithm using a successive convex approximation method to iteratively transform the original nonconvex problem into approximated convex problems, then the global optimal solution can converge distributively with message-passing. Thanks to the tightness of the bounds and successive approximations, numerical results show that the gap between three algorithms is almost indistinguishable. Despite the same type of the complete-convexification method, the numerical comparison shows that the second near-optimal scheme has a faster convergence rate than that of the first optimal one, which make the near-optimal scheme more favorable and applicable in practice. Meanwhile, the third optimal scheme also has a faster convergence rate than that of a previous work using logarithm successive approximation method.

Enhancing TCP throughput of highly available virtual machines via speculative communication

Checkpoint-recovery based virtual machine (VM) replication is an attractive technique for accommodating VM installations with high-availability. It provides seamless failover for the entire software stack executed in the VM regardless the application or the underlying operating system (OS), it runs on commodity hardware, and it is inherently capable of dealing with shared memory non-determinism of symmetric multiprocessing (SMP) configurations. There have been several studies aiming at alleviating the overhead of replication, however, due to consistency requirements, network performance of the basic replication mechanism remains extremely poor., In this paper we revisit the replication protocol and extend it with speculative communication. Speculative communication silently acknowledges TCP packets of the VM, enabling the guest's TCP stack to progress with transmission without exposing the messages to the clients before the corresponding execution state is checkpointed to the backup host. Furthermore, we propose replication aware congestion control, an extension to the guest's TCP stack that aggressively fills up the VMM's replication buffer so that speculative packets can be backed up and released earlier to the clients. We observe up to an order of magnitude improvement in bulk data transfer with speculative communication, and close to native VM network performance when replication awareness is enabled in the guest OS. We provide results of micro-, as well as application-level benchmarks.

Dual-Resource TCP/AQM for Processing-Constrained Networks

This paper examines congestion control issues for TCP flows that require in-network processing on the fly in network elements such as gateways, proxies, firewalls and even routers. Applications of these flows are increasingly abundant in the future as the Internet evolves. Since these flows require use of CPUs in network elements, both bandwidth and CPU resources can be a bottleneck and thus congestion control must deal with ldquocongestionrdquo on both of these resources. In this paper, we show that conventional TCP/AQM schemes can significantly lose throughput and suffer harmful unfairness in this environment, particularly when CPU cycles become more scarce (which is likely the trend given the recent explosive growth rate of bandwidth). As a solution to this problem, we establish a notion of dual-resource proportional fairness and propose an AQM scheme, called Dual-Resource Queue (DRQ), that can closely approximate proportional fairness for TCP Reno sources with in-network processing requirements. DRQ is scalable because it does not maintain per-flow states while minimizing communication among different resource queues, and is also incrementally deployable because of no required change in TCP stacks. The simulation study shows that DRQ approximates proportional fairness without much implementation cost and even an incremental deployment of DRQ at the edge of the Internet improves the fairness and throughput of these TCP flows. Our work is at its early stage and might lead to an interesting development in congestion control research.

A passive state-machine approach for accurate analysis of TCP out-of-sequence segments

In this paper we describe a new tool being made available to the networking research community for passive analysis of TCP segment traces. The purpose of the tool is to provide more complete and accurate classification of out-of-sequence segments than those provided by prior tools. One of the crucial factors that limits the accuracy of prior tools is that these do not incorporate variations across TCP implementations (for different operating systems) that have different parameters (e.g., timer granularity, minimum RTO, duplicate ACK thresholds, etc.) or algorithms that influence what can be inferred about out-of-sequence segments. Our tool explicitly accounts for implementation-specific details in four prominent TCP stacks (Windows, Linux, FreeBSD/Mac OS-X, and Solaris). We validate our tool through several controlled experiments with instances of all four OS-specific implementations used in the analysis. We then run this tool on packet traces of 52 million Internet TCP connections collected from 5 different locations and present the results. We also include comparisons with results from running selected prior tools on the same traces.

De-layered grid storage server

Networks have become faster and disks have become fatter at a pace that, despite Moore's law, CPU developments have simply not been been able to keep up with. We present a Grid Storage Server which is capable of scaling up to meet the "terabit-terabyte" demands of very large scale grid computation applications with large data sets.The Grid Storage Server is implemented almost completely in silicon, whether FPGA or ASIC; the fast-path of this server does not use a CPU or von Neumann style (instruction/data) machine. Instead, multiple layers of a protocol stack are compiled into a hardware engine that processes all layers concurrently on-chip. This "serverless" design allows it to scale up to match "terabit" network speeds and "terabyte" disk capacities enabling large scale grid applications.At a price-point a small percent of that of a server-based design, the Grid Server incorporates a standards compliant high-performance TCP stack that can saturate 40Gbps using a single or multiple TCP connections. The current design directly attaches to a storage array of 48TB capacity. The storage array is organized with a fault-tolerant RAID for performance and reliability; the RAID configuration is adaptive and can be tuned to conflicting application needs. As a bonus, because the control-plane in the silicon-based TCP engine has very low jitter, the protocol engine has mechanisms for nanosecond precision clock synchronisation across very large distances, thus, for the first time, enabling transcontinental real-time and temporal grid computation and database applications.

TCP smart framing: a segmentation algorithm to reduce TCP latency

TCP Smart Framing, or TCP-SF for short, enables the Fast Retransmit/Recovery algorithms even when the congestion window is small. Without modifying the TCP congestion control based on the additive-increase/multiplicative-decrease paradigm, TCP-SF adopts a novel segmentation algorithm: while Classic TCP always tries to send full-sized segments, a TCP-SF source adopts a more flexible segmentation algorithm to try and always have a number of in-flight segments larger than 3 so as to enable Fast Recovery. We motivate this choice by real traffic measurements, which indicate that today's traffic is populated by short-lived flows, whose only means to recover from a packet loss is by triggering a Retransmission Timeout. The key idea of TCP-SF can be implemented on top of any TCP flavor, from Tahoe to SACK, and requires modifications to the server TCP stack only, and can be easily coupled with recent TCP enhancements. The performance of the proposed TCP modification were studied by means of simulations, live measurements and an analytical model. In addition, the analytical model we have devised has a general scope, making it a valid tool for TCP performance evaluation in the small window region. Improvements are remarkable under several buffer management schemes, and maximized by byte-oriented schemes.

Microscopic examination of TCP flows over transatlantic links

Much of the recent research and development in the area of high-speed TCP is focused on the steady state behavior of TCP flows. However, our experience with the first research only transatlantic 2.5 Gbps Lambda link clearly demonstrates the need to focus on the initial stages of TCP. The work we present here examines the behavior of TCP flows at microscopic level over high-bandwidth long delay networks. This examination has led us to study the influence of the minute properties of the underlying network on bursty protocols such as TCP at these very high speeds combined with high latency. In this paper we briefly describe the requirements for such an extreme network environment to support high-speed TCP flows. We also present results collected using transatlantic links at iGrid2002 where we tuned various host parameters and used modified TCP stacks.

Scalable TCP

TCP congestion control can perform badly in highspeed wide area networks because of its slow response with large congestion windows. The challenge for any alternative protocol is to better utilize networks with high bandwidth-delay products in a simple and robust manner without interacting badly with existing traffic. Scalable TCP is a simple sender-side alteration to the TCP congestion window update algorithm. It offers a robust mechanism to improve performance in highspeed wide area networks using traditional TCP receivers. Scalable TCP is designed to be incrementally deployable and behaves identically to traditional TCP stacks when small windows are sufficient. The performance of the scheme is evaluated through experimental results gathered using a Scalable TCP implementation for the Linux operating system and a gigabit transatlantic network. The preliminary results gathered suggest that the deployment of Scalable TCP would have negligible impact on existing network traffic at the same time as improving bulk transfer performance in highspeed wide area networks.

Evaluation of Advanced TCP Stacks on Fast Long-Distance Production Networks

With the growing needs of data intensive science, such as high energy physics, and the need to share data between multiple remote computer and data centers worldwide, the necessity for high network performance to replicate large volumes (TBytes) of data between remote sites in Europe, Japan and the U.S. is imperative. Currently, most production bulk-data replication on the network utilizes multiple parallel standard (Reno based) TCP streams. Optimizing the window sizes and number of parallel stream is time consuming, complex, and varies (in some cases hour by hour) depending on network configurations and loads. We therefore evaluated new advanced TCP stacks that do not require multiple parallel streams while giving good performances on high speed long-distance network paths. In this paper, we report measurements made on real production networks with various TCP implementations on paths with different Round Trip Times (RTT) using both optimal and sub-optimal window sizes.We compared the New Reno TCP with the following stacks: HS-TCP, Fast TCP, S-TCP, HSTCP-LP, H-TCP and Bic-TCP. The analysis will compare and report on the stacks in terms of achievable throughput, impact on RTT, intra- and inter-protocol fairness, stability, as well as the impact of reverse traffic.We also report on some tentative results from tests made on unloaded 10 Gbps paths during SuperComputing 2003.

The Cyclone Server Architecture: streamlining delivery of popular content

We propose a new web server architecture optimized for delivery of large, popular files. Delivery of such files currently pose a scalability problem for conventional content providers, which must devote server-side resources in direct proportion to the high multiprogramming level induced by a set of these connections. While use of scalable multicast may remedy this problem some day, multicast is rarely supported in today's wide-area infrastructure. Our approach alleviates many of the most serious scalability problems by developing new server-side mechanisms capable of managing a large set of TCP connections transporting the same content. The strategy we employ relies on the use of fast forward error correcting (FEC) codes to generate encodings of popular content, of which only a small sliding window is cached in memory at any time instant. The concurrent TCP connections then access the content only from this shared window, which is globally useful to all clients. Our method hinges on eliminating unscalable TCP retransmission buffers, as we can “retransmit” fresh encoding packets in lieu of the originals with no performance degradation and with no modifications to client TCP stacks. Ultimately, our Cyclone server capitalizes on concurrency to maximize sharing of state across different request threads while minimizing context switching, disk I/O, thrashing under high load and the cache memory footprint. In this paper, we describe the design and prototype implementation of our approach as a Linux kernel subsystem.

Improving TCP performance over asymmetric networks

Bandwidth asymmetry is quite common among modern networks; e.g., ADSL, cable TV, wireless, and satellite link with a terrestrial return path. In these networks, the bandwidth over one direction can be orders of magnitude smaller than that over the other. The performance of TCP transfer in the high bandwidth direction can be severly reduced by the delay acknowledgment packets experienced in the reverse direction. In this paper, we describe our proposed solution ACE (Acknowledgement Based on Cwnd Estimation). In comparison to other solutions, ACE requires only modification of the TCP stack at terminals attached to an asymmetric network. We evaluated the performance of ACE over a cable modem network by simulation. We have also implemented ACE on the Linux platform and tested it on a small testbed network with an emulated asymmetric link. The performance improvement was significant especially when there is a high degree of asymmetry.

Effects of ensemble-TCP

TCP currently recalculates the state of each connection from a fixed set of initial parameters; this recalculation occurs over several round trips, during which the connection can be less than efficient. TCP control block sharing is a technique for reusing information among connections in series and aggregating it among connections in parallel. This paper explores the design space of a modified TCP stack that utilizes these two ideas, and one possible design (E-TCP) is presented in detail. E-TCP has been designed so that the network transmission behavior of group of parallel E-TCP connections closely resembles that of a single TCP/Reno connection. Simulated web accesses using HTTP/1.0 over E-TCP show a significant performance improvement compared to TCP/Reno connection bundles. This paper is first to evaluate performance using four different intra-ensemble schedulers for different workloads. In one scenario simulating a common case, E-TCP is 4-75% faster than Reno for transmitting the HTML parts of various pages, and 17-61% faster transmitting the whole pages. In the same scenario, reusing cached state speeds up repeated E-TCP page accesses by 17-53% for the HTML parts and 10-28% for the whole pages, compared to the initial access. E-TCP can also be integrated with other proposed TCP extensions (such as TCP/Vegas or TCP/SACK), to further improve performance.

Automatic TCP buffer tuning

With the growth of high performance networking, a single host may have simultaneous connections that vary in bandwidth by as many as six orders of magnitude. We identify requirements for an automatically-tuning TCP to achieve maximum throughput across all connections simultaneously within the resource limits of the sender. Our auto-tuning TCP implementation makes use of several existing technologies and adds dynamically adjusting socket buffers to achieve maximum transfer rates on each connection without manual configuration.Our implementation involved slight modifications to a BSD-based socket interface and TCP stack. With these modifications, we achieved drastic improvements in performance over large bandwidth delay paths compared to the default system configuration, and significant reductions in memory usage compared to hand-tuned connections, allowing servers to support at least twice as many simultaneous connections.