User-level Protocol Research Articles

Stackable Transactional File System Using Kernel-Level WAL

User applications that require transactional file access generally use user-level transactional protocols such as logging, atomic rename, since the POSIX interface does not support transactional file operations. The user-level protocols are complex and require specific persistence properties of file systems for correctness. If the kernel provides transactional access to file data, application developers can avoid the complex and error-prone application-level protocols. Many previous studies on kernel-level transactional support have several critical limitations such as complex APIs, file system-dependence, and low transaction concurrency. In this paper, we propose a kernel-level write-ahead logging (WAL) technique, called KWAL. KWAL maintains user-transparent WAL files and redirects transactional IOs to them. Since KWAL is implemented as a stackable file system, it can be mounted over any native file systems, without modifying them. To provide a higher transaction concurrency while guaranteeing isolation, KWAL supports byte-level write conflict detection and repeatable read isolation. We also utilize a block remap technique to resolve the duplicate write problem of WAL-based transaction scheme. To demonstrate the usefulness of KWAL, we implemented a simple key-value store using KWAL APIs, fixed the wrong transaction implementation of the Git application, and simplified the implementation of the storage engines of MySQL and SQLite.

Performance analysis of Google’s Quick UDP Internet Connection Protocol under Software Simulator

The QUIC protocol (Quick UDP Internet Connection) is a transport based over UDP Protocol (User Datagram Protocol) that provides safe, reliable and fast service on the Internet. Google proposed it to solve the problem of network delay. It is efficient, fast, and takes up fewer resources. The QUIC gathers the advantages of both TCP and UDP. QUIC is a user-level protocol running on top of UDP, which considered by IETF (Internet Engineering Task Force). Google assumes the response time of page load was short so that end-user performance was better. In this paper, the results presented on a local test using NS-3 (network simulator version 3) that allows QUIC output to test and design choices and possible limitations present the description of the functionalities and the main assumptions of the internet QUIC.

Rekindling network protocol innovation with user-level stacks

Recent studies show that more than 86% of Internet paths allow well-designed TCP extensions, meaning that it is still possible to deploy transport layer improvements despite the existence of middleboxes in the network. Hence, the blame for the slow evolution of protocols (with extensions taking many years to nbecome widely used) should be placed on end systems. In this paper, we revisit the case for moving protocols stacks up into user space in order to ease the deployment of new protocols, extensions, or performance optimizations. We present MultiStack, operating system support for user-level protocol stacks. MultiStack runs within commodity operating systems, can concurrently host a large number of isolated stacks, has a fall-back path to the legacy host stack, and is able to process packets at rates of 10Gb/s. We validate our design by showing that our mux/demux layer can validate and switch packets at line rate (up to 14.88 Mpps) on a 10 Gbit port using 1-2 cores, and that a proof-of-concept HTTP server running over a basic userspace TCP outperforms by 18-90% both the same server and nginx running over the kernel's stack.

Signals, timers, and continuations for multithreaded user-level protocols

Signals, timers, and continuations for multithreaded user-level protocols

Exploiting NIC architectural support for enhancing IP-based protocols on high-performance networks

While a number of user-level protocols have been developed to reduce the gap between the performance capabilities of the physical network and the performance actually available, their compatibility issues with the existing sockets-based applications and IP-based infrastructure has been an area of major concern. To address these compatibility issues while maintaining a high performance, a number of researchers have been looking at alternative approaches to optimize the existing traditional protocol stacks. Broadly, previous research has broken up the overheads in the traditional protocol stack into four related aspects, namely: (i) compute requirements and contention, (ii) memory contention, (iii) I/O bus contention and (iv) system resources’ idle time. While previous research dealing with some of these aspects exists, to the best of our knowledge, there is no work which deals with all these issues in an integrated manner while maintaining backward compatibility with existing applications and infrastructure. In this paper, we address each of these issues, propose solutions for minimizing these overheads by exploiting the emerging architectural features provided by modern Network Interface Cards (NICs) and demonstrate the capabilities of these solutions using an implementation based on UDP/IP over Myrinet. Our experimental results show that with our implementation of UDP, termed as E-UDP, can achieve up to 94% of the theoretical maximum bandwidth. We also present a mathematical performance model which allows us to study the scalability of our approach for different system architectures and network speeds.

Software Distributed Shared Memory: a VIA-based implementation and comparison of sequential consistency with home-based lazy release consistency

A Distributed Shared Memory (DSM) system provides a distributed application with a shared virtual address space. This article proposes a design for implementing the DSM communication layer on top of the Virtual Interface Architecture (VIA), an industry standard for user-level networking protocols on high-speed clusters. User-level communication protocols operate in user mode, thus removing the operating system kernel's overhead from the critical communication pass, and significantly diminishing communication overhead as a result. We analyze VIA's facilities and limitations in order to ascertain which implementation trade-offs can be best applied to our development of an efficient communication substrate optimized for DSM requirements. We then implement a multithreaded version of the Home-based Lazy Release Consistency (HLRC) protocol on top of this substrate. In addition, we compare the performance of this HLRC protocol with that of the Sequential Consistency (SC) protocol in which a MultiView (MV) memory mapping technique was used. This technique enables a fine-grained access to shared memory, while still relying on the virtual memory hardware to track memory accesses. We perform an ‘apple-to-apple’ comparison on the same testbed environment and benchmark suite, and investigate the effectiveness and scalability of both protocols. Copyright © 2005 John Wiley & Sons, Ltd.

Firmware-Level Latency Analysis on a Gigabit Network

Gigabit networks are equipped with “increasingly” intelligent network interface cards, and the firmware running in the cards does various tasks related to end-to-end communication. For an accurate performance evaluation of gigabit networks, it is very important to characterize and quantify the firmware. However, the firmware has been neglected in the latency analyzes of network protocols. This paper presents an in-depth latency analysis of Myrinet. Our findings include that the major bottleneck is the network interface card itself. This is true especially for so-called lightweight user-level protocols (such as BPI of Myrinet) designed for high-speed communication. Although BPI is very lean and efficient in the host, its sending throughput becomes similar to UDP. This result is very unexpected and surprising. Through firmware-level measurements, we identify that the cause of bottleneck is the DMA performance.

A new fast message passing communication system for multiprocessor workstation clusters

In this paper, a fast message passing communication architecture (FMP) is proposed in order to reduce the software overhead of communication, which has been a great impediment to the good performance of workstation clusters. A FMP system is implemented over the SUN Ultra2-Myrinet platform. Measurements show that this implementation has achieved a one-way latency of 11.2us for one-byte packets in network communication, and only 4.9us in local communication. Its bandwidth can reach as high as 338Mb/s for 8KB packets in network communication, and more than 770Mb/s in local message passing. These results tell us that FMP can really exploit the performance of supercomputers and high-speed networks. Local communication is implemented through shared memory within a single host, while the methods to decrease overheads of network communication depend on user-level communication protocol, pipeline transmission, credit flow control and multithreading. Furthermore, the whole system provides the interface for both network and local communication.

Design and implementation of efficient communication abstractions on the Virtual Interface Architecture: Stream sockets and RPC experience

AbstractThe emergence and standardization of system area networks (SANs) has provided distributed applications with a medium for high‐bandwidth, low‐latency communication. Standard user‐level networking architecture such as the Virtual Interface (VI) Architecture enables distributed applications to perform low overhead communication over SANs. The VI Architecture significantly reduces system processing overheads and provides each consumer process with a protected, directly accessible interface to the network hardware. Developing distributed applications using low‐level primitives provided by user‐level networking architecture like the VI Architecture is complex and requires significant effort.This paper describes how high‐level communication paradigms like stream sockets and remote procedure call (RPC) can be efficiently built over the VI Architecture. To evaluate performance benefits for standard client–server and multi‐threaded environments, our focus is on off‐the‐shelf sockets and RPC interfaces and commercially available VI Architecture‐based SANs. The key design techniques developed in this paper include credit‐based flow control, decentralized user‐level protocol processing, caching of pinned communication buffers, and deferred processing of completed send operations.In the experimental evaluation, the one‐way bandwidth achieved by stream sockets over VI Architecture was three to four times better than the bandwidth achieved by running legacy protocols over the same interconnect. On the same SAN, high‐performance stream sockets and RPC over VI Architecture achieve significantly better (between 2× and 3× less) latency than conventional stream sockets and RPC over standard networking protocols in a Windows NT™ 4.0 environment. Furthermore, our high‐performance RPC transparently improved the network performance of the distributed component object model (DCOM) by a factor of two to three. Copyright © 2001 John Wiley & Sons, Ltd.

Supporting Cost-Effective Fault Tolerance in Distributed Message-Passing Applications with File Operations

In this paper we present an approach to reliable distributed computing, which incorporates fault tolerance into applications at low cost, in terms of both run-time performance and programming effort required to construct reliable application software. In our model fault tolerance is based on distributed consistent checkpointing and rollback-recovery integrated with a user-level reliable transmission protocol. By employing novel techniques 8and algorithms, our approach is distinguished from other consistent checkpointing schemes by the following features: first, minimum communication overhead for constructing a consistent distributed checkpoint and catching messages in transit during checkpointing; second, tolerance to message losses due to site failures or unreliable non-FIFO networks; and third, efficient checkpointing and recovery of persistent state, i.e., user files. Based on the model, a software library prototype called Libra has been implemented for supporting fault tolerance in distributed message-passing applications with file operations. The library provides an easy to use programming interface including message-passing and file I/O primitives, which hides the complexity of both fault-tolerant network communications and checkpointing and recovering user files from the application level. Experience with a number of long-running distributed applications shows that Libra can provide fault tolerance in a cost-effective manner.

Operating system support for distributed multimedia

We have been investigating an end system architecture to support networking with quality of service guarantees. For user level protocol code in our architecture to access the network, we have designed a kernel-user interface. The interface targets three areas for improvement: reduced copying, reduced reliance on explicit kernel-user interactions, and provision of rate-based flow control. In this paper, we present the concept of input-output efficient buffers for reduced copying, the concept of fast system calls for low-latency network access, and the concept of kernel threads for flow control. Also included is a concept called direct media streaming which is suitable for applications that require limited user processing of media data. These concepts have been implemented as an extension to SunOS 5.3 (the operating system component of Solaris 2.3). We report some experimental results on the performance of our current system.

Network monitoring system design

Computer networks can be very useful, but difficult to manage. Current network management tools tend to be either very expensive or inadequate for classroom modification. XSNIFF is a project to develop an alternative network monitoring tool with a graphical display in the X-Window environment. XSNIFF uses an efficient network packet filtering system developed by Jacobson, Leres, and McCanne of Lawrence Berkeley Laboratory[Jacobson, 1994], to receive all network traffic from the network wire and then translate it into usable information. The network listening device should not affect the normal flow of network traffic, but should observe it. XSNIFF has a graphical interface that allows the user to select between statistical information and a TCP connection graph. The statistical information displays data belonging to different network level protocols and different user level protocols. The connection information is displayed as a graph showing which network nodes are connected to what other nodes. The information can be used by a teacher in a computer network class to help students visualize network traffic and as a learning tool to help students understand computer network protocols.

Migrating sockets-end system support for networking with quality of service guarantees

We present an end system architecture designed to support networking with quality of service (QoS) guarantees. The protocol processing component of the architecture, called migrating sockets, has been designed with minimal hidden scheduling which enables accurate determination of the rate requirement of a user application. The end system provides QoS guarantees using: 1) an adaptive rate-controlled scheduler; 2) rate-based flow control on the send side for access to reserved-rate network connections; and 3) a constant overhead active demultiplexing mechanism on the receive side which can be transparently enabled in wide-area TCP/IP internetworking (although it is not restricted to TCP/IP). To achieve efficiency, migrating sockets lets user applications manage network endpoints with minimal system intervention, provides user level protocols read-only access to routing information, and integrates kernel level support previously built for efficient data movement. Migrating sockets is backward compatible with Unix semantics and Berkeley sockets. It has been used to implement Internet protocols such as TCP, UDP, and IP (including IP multicast), and run existing applications such as vic. Migrating sockets has been implemented in Solaris 2.5.1. We discuss our implementation experience, and present performance results of our system running on Sun Sparc and Ultra workstations, as well as Pentium-II desktops.

An efficient approach for constructing reliable distributed applications

A trend of supporting fault tolerance in distributed computing systems is to incorporate fault tolerance into applications at low cost, in terms of both run-time performance and programming effort required to construct reliable application software. To meet this objective, we have designed and implemented Libra, a software library running on a network of workstations to support reliable distributed computing efficiently. By providing fault tolerance transparency and a simple, easy to use high-level message-passing interface, Libra simplifies the development of reliable distributed applications. Fault tolerance is based on distributed consistent checkpointing and rollback-recovery integrated with a user-level network communication protocol. To address real-life problems, the library tolerates messages losses, and efficiently supports checkpointing and recovery of user files. Performance evaluations show that Libra imposes low run-time overhead, and minimizes communication overhead for taking a consistent distributed checkpoint and catching messages in transit during checkpointing.

Efficient Multithreaded User-Space Transport for Network Computing: Design and Test of the TRAP Protocol

We present a novel user-space and transaction-oriented protocol for use in high-performance distributed computing applications. The TRAP protocol is designed to support low-latency communication in multithreaded message-passing libraries. To demonstrate the viability of TRAP, we implemented and tested the fully operational protocol and conducted several experiments comparing its runtime performance with that of popular communication libraries. For fairness, the experiments did not exploit the TRAP protocol's multithreading capability at the application level. The results show that a TRAP-based communication library can, in various scenarios, outperform extant libraries based on in-kernel or user-level protocols. The results of these experiments also show that the TRAP protocol exhibits scalability characteristics, with respect to number of computing and communicating nodes, that is superior to regular in-kernel TCP/IP and other user-space protocol implementations.

An interprocess communication model for a distributed software testbed

This paper describes design and implementation of an IPC model for a distributed software testbed Shoshin. The IPC model was designed to provide a reliable message communication among distributed processes and to support implementations of user level protocols for various applications. A multiprocess structure, which separates a link level and a transport level, has been used to provide a flexible protocol management. The basic performance of the current implementation and the extensibility of the model are also discussed.

Remarks on negotiation mechanism and attention handling

Preliminary studies are at present being made for building a computer network linking the computer centres of the different Belgian Universities. This network would include:1. a packet switching network with the X25 service on top of it;2. a transport protocol;3. user level protocols (a virtual terminal protocol and a file transfer protocol).Cooperation between terminals and applications will be organized using a virtual terminal approach.Assuming a transport function offering:1. block transmission with error control on contents, error control on arrival and sequencing;2. telegram transmission with error control on contents and error control on arrival,we present the attention handling scheme proposed for the Virtual Terminal Protocol of the Belgian University Network.The proposed scheme includes a renegotiation of the data flow mode (free running or alternate).A detailed algorithmic description of the proposed attention handling scheme is given using a state diagram representation.