Design and testbed evaluation of RDMA-based middleware for high-performance data transfer applications

Abstract

Providing high-speed data transfer is vital to various data-intensive applications supported by data center networks. We design a middleware layer of high-speed communication based on Remote Direct Memory Access (RDMA) that serves as the common substrate to accelerate various data transfer tools, such as FTP, HTTP, file copy, sync and remote file I/O. This middleware offers better end-to-end bandwidth performance than the traditional TCP-based alternatives, while it hides the heterogeneity of the underlying high-speed architecture. This paper describes this middleware's function modules, including resource abstraction and task synchronization and scheduling, that maximize the parallelism and performance of RDMA operations. For networks without RDMA hardware acceleration, we integrate Linux kernel optimization techniques to reduce data copy and processing in the middleware. We provide a reference implementation of the popular file-transfer protocol over this RDMA-based middleware layer, called RFTP. Our experimental results show that our RFTP outperforms several TCP-based FTP tools, such as GridFTP, while it maintains very low CPU consumption on a variety of data center platforms. Furthermore, those results confirm that our RFTP tool achieves near line-speed performance in both LAN and WAN, and scales consistently from 10Gbps Ethernet to 40Gbps Ethernet and InfiniBand environments.