RDMA Based Performance Optimization on Distributed Database Systems: A Case Study with GoldenX

Abstract

The performance of distributed database system heavily relies on network to assist collaboration among nodes, while the traditional TCP/IP network has become the major bottleneck of the distributed database. Meanwhile, the emerging technology RDMA, featured with CPU offloading, bypassing the operating system kernel and zero copy, is envisioned to achieve low latency and high throughput data transmission between nodes. For this purpose, in this paper, we design and implement a high performance data transmission scheme based on RDMA and apply it to the distributed database system. Firstly, the paper analyzes the application scenarios and internal communication requirements of distributed database systems, combined with RDMA hardware characteristics. Secondly, based on the analysis results, we perform targeted design optimizations in three aspects, RDMA data transmission and memory region management, data placement, and congestion control. We propose efficient variable length data transmission mechanism based on the sliding window, application friendly data placement mechanism, and priority based adaptive congestion control mechanism to optimize the three aspects respectively. We further implement our idea with the typical distributed database GoldenX, and perform comprehensive experiments. Results shows that, compared with using RDMA network directly, the performance of GoldenX integrated with the optimizations is increased to 2.95 times, and the throughput is up to 2.61 times.