Exploiting RDMA for Distributed Low-Latency Key/Value Store on Non-volatile Main Memory

Abstract

In-memory key/value stores have been widely deployed in many large systems for high throughput and low latency data access. To offer durability and fault tolerance, many of them (e.g., Memcached, Redis) flush data to RAID or SSD in local and remote with primary backup replication (PBR), in which the RAID and network become the bottleneck of throughput and latency. Recently, two promising technologies are emerging, i.e., non-volatile memory (NVM) and remote direct memory access (RDMA). NVMs are integrating into the main memory for their high density, high access speed, low cost and byte addressability. And RDMA allows direct access to the memory of a remote machine without invoking the peer CPU, thus reducing latency. To overcome the limitations of traditional key/value stores, this paper explores the design of a distributed low-latency key/value store on non-volatile main memory by leveraging RDMA. NVM not only eliminates disk IO operations, but also makes fast repli- cation possible. For a PUT operation, we collect modifications (data and metadata) at the local NVM of the primary server and replicate them to the same positions of backup servers by RDMA writes. A prototype distributed key/value store, called NVDS, is designed and implemented. Extensive experiments on the micro benchmark show that NVDS achieves significant latency reduction, compared to existing key/value stores. It replicates a 32bytes key/value PUT operation to two backup servers in less than 2μs and the whole request latency is less than 10μs.