Byte-addressable Persistent Memory Research Articles

Perseid: A Secondary Indexing Mechanism for LSM-Based Storage Systems

LSM-based storage systems are widely used for superior write performance on block devices. However, they currently fail to efficiently support secondary indexing, since a secondary index query operation usually needs to retrieve multiple small values, which scatter in multiple LSM components. In this work, we revisit secondary indexing in LSM-based storage systems with byte-addressable persistent memory (PM). Existing PM-based indexes are not directly competent for efficient secondary indexing. We propose Perseid , an efficient PM-based secondary indexing mechanism for LSM-based storage systems, which takes into account both characteristics of PM and secondary indexing. Perseid consists of (1) a specifically designed secondary index structure that achieves high-performance insertion and query, (2) a lightweight hybrid PM-DRAM and hash-based validation approach to filter out obsolete values with subtle overhead, and (3) two adapted optimizations on primary table searching issued from secondary indexes to accelerate non-index-only queries. Our evaluation shows that Perseid outperforms existing PM-based indexes by 3–7× and achieves about two orders of magnitude performance of state-of-the-art LSM-based secondary indexing techniques even if on PM instead of disks.

FluidKV: Seamlessly Bridging the Gap between Indexing Performance and Memory-Footprint on Ultra-Fast Storage

Our extensive experiments reveal that existing key-value stores (KVSs) achieve high performance at the expense of a huge memory footprint that is often impractical or unacceptable. Even with the emerging ultra-fast byte-addressable persistent memory (PM), KVSs fall far short of delivering the high performance promised by PM's superior I/O bandwidth. To find the root causes and bridge the huge performance/memory-footprint gap, we revisit the architectural features of two representative indexing mechanisms (single-stage and multi-stage) and propose a three-stage KVS called FluidKV. FluidKV effectively consolidates these indexes by fast and seamlessly running incoming key-value request stream from the write-concurrent frontend stage to the memory-efficient backend stage across an intermediate stage. FluidKV also designs important enabling techniques, such as thread-exclusive logging, PM-friendly KV-block structures, and dual-grained indexes, to fully utilize both parallel-processing and high-bandwidth capabilities of ultra-fast storage hardware while reducing the overhead. We implemented a FluidKV prototype and evaluated it under a variety of workloads. The results show that FluidKV outperforms the state-of-the-art PM-aware KVSs, including ListDB and FlatStore with different indexes, by up to 9× and 3.9× in write and read throughput respectively, while cutting up to 90% of the DRAM footprint.

ESH: Design and Implementation of an Optimal Hashing Scheme for Persistent Memory

Recent advancements in memory technology have opened up a wealth of possibilities for innovation in data structures. The emergence of byte-addressable persistent memory (PM) with its impressive capacity and low latency has accelerated the adoption of PM in existing hashing-based indexes. As a result, several new hashing schemes utilizing emulators have been proposed. However, these schemes have proven to be suboptimal, lacking scalability when implemented on real devices. Only a handful of hash table designs have successfully addressed critical properties such as load factor, scalability, efficient memory utilization, and recovery. One of the main challenges in redesigning data structures for an effective hashing scheme in PM is minimizing the overhead associated with dynamic hashing operations in the hash table. To tackle this challenge, this paper introduces ESH, an efficient and scalable hashing scheme that significantly improves memory efficiency, scalability, and overall performance on PM. The ESH scheme maximizes the utilization of the hash table’s available space, thus reducing the frequency of full-table rehashing and improving performance. Consequently, this scheme achieves a high load factor while minimizing the need for rehashing. To evaluate the effectiveness of the ESH scheme, we compare it to widely used dynamic hashing schemes employing similar techniques on Intel Optane® DC persistent memory (DCPMM). The experimental results demonstrate that ESH outperforms CCEH and Dash in terms of data insertion performance, exhibiting a 30% improvement over CCEH and a 4% improvement over Dash. Furthermore, ESH improves the lookup operation by nearly 10% compared to Dash, while achieving a load factor of up to 91%, surpassing its competitors.

TPFS: A High-Performance Tiered File System for Persistent Memories and Disks

Emerging fast, byte-addressable persistent memory (PM) promises substantial storage performance gains compared with traditional disks. We present TPFS, a tiered file system that combines PM and slow disks to create a storage system with near-PM performance and large capacity. TPFS steers incoming file input/output (I/O) to PM, dynamic random access memory (DRAM), or disk depending on the synchronicity, write size, and read frequency. TPFS profiles the application’s access stream online to predict the behavior of file access. In the background, TPFS estimates the “temperature” of file data and migrates the write-cold and read-hot file data from PM to disks. To fully utilize disk bandwidth, TPFS coalesces data blocks into large, sequential writes. Experimental results show that with a small amount of PM and a large solid-state drive (SSD), TPFS achieves up to 7.3× and 7.9× throughput improvement compared with EXT4 and XFS running on an SSD alone, respectively. As the amount of PM grows, TPFS’s performance improves until it matches the performance of a PM-only file system.

PetaKV: Building Efficient Key-Value Store for File System Metadata on Persistent Memory

Previous works proposed building file systems and organizing the metadata with KV stores because KV stores handle entries of various sizes efficiently and have excellent scalability. The emergence of the byte-addressable persistent memory (PM) enables metadata service to be faster than before by tailoring the KV store for the PM. However, existing PM-based KV stores cannot handle the workloads of file systems’ metadata well because simply depending on hash tables or trees cannot simultaneously provide fast file accessing and efficient directory traversing. In this paper, we exploit the insight of the metadata operations and propose the PetaKV, a KV store tailored for the metadata management of file systems on PM. PetaKV leverages dual hash indexing to achieve fast file put and get operations. Moreover, it cooperates with PM-tailored peta logs to collocate KV entries for each directory, thus supporting efficient directory scans. Our evaluation indicates PetaKV outperforms state-of-art tree-based KV stores on put, get and scan <inline-formula><tex-math notation="LaTeX">$2.5\times$</tex-math></inline-formula> , <inline-formula><tex-math notation="LaTeX">$3.2\times$</tex-math></inline-formula> , and <inline-formula><tex-math notation="LaTeX">$2.8\times$</tex-math></inline-formula> on average, respectively. Moreover, the file system built with PetaKV achieves <inline-formula><tex-math notation="LaTeX">$1.2\times$</tex-math></inline-formula> to <inline-formula><tex-math notation="LaTeX">$6.4\times$</tex-math></inline-formula> speedup compared to those built with tree-based KV stores on the metadata operations.

Hydra: A Decentralized File System for Persistent Memory and RDMA Networks

Emerging byte-addressable persistent memory (PM) has the potential to disrupt the boundary between memory and storage. Combined with high-speed RDMA networks, distributed PM-based storage systems offer the opportunity to provide huge increases in storage performance by closely coupling PM and RDMA features. However, existing distributed file systems adopt the conventional centralized client-server architecture designed for traditional disks, leading to excessive access latency, limited scalability, and high recovery overhead. In this paper, we propose a fully decentralized PM-based file system, Hydra. By exploiting the performance advantages of local PM, Hydra leverages data access locality to achieve high performance. To accelerate file transmission among Hydra nodes, file metadata and data are decoupled and updated differentially through one-sided RDMA reads. Hydra also batches RDMA requests and classifies RPCs into synchronous and asynchronous types to minimize network overhead. Decentralization enables Hydra to tolerate node failures and achieve load balancing. Experimental results show that Hydra outperforms existing distributed file systems by a large margin, and shows good scalability on multi-threaded and parallel workloads.

Pivotal B+tree for Byte-Addressable Persistent Memory

Over the past few years, various indexes have been redesigned for byte-addressable persistent memory. In this work, we design and implement <i>PB</i>&#x002B;<i>tree</i> (Pivotal B&#x002B;tree) that resolves the limitations of state-of-the-art fully persistent B&#x002B;trees. First, PB&#x002B;tree reduces the number of expensive shift operations by up to half by managing two sub-arrays separated by a <i>pivot</i> key. Second, PB&#x002B;tree reads cachelines in ascending order, which makes PB&#x002B;tree benefit from hardware prefetchers and run faster than state-of-the-art persistent B&#x002B;trees that access cachelines in non-contiguous or descending order. Third, PB&#x002B;tree employs an optimistic lock-free search algorithm to avoid repeatedly visiting the same tree node. Although the optimistic lock-free search algorithm involves a risk of visiting incorrect child nodes, PB&#x002B;tree guarantees correct search results using the <i>lazy correction</i> algorithm using doubly linked sibling pointers. Our performance study shows that PB&#x002B;tree outperforms the state-of-the-art fully persistent indexes by a large margin. A search algorithm without optimistic locking risks visiting the wrong child node, but PB&#x002B;tree uses a <i>lazy correction</i> algorithm with doubly linked sibling pointers to ensure correct search results. Our performance studies show that PB&#x002B;trees outperform state-of-the-art fully persistent indexes.

Two Birds With One Stone: Boosting Both Search and Write Performance for Tree Indices on Persistent Memory

The advance of byte-addressable persistent memory (PM) makes it a hot topic to revisit traditional tree indices such as B+-tree and radix tree, and a few new persistent memory-friendly tree indices have been proposed. However, due to the special features of persistent memory compared to DRAM and the limitations of B+-tree-like indices, it is much harder to optimize both search and write performance for tree indices on persistent memory. As a result, most existing indices for persistent memory, e.g., WB-tree, proposed to improve write performance while sacrificing search performance. Aiming to optimize both write and search performance for tree indices on persistent memory, in this paper, we first propose a novel Two-Layer Architecture (TLA) for constructing tree indices on persistent memory. The key idea, of TLA is to organize the index with a search-optimized top layer and a write-optimized bottom layer, letting the top layer optimize search performance and the bottom layer improve write performance. By adopting efficient structures for the two layers, TLA can boost both write and search performance for tree indices on persistent memory. Following the TLA architecture, we present a new index called TLBtree (Two-Layer B+-tree) offering high search and write performance for persistent memory. Moreover, we develop a concurrent TLBtree to support non-blocking read operations in multi-core environment. We evaluate our proposals under a server equipped with real Intel Optane persistent memory. The results show that TLBtree outperforms the state-of-the-art tree indices, including WB-tree, Fast&amp;Fair, and FPTree, in both search and write performance. Also, the concurrent TLBtree can achieve up to 3.7x speedup than its competitors under the multi-core environment.

Technical Perspective

Byte-addressable persistent memory was considered in the data management community as long ago as 1986. Thatte saw the advantages for programmability in unifying the abstractions of byte-addressable RAM with persistence [2]. Thatte's context was object-oriented databases containing a variety data structures that would be awkward to transform into the block-oriented abstractions provided by typical secondary storage. Thatte's proposed physical instantiation of persistent memory was a disk-backed device, although it is unclear whether such a device was ever built. Thatte recognized the importance of recovery to the overall scheme.

Scaling Dynamic Hash Tables on Real Persistent Memory

Byte-addressable persistent memory (PM) brings hash tables the potential of low latency, cheap persistence and instant recovery. The recent advent of Intel Optane DC Persistent Memory Modules (DCPMM) further accelerates this trend. Many new hash table designs have been proposed, but most of them were based on emulation and perform sub-optimally on real PM. They were also piecewise and partial solutions that side-stepped many important properties, in particular good scalability, high load factor and instant recovery.

Usage Scenarios for Byte-Addressable Persistent Memory in High-Performance and Data Intensive Computing

Byte-addressable persistent memory (B-APM) presents a new opportunity to bridge the performance gap between main memory and storage. In this paper, we present the usage scenarios for this new technology, based on the capabilities of Intel's DCPMM. We outline some of the basic performance characteristics of DCPMM, and explain how it can be configured and used to address the needs of memory and I/O intensive applications in the HPC and data intensive domains. Two decision trees are presented to advise on the configuration options for B-APM; their use is illustrated with two examples. We show that the flexibility of the technology has the potential to be truly disruptive, not only because of the performance improvements it can deliver, but also because it allows systems to cater for wider range of applications on homogeneous hardware.

Contour: A Process Variation Aware Wear-Leveling Mechanism for Inodes of Persistent Memory File Systems

Existing persistent memory file systems exploit the fast, byte-addressable persistent memory (PM) to boost storage performance but ignore the limited endurance of PM. Particularly, the PM storing the inode section is extremely vulnerable for the inodes are most frequently updated, fixed on a location throughout lifetime, and require immediate persistency. The huge endurance variation of persistent memory domains caused by process variation makes things even worse. In this article, we propose a process variation aware wear leveling mechanism called Contour for the inode section of persistent memory file system. Contour first enables the movement of inodes by virtualizing the inodes with a deflection table. Then, Contour adopts cross-domain migration algorithm and intra-domain migration algorithm to balance the writes across and within the memory domains. We implement the proposed Contour mechanism in Linux kernel 4.4.30 based on a real persistent memory file system, SIMFS. We use standard benchmarks, including Filebench, MySQL, and FIO, to evaluate Contour. Extensive experimental results show Contour can improve the wear ratios of pages 417.8× and 4.5× over the original SIMFS and PCV, the state-of-the-art inode wear-leveling algorithm, respectively. Meanwhile, the average performance overhead and wear overhead of Contour are 0.87 and 0.034 percent in application-level workloads, respectively.

Dash

Byte-addressable persistent memory (PM) brings hash tables the potential of low latency, cheap persistence and instant recovery. The recent advent of Intel Optane DC Persistent Memory Modules (DCPMM) further accelerates this trend. Many new hash table designs have been proposed, but most of them were based on emulation and perform sub-optimally on real PM. They were also piece-wise and partial solutions that side-step many important properties, in particular good scalability, high load factor and instant recovery. We present Dash, a holistic approach to building dynamic and scalable hash tables on real PM hardware with all the aforementioned properties. Based on Dash, we adapted two popular dynamic hashing schemes (extendible hashing and linear hashing). On a 24-core machine with Intel Optane DCPMM, we show that compared to state-of-the-art, Dash-enabled hash tables can achieve up to ∼3.9× higher performance with up to over 90% load factor and an instant recovery time of 57ms regardless of data size.

Compatible byte-addressable direct I/O for peripheral memory devices in Linux

Memory devices can be used as storage systems to provide a lower latency that can be achieved by disk and flash storage. However, traditional buffered input/output (I/O) and direct I/O are not optimized for memory-based storages. Traditional buffered I/O includes a redundant memory copy with a disk cache. Traditional direct I/O does not support byte addressing. Memory-mapped direct I/O, which optimizes file operations for byte-addressable persistent memory and appears to the CPU as a main memory. However, it has an interface that is not always compatible with existing applications. In addition, it cannot be used for peripheral memory devices (e.g., networked memory devices and hardware RAM drives) that are not interfaced with the memory bus. This paper presents a new Linux I/O layer, byte direct I/O (BDIO), that can process byte-addressable direct I/O using the standard application programming interface. It requires no modification of existing application programs and can be used not only for the memory but also for the peripheral memory devices that are not addressable by a memory management unit. The proposed BDIO layer allows file systems and device drivers to easily support BDIO. The new I/O achieved 18% to 102% performance improvements in the evaluation experiments conducted with online transaction processing, file server, and desktop virtualization storage.