Crash Consistency Research Articles

Delaying Crash Consistency for Building A High-Performance Persistent Memory File System

Delaying Crash Consistency for Building A High-Performance Persistent Memory File System

A Novel Efficient Crash Consistency Solution Enabling Rollback Recovery for Secure NVM in Low-Power Energy Harvesting Systems

A Novel Efficient Crash Consistency Solution Enabling Rollback Recovery for Secure NVM in Low-Power Energy Harvesting Systems

Anchor: A Library for Building Secure Persistent Memory Systems

Cloud infrastructure is experiencing a shift towards disaggregated setups, especially with the introduction of the Compute Express Link (CXL) technology, where byte-addressable ersistent memory (PM) is becoming prominent. To fully utilize the potential of such devices, it is a necessity to access them through network stacks with equivalently high levels of performance (e.g., kernel-bypass, RDMA). While, these advancements are enabling the development of high-performance data management systems, their deployment on untrusted cloud environments also increases the security threats. To this end, we present Anchor, a library for building secure PM systems. Anchor provides strong hardware-assisted security properties, while ensuring crash consistency. Anchor exposes APIs for secure data management within the realms of the established PM programming model, targeting byte-addressable storage devices. Anchor leverages trusted execution environments (TEE) and extends their security properties on PM. While TEE's protected memory region provides a strong foundation for building secure systems, the key challenge is that: TEEs are fundamentally incompatible with PM and kernel-bypass networking approaches-in particular, TEEs are neither designed to protect untrusted non-volatile PM, nor the protected region can be accessed via an untrusted DMA connection. To overcome this challenge, we design a PM engine that ensures strong security properties for the PM data, using confidential and authenticated PM data structures, while preserving crash consistency through a secure logging protocol. We further extend the PM engine to provide remote PM data operations via a secure network stack and a formally verified remote attestation protocol to form an end-to-end system. Our evaluation shows that Anchor incurs reasonable overheads, while providing strong security properties.

PH-ORAM: An efficient persistent ORAM design for hybrid memory systems

The hybrid memory system (HMS), composed of DRAM and NVM, shows great potential for constructing large-scale, high-performance, and persistent memory systems. However, it is vulnerable to side-channel attacks that exploit the memory access pattern of programs to infer sensitive information. While Oblivious RAM (ORAM) is a general solution to defend against such attacks, it is not ready to be applied to HMSs. ORAM randomly distributes data across the entire memory space, causing the HMSs to lose the data persistence feature of NVM. An unexpected system crash will cause the DRAM data to be lost. Moreover, inconsistency may occur between metadata and data when the system crash interrupts ongoing ORAM operations. The inconsistency can cause the system to lose some NVM blocks after a crash reconstruction. How to effectively ensure data persistence and crash consistency in oblivious (or ORAM-protected) hybrid memory systems is still an open problem.This paper proposes PH-ORAM, an efficient ORAM design that provides a secure and persistent program execution environment on hybrid memory systems. First, we analyze the challenges of designing a persistent oblivious HMS and propose Persistent Duplication, an NVM-friendly data persistence scheme that does not require extra NVM writes or battery backup to persist DRAM data. Then, we analyze the consistency issues in NVM-based Ring ORAM systems and propose State-Aware stash management, a lightweight crash-consistent scheme that does not degrade system security and performance. Finally, we conduct cycle-accurate simulations to analyze the efficiency of PH-ORAM. Compared with the baseline Ring ORAM without any persistence support, PH-ORAM only increases the execution time by 0.6% on average.

Memento: A Framework for Detectable Recoverability in Persistent Memory

Persistent memory (PM) is an emerging class of storage technology that combines the performance of DRAM with the durability of SSD, offering the best of both worlds. This had led to a surge of research on persistent objects in PM. Among such persistent objects, concurrent data structures (DSs) are particularly interesting thanks to their performance and scalability. One of the most widely used correctness criteria for persistent concurrent DSs is detectable recoverability , ensuring both thread safety (for correctness in non-crashing concurrent executions) and crash consistency (for correctness in crashing executions). However, the existing approaches to designing detectably recoverable concurrent DSs are either limited to simple algorithms or suffer from high runtime overheads. We present Memento: a general and high-performance programming framework for detectably recoverable concurrent DSs in PM. To ensure general applicability to various DSs, Memento supports primitive operations such as checkpoint and compare-and-swap and their composition with control constructs. To ensure high performance, Memento employs a timestamp-based recovery strategy that requires fewer writes and flushes to PM than the existing approaches. We formally prove that Memento ensures detectable recoverability in the presence of crashes. To showcase Memento, we implement a lock-free stack, list, queue, and hash table, and a combining queue that detectably recovers from random crashes in stress tests and performs comparably to existing hand-tuned persistent DSs with and without detectable recoverability.

DONUTS: An efficient method for checkpointing in non‐volatile memories

SummaryNon‐volatile memory (NVM) is an emerging technology being explored as an alternative to DRAM main memory in computing systems because of its persistence, higher storage density, lower energy consumption, and access latency close to DRAM. However, persistent memory systems must ensure data consistency on system failures, a property known as crash consistency. One of the main challenges in these systems is creating efficient checkpointing mechanisms in terms of performance and usability. Thus, it is necessary to remove persistence from the critical execution path and reduce the excessive number of writes to NVM caused by logging operations, resulting in increased memory bandwidth usage. Another limitation is that most proposed mechanisms restrict application source code to programming interfaces based on transactional models, typed as software‐based approaches. These factors make it challenging to adopt NVM systems. This article presents a software‐transparent mechanism based on dynamic epochs with logging operations via processing‐in‐memory and checkpoints integrated into the cache replacement policy. Compared to the previous best‐performing system, our strategy reduces 50.6% of writes to NVM. Furthermore, it does not increase the average memory bandwidth usage, providing crash consistency with less than 2% runtime overhead.

Efficient Crash Consistency for NVMe over PCIe and RDMA

This article presents crash-consistent Non-Volatile Memory Express (ccNVMe), a novel extension of the NVMe that defines how host software communicates with the non-volatile memory (e.g., solid-state drive) across a PCI Express bus and RDMA-capable networks with both crash consistency and performance efficiency. Existing storage systems pay a huge tax on crash consistency, and thus cannot fully exploit the multi-queue parallelism and low latency of the NVMe and RDMA interfaces. ccNVMe alleviates this major bottleneck by coupling the crash consistency to the data dissemination. This new idea allows the storage system to achieve crash consistency by taking the free rides of the data dissemination mechanism of NVMe, using only two lightweight memory-mapped I/Os (MMIOs), unlike traditional systems that use complex update protocol and synchronized block I/Os. ccNVMe introduces a series of techniques including transaction-aware MMIO/doorbell and I/O command coalescing to reduce the PCIe traffic as well as to provide atomicity. We present how to build a high-performance and crash-consistent file system named MQFS atop ccNVMe. We experimentally show that MQFS increases the IOPS of RocksDB by 36% and 28% compared to a state-of-the-art file system and Ext4 without journaling, respectively.

PLIN

Non-Volatile Memory (NVM) has emerged as an alternative to next-generation main memories. Although many tree indices have been proposed for NVM, they generally use B+-tree-like structures. To further improve the performance of NVM-aware indices, we consider integrating learned indexes into NVM. The challenges of such an integration are two fold: (1) existing NVM indices rely on small nodes to accelerate insertions with crash consistency, but learned indices use huge nodes to obtain a flat structure. (2) the node structure of learned indices is not NVM friendly, meaning that accessing a learned node will cause multiple NVM block misses. Thus, in this paper, we propose a new persistent learned index called PLIN. The novelty of PLIN lies in four aspects: an NVM-aware data placement strategy, locally unordered and globally ordered leaf nodes, a model copy mechanism, and a hierarchical insertion strategy. In addition, PLIN is proposed for the NVM-only architecture, which can support instant recovery. We also present optimistic concurrency control and fine-grained locking mechanisms to make PLIN scalable to concurrent requests. We conduct experiments on real persistent memory with various workloads and compare PLIN with APEX, PACtree, ROART, TLBtree, and Fast&Fair. The results show that PLIN achieves 2.08x higher insertion performance and 4.42x higher query performance than its competitors on average. Meanwhile, PLIN only needs ~30 μs to recover from a system crash.

Crash Consistency

Keeping data safe in the presence of crashes is a fundamental problem in storage systems. Although the high-level ideas for crash consistency are relatively well understood, realizing them in practice is surprisingly complex and full of challenges. The systems research community is actively working on solving this challenge, and the papers examined here offer three solutions.

SecNVM: An Efficient and Write-Friendly Metadata Crash Consistency Scheme for Secure NVM

Data security is an indispensable part of non-volatile memory (NVM) systems. However, implementing data security efficiently on NVM is challenging, since we have to guarantee the consistency of user data and the related security metadata. Existing consistency schemes ignore the recoverability of the SGX style integrity tree (SIT) and the access correlation between metadata blocks, thereby generating unnecessary NVM write traffic. In this article, we propose SecNVM, an efficient and write-friendly metadata crash consistency scheme for secure NVM. SecNVM utilizes the observation that for a lazily updated SIT, the lost tree nodes after a crash can be recovered by the corresponding child nodes in NVM. It reduces the SIT persistency overhead through a restrained write-back metadata cache and exploits the SIT inter-layer dependency for recovery. Next, leveraging the strong access correlation between the counter and DMAC, SecNVM improves the efficiency of security metadata access through a novel collaborative counter-DMAC scheme. In addition, it adopts a lightweight address tracker to reduce the cost of address tracking for fast recovery. Experiments show that compared to the state-of-the-art schemes, SecNVM improves the performance and decreases write traffic a lot, and achieves an acceptable recovery time.

Failure-Atomic Byte-Addressable R-tree for Persistent Memory

In this article, we propose Failure-atomic Byte-addressable R-tree (FBR-tree) that leverages the byte-addressability, persistence, and high performance of persistent memory while guaranteeing the crash consistency. We carefully control the order of store and cacheline flush instructions and prevent any single store instruction from making an FBR-tree inconsistent and unrecoverable. We also develop a non-blocking lock-free range query algorithm for FBR-tree. Since FBR-tree allows read transactions to detect and ignore any transient inconsistent states, multiple read transactions can concurrently access tree nodes without using shared locks while other write transactions are making changes to them. Our performance study shows that FBR-tree successfully reduces the legacy logging overhead and the lock-free range query algorithm shows up to 2.6x higher query processing throughput than the shared lock-based crabbing concurrency protocol.

A survey of operating system support for persistent memory

Emerging persistent memory technologies, like PCM and 3D XPoint, offer numerous advantages, such as higher density, larger capacity, and better energy efficiency, compared with the DRAM. However, they also have some drawbacks, e.g., slower access speed, limited write endurance, and unbalanced read/write latency. Persistent memory technologies provide both great opportunities and challenges for operating systems. As a result, a large number of solutions have been proposed. With the increasing number and complexity of problems and approaches, we believe this is the right moment to investigate and analyze these works systematically. To this end, we perform a comprehensive and in-depth study on operating system support for persistent memory within three steps. First, we present an overview of how to build the operating system on persistent memory from three perspectives: system abstraction, crash consistency, and system reliability. Then, we classify the existing research works into three categories: storage stack, memory manager, and OS-bypassing library. For each category, we summarize the major research topics and discuss these topics deeply. Specifically, we present the challenges and opportunities in each topic, describe the contributions and limitations of proposed approaches, and compare these solutions in different dimensions. Finally, we also envision the future operating system based on this study.

InK: In-Kernel Key-Value Storage with Persistent Memory

Block-based storage devices exhibit different characteristics from main memory, and applications and systems have been optimized for a long time considering the characteristics in mind. However, emerging non-volatile memory technologies are about to change the situation. Persistent Memory (PM) provides a huge, persistent, and byte-addressable address space to the system, thereby enabling new opportunities for systems software. However, existing applications are usually apt to indirectly utilize PM as a storage device on top of file systems. This makes applications and file systems perform unnecessary operations and amplify I/O traffic, thereby under-utilizing the high performance of PM. In this paper, we make the case for an in-Kernel key-value storage service optimized for PM, called InK. While providing the persistence of data at a high performance, InK considers the characteristics of PM to guarantee the crash consistency. To this end, InK indexes key-value pairs with B+ tree, which is more efficient on PM. We implemented InK based on the Linux kernel and evaluated its performance with Yahoo Cloud Service Benchmark (YCSB) and RocksDB. Evaluation results confirms that InK has advantages over LSM-tree-based key-value store systems in terms of throughput and tail latency.

TH-DPMS

The rapidly increasing data in recent years requires the datacenter infrastructure to store and process data with extremely high throughput and low latency. Fortunately, persistent memory (PM) and RDMA technologies bring new opportunities towards this goal. Both of them are capable of delivering more than 10 GB/s of bandwidth and sub-microsecond latency. However, our past experiences and recent studies show that it is non-trivial to build an efficient and distributed storage system with such new hardware. In this article, we design and implement TH-DPMS (<underline>T</underline>sing<underline>H</underline>ua <underline>D</underline>istributed <underline>P</underline>ersistent <underline>M</underline>emory <underline>S</underline>ystem) based on persistent memory and RDMA, which unifies the memory, file system, and key-value interface in a single system. TH-DPMS is designed based on a unified distributed persistent memory abstract, pDSM. pDSM acts as a generic layer to connect the PMs of different storage nodes via high-speed RDMA network and organizes them into a global shared address space. It provides the fundamental functionalities, including global address management, space management, fault tolerance, and crash consistency guarantees. Applications are enabled to access pDSM with a group of flexible and easy-to-use APIs by using either raw read/write interfaces or the transactional ones with ACID guarantees. Based on pDSM, we implement a distributed file system and a key-value store named pDFS and pDKVS, respectively. Together, they uphold TH-DPMS with high-performance, low-latency, and fault-tolerant data storage. We evaluate TH-DPMS with both micro-benchmarks and real-world memory-intensive workloads. Experimental results show that TH-DPMS is capable of delivering an aggregated bandwidth of 120 GB/s with 6 nodes. When processing memory-intensive workloads such as YCSB and Graph500, TH-DPMS improves the performance by one order of magnitude compared to existing systems and keeps consistent high efficiency when the workload size grows to multiple terabytes.

Crab-tree

In recent years, the next-generation non-volatile memory (NVM) technologies have emerged with DRAM-like byte addressability and disk-like durability. Computer architects have proposed to use them to build persistent memory that blurs the conventional boundary between volatile memory and non-volatile storage. However, ARM processors, ones that are widely used in embedded computing systems, start providing architectural supports to utilize NVM since ARMv8. In this article, we consider tailoring B+-tree for NVM operated by a 64-bit ARMv8 processor. We first conduct an empirical study of performance overhead in writing and reading data for a B+-tree with an ARMv8 processor, including the time cost of cache line flushes and memory fences for crash consistency as well as the execution time of binary search compared to that of linear search. We hence identify the key weaknesses in the design of B+-tree with ARMv8 architecture. Accordingly, we develop a new B+-tree variant, namely, <underline>c</underline> rash <underline>r</underline> ecoverable <underline>A</underline> RMv8-oriented <underline>B </underline>+-tree (Crab-tree). To insert and delete data at runtime, Crab-tree selectively chooses one of two strategies, i.e., copy on write and shifting in place, depending on which one causes less consistency cost. Crab-tree regulates a strict execution order in both strategies and recovers the tree structure in case of crashes. To further improve the performance of Crab-tree, we employ three methods to reduce software overhead, cache misses, and consistency cost, respectively. We have implemented and evaluated Crab-tree in Raspberry Pi 3 Model B+ with emulated NVM. Experiments show that Crab-tree significantly outperforms state-of-the-art B+-trees designed for persistent memory by up to 2.2× and 3.7× in write and read performances, respectively, with both consistency and scalability achieved.

NVGraph: Enforcing Crash Consistency of Evolving Network Analytics in NVMM Systems

Many complex networks can be modeled as evolving graphs. Existing in-memory graph data structures are designed for DRAM. Thus, they cannot effectively exploit the current and ongoing adoption of emerging non-volatile main memory (NVMM) for two reasons. (1) Ephemeral graph data structures are not crash-consistent for NVMM. (2) NVMM writes and reads and may incur higher latency than DRAM. In this article, we propose a novel persistent evolving graph data structure, named NVG RAPH , for both computing and in-memory storage of evolving graphs in NVMM. We devise NVG RAPH as a multi-version data structure, wherein a minimum of one version of its data is stored in NVMM to provide the desired durability at runtime for failure recovery, and another version is stored in both DRAM and NVMM to reduce the NVMM-induced memory latency. NVG RAPH is also a partitioned data structure. We dynamically transform the layout of NVG RAPH (e.g., changing the size of its partition in DRAM) exploiting network properties and data access patterns of workloads. For the evaluation of NVG RAPH , we implement four representative real-world graph applications: pagerank, BFS, influence maximization, and rumor source detection. The experimental results show that the performance of NVG RAPH is comparable to other in-memory data structure (e.g., CSR and LLAMA) while using 70 percent less DRAM. It scales well up to 10 billion edges and 201 snapshots and supports crash consistency. It offers up to the 21X speedup of execution time compared to the scale-up graph computation approaches (e.g., GraphChi and X-stream).

ShieldNVM

Data encryption and authentication are essential for secure non-volatile memory (NVM). However, the introduced security metadata needs to be atomically written back to NVM along with data, so as to provide crash consistency, which unfortunately incurs high overhead. To support fine-grained data protection and fast recovery for a secure NVM system without compromising the performance, we propose ShieldNVM. It first proposes an epoch-based mechanism to aggressively cache the security metadata in the metadata cache while retaining the consistency of them in NVM. Deferred spreading is also introduced to reduce the calculating overhead for data authentication. Leveraging the ability of data hash message authentication codes, we can always recover the consistent but old security metadata to its newest version. By recording a limited number of dirty addresses of the security metadata, ShieldNVM achieves fast recovering the secure NVM system after crashes. Compared to Osiris, a state-of-the-art secure NVM, ShieldNVM reduces system runtime by 39.1% and hash message authentication code computation overhead by 80.5% on average over NVM workloads. When system crashes happen, ShieldNVM’s recovery time is orders of magnitude faster than Osiris. In addition, ShieldNVM also recovers faster than AGIT, which is the Osiris-based state-of-the-art mechanism addressing the recovery time of the secure NVM system. Once the recovery process fails, instead of dropping all data due to malicious attacks, ShieldNVM is able to detect and locate the area of the tampered data with the help of the tracked addresses.

Making Application-Level Crash Consistency Practical on Flash Storage

We present the design, implementation, and evaluation of a new file system, called ACCFS , supporting application-level crash consistency as its first-class citizen functionality. With ACCFS , application data can be correctly recovered in the event of system crashes without any complex update protocol at the application level. With the help of the SHARE interface supporting atomic address remapping at the flash storage layer, ACCFS can easily and efficiently achieve crash consistency as well as single-write journaling. We prototyped ACCFS by slightly modifying the full data journal mode in ext4, implemented the SHARE interface as firmware in a commercial SSD available in the market, and carried out various experiments by running ACCFS on top of the SSD. Our preliminary experimental results are very promising. For instance, the performance of an OLTP benchmark using MySQL/InnoDB engine can be boosted by more than 2–6x by offloading the responsibility of guaranteeing the atomic write of MySQL data pages from the InnoDB engine's own journaling mechanism to ACCFS . This impressive performance gain is in part due to the single-write journaling in ACCFS and in part comes from the fact that the frequent fsync() calls caused by the complex update protocol at the application level can be avoided. ACCFS is a practical solution for the crash consistency problem in that (1) the SHARE interface can be, like the TRIM command, easily supported by commercial SSDs, (2) it can be embodied with a minor modification on the existing ext4 file system, and (3) the existing applications can be made crash consistent simply by opening files in O_ATOMIC mode while the legacy applications can be run without any change.

NV-Journaling: Locality-Aware Journaling Using Byte-Addressable Non-Volatile Memory

Modern file systems rely on the journaling mechanism to maintain crash consistency. The use of non-volatile memory (NVM) significantly improves the performance of journaling file systems. However, the superior performance of NVM will increase the likelihood of the journal filling up more often, thereby increasing the frequency of checkpointing. Together with the large amount of random checkpointing I/O found in most use cases, the checkpointing process becomes a new performance bottleneck. This paper proposes NV-Journaling, a strategy that reduces the frequency of checkpointing as well as reshapes the I/O pattern of checkpointing from one of random I/O to that which is more sequential I/O. NV-Journaling introduces fine-grained commits along with a cache-friendly NVM journaling layout that exploits the idiosyncrasies of NVM technology. Under this scheme, only the modified portion of a block, rather than the entire block, is written into the NVM journal device. Doing so significantly reduces checkpoint frequency and achieves better space utilization. NV-Journaling further reshapes the I/O pattern of checkpoint using a locality-aware checkpointing process. Checkpointed blocks are classified into hot and cold blocks. NV-Journaling maintains a hot block list to absorb repeated updates, and a cold bucket list to group blocks by their proximity on disk. When a checkpoint is required, cold buckets are selected such that blocks are sequentially flushed to the hard disk. We built a prototype of NV-Journaling by modifying the JBD2 layer in the Linux kernel and evaluated it using different workloads. Our experimental results show that NV-Journaling can improve performance by up to 4.3× compared to traditional journaling.

An Empirical Performance Evaluation of Transactional Solid-State Drives

Solid-state drives (SSDs) have accelerated the architectural evolution of storage systems with several characteristics (e.g., out-of-place update) compared with hard disk drives (HDD). Out-of-place update of SSDs naturally can support transaction mechanism which is commonly used in systems to provide crash consistency. Thus, transactional functionality has been recently implemented inside solid-state drives (SSDs). However, this approach must be re-evaluated for enterprise storage with a standard interface to investigate their benefits in a more realistic and standard fashion. In this article, we explore the implications and challenges of transactional SSDs with different experiments. To evaluate the potential benefit of transactional SSDs, we design and implement the transactional functionality in a Samsung enterprise-class and SATA-based SSD (i.e., SM843TN) called TxSSD. We modify the local file systems (i.e., ext4 and btrfs) and a distributed parallel file system (i.e., Lustre) to utilize TxSSDs. Our modified file systems with TxSSDs provide crash consistency without redundant writes. We evaluate our file systems by using multiple micro and macro benchmarks. We analyze the performance results and demonstrate that TxSSDs may generate an overhead for supporting transactional functionality inside SSD.