NVMe SSD Research Articles

Delay-D: Research on the Lifespan and Performance of Storage Devices in Unmanned Aerial Vehicles

Despite the technological achievements of unmanned aerial vehicles (UAVs) growing in academia and industry, there is a lack of studies on the storage devices in UAVs. However, this is an important aspect because the storage devices in UAVs have a limited lifespan and performance and are rarely replaced due to a system-on-chip architecture. In this paper, we study how UAVs impact the lifespan and performance of the underlying storage device while capturing images during overflight. We also propose a new lifespan and performance-saving mechanism, called Delay-D, which is designed at the kernel level to efficiently utilize the features of NAND flash-based storage devices. To confirm the effectiveness of Delay-D, we implement a simulator that replays realistic write patterns on UAVs and evaluate quantitative experiments in two different experimental environments. In our evaluation, Delay-D demonstrates the dramatic extension possibility of the lifespan by reducing the number of extra writes inside the storage device and improving the overall performance by up to 2.1× on the commercial NVMe SSD.

Fastensor: Optimise the Tensor I/O Path from SSD to GPU for Deep Learning Training

In recent years, benefiting from the increase in model size and complexity, deep learning has achieved tremendous success in computer vision (CV) and (NLP). Training deep learning models using accelerators such as GPUs often requires much iterative data to be transferred from NVMe SSD to GPU memory. Much recent work has focused on data transfer during the pre-processing phase and has introduced techniques such as multiprocessing and GPU Direct Storage (GDS) to accelerate it. However, tensor data during training (such as Checkpoints, logs, and intermediate feature maps), which is also time-consuming, is often transferred using traditional serial, long-I/O-path transfer methods. In this article, based on GDS technology, we built Fastensor, an efficient tool for tensor data transfer between the NVMe SSDs and GPUs. To achieve higher tensor data I/O throughput, we optimized the traditional data I/O process. We also proposed a data and runtime context-aware tensor I/O algorithm. Fastensor can select the most suitable data transfer tool for the current tensor from a candidate set of tools during model training. The optimal tool is derived from a dictionary generated by our adaptive exploration algorithm in the first few training iterations. We used Fastensor’s unified interface to test the read/write bandwidth and energy consumption of different transfer tools for different sizes of tensor blocks. We found that the execution efficiency of different tensor transfer tools is related to both the tensor block size and the runtime context. We then deployed Fastensor in the widely applicable Pytorch deep learning framework. We showed that Fastensor could perform superior in typical scenarios of model parameter saving and intermediate feature map transfer with the same hardware configuration. Fastensor achieves a 5.37x read performance improvement compared to torch.save () when used for model parameter saving. When used for intermediate feature map transfer, Fastensor can increase the supported training batch size by 20x, while the total read and write speed is increased by 2.96x compared to the torch I/O API.

DAC: A dynamic active and collaborative cache management scheme for solid state disks

NAND Flash memory-based SSDs have been widely used in modern storage systems to improve the performance and energy consumption. Existing caching management schemes rarely consider the utilization of parallel resources in flash memory array inside an SSD while these parallel resources cannot be fully exploited. The I/Os-access conflicts deteriorate the SSD performance and enlarge the response time under write-intensive workloads. Most cache replacement strategies give priority to clean data to improve the I/O-access conflicts, thereby boosting the SSD performance. However, clean data entirely depends on missed read-I/Os requests. Ideally, data replacement ought to evict clean data in the cache thanks to low replacement cost. As such, flash memory-based caching schemes give the highest priority to clean data. However, the proportion of replaced clean data is low in cache data. To solve this issue, we propose a dynamic active and collaborative cache management named DAC, in which the cache is composed of cold cache, hot cache, ghost cold cache, and ghost hot cache. DAC determines hot-cold changes of the I/O requests in workloads based on ghost cache size, thereby adjusting the real cache size to well serve I/O requests. The dynamic write-back window (DWW) mechanism dynamically adjusts the write-back window size in the cold cache. The write-back thresholds automatically update according to the changes in I/O patterns in workloads. When a flash chip is in the idle state, DAC produces clean data by proactively writing normal cold cache data back into flash memory, and the clean data is migrated into the active cold cache. DAC prioritizes replacing data in the active cold cache, aiming to avoid evicting data that are just read from flash memory. This method is adept at boosting the replacement rate of clean data, thereby optimizing SSD performance. We undertake extensive experiments in the latest NVMe SSD simulator to validate the DAC’s edge over the seven existing caching schemes including LRU, CFLRU, GCaR-CFLRU, LCR, ARC, AD-LRU, and MQsim. The results unveil that our DAC shortens the response time by up to 60.8% with an average optimization rate of 24.4%. When it comes to response-time cliff, the maximum improvement offered by DAC reaches approximately 68.71%; meanwhile, DAC delivers an average optimization rate of 26.9%. Besides, the number of erase count reduces by up to 93.3% with an average improvement rate of 17.04%.

FragTracer: Real-Time Fragmentation Monitoring Tool for F2FS File System.

Emerging hardware devices (e.g., NVMe SSD, RISC-V, etc.) open new opportunities for improving the overall performance of computer systems. In addition, the applications try to fully utilize hardware resources to keep up with those improvements. However, these trends can cause significant file system overheads (i.e., fragmentation issues). In this paper, we first study the reason for the fragmentation issues on an F2FS file system and present a new tool, called FragTracer, which helps to analyze the ratio of fragmentation in real-time. For user-friendly usage, we designed FragTracer with three main modules, monitoring, pre-processing, and visualization, which automatically runs without any user intervention. We also optimized FragTracer in terms of performance to hide its overhead in tracking and analyzing fragmentation issues on-the-fly. We evaluated FragTracer with three real-world databases on the F2FS file system, so as to study the fragmentation characteristics caused by databases, and we compared the overhead of FragTracer. Our evaluation results clearly show that the overhead of FragTracer is negligible when running on commodity computing environments.

Realizing Strong Determinism Contract on Log-Structured Merge Key-Value Stores

We propose Vigil-KV , a hardware and software co-designed framework that eliminates long-tail latency almost perfectly by introducing strong latency determinism. To make Get latency deterministic, Vigil-KV first enables a predictable latency mode (PLM) interface on a real datacenter-scale NVMe SSD, having knowledge about the nature of the underlying flash technologies. Vigil-KV at the system-level then hides the non-deterministic time window (associated with SSD’s internal tasks and/or write services) by internally scheduling the different device states of PLM across multiple physical functions. Vigil-KV further schedules compaction/flush operations and client requests being aware of PLM’s restrictions thereby integrating strong latency determinism into LSM KVs. We implement Vigil-KV upon a 1.92TB NVMe SSD prototype and Linux 4.19.91, but other LSM KVs can adopt its concept. We evaluate diverse Facebook and Yahoo scenarios with Vigil-KV, and the results show that Vigil-KV can reducethe tail latency of a baseline KV system by 3.19× while reducing the average latency by 34%, on average.

When Database Meets New Storage Devices: Understanding and Exposing Performance Mismatches via Configurations

NVMe SSD hugely boosts the I/O speed, with up to GB/s throughput and microsecond-level latency. Unfortunately, DBMS users can often find their high-performanced storage devices tend to deliver less-than-expected or even worse performance when compared to their traditional peers. While many works focus on proposing new DBMS designs to fully exploit NVMe SSDs, few systematically study the symptoms, root causes and possible detection methods of such performance mismatches on existing databases. In this paper, we start with an empirical study where we systematically expose and analyze the performance mismatches on six popular databases via controlled configuration tuning. From the study, we find that all six databases can suffer from performance mismatches. Moreover, we conclude that the root causes can be categorized as databases' unawareness of new storage devices characteristics in I/O size, I/O parallelism and I/O sequentiality. We report 17 mismatches to developers and 15 are confirmed. Additionally, we realize testing all configuration knobs yields low efficiency. Therefore, we propose a fast performance mismatch detection framework and evaluation shows that our framework brings two orders of magnitude speedup than baseline without sacrificing effectiveness.

TomocuPy - efficient GPU-based tomographic reconstruction with asynchronous data processing.

Fast 3D data analysis and steering of a tomographic experiment by changing environmental conditions or acquisition parameters require fast, close to real-time, 3D reconstruction of large data volumes. Here a performance-optimized TomocuPy package is presented as a GPU alternative to the commonly used central processing unit (CPU) based TomoPy package for tomographic reconstruction. TomocuPy utilizes modern hardware capabilities to organize a 3D asynchronous reconstruction involving parallel read/write operations with storage drives, CPU-GPU data transfers, and GPU computations. In the asynchronous reconstruction, all the operations are timely overlapped to almost fully hide all data management time. Since most cameras work with less than 16-bit digital output, the memory usage and processing speed are furthermore optimized by using 16-bit floating-point arithmetic. As a result, 3D reconstruction with TomocuPy became 20-30 times faster than its multi-threaded CPU equivalent. Full reconstruction (including read/write operations and methods initialization) of a 20483 tomographic volume takes less than 7 s on a single Nvidia Tesla A100 and PCIe 4.0 NVMe SSD, and scales almost linearly increasing the data size. To simplify operation at synchrotron beamlines, TomocuPy provides an easy-to-use command-line interface. Efficacy of the package was demonstrated during a tomographic experiment on gas-hydrate formation in porous samples, where a steering option was implemented as a lens-changing mechanism for zooming to regions of interest.

CFIO: A conflict-free I/O mechanism to fully exploit internal parallelism for Open-Channel SSDs

I/O access conflicts make utilization within NVMe SSDs seriously low, which introduces unpredictable performance loss of NVMe SSDs. Although existing works adopt I/O isolation or conflict-aware I/O scheduling to avoid access conflicts, they can result in an unbalanced utilization and reduce the lifetime of NVMe SSDs. In this paper, we design and implement CFIO, a low-overhead conflict-aware I/O mechanism that achieves conflict-free I/Os to exploit the internal parallelism in NVMe SSDs. CFIO improves PU utilization and reduces I/O latency with two novel mechanisms. First, a conflict-free (CF) lane is proposed to eliminate conflicts by dividing I/O requests into conflict-free PU queues based on physical addresses. The PU queues correspond to the PU resources within the NVMe SSDs. Second, a k-RR scheduler is designed to dispatch reading and writing requests to NVMe SSDs in batches and separately. K-RR scheduler can fully exploit the internal parallelism of NVMe SSDs and form an I/O pipeline based on the dual registers of PU. Finally, we integrate CFIO into the LightNVM with Open-Channel NVMe SSD (OCSSD) and compare it with several existing solutions. Our evaluations show that CFIO improves the throughput of OCSSD by 19.32% and reduces its tail latency by 23.71%, compared to state-of-the-art methods.

Evaluation of gem5 for performance modeling of ARM Cortex-R based embedded SoCs

ARM CPUs are prevalent in embedded systems ranging from low-power IoT to reasonably high-powered mobile phones and devices. Embedded SoCs integrate a number of accelerators with CPUs to achieve the tight performance and power budget. To overcome the challenges of architecting these complex SoCs, designers employ performance simulators as a design space exploration tool. Developing such a simulator infrastructure in itself is challenging and depends on accurately modeling the ARM CPUs and the interaction of various accelerators with the CPU. The widely used, open-source gem5 simulator provides the necessary components to develop such an infrastructure in a time and cost-effective manner. In this paper, we present the limit study and evaluation of using gem5 for building a performance modeling and analysis framework targeted to ARM R series CPU-based embedded SoC. First, we illustrate the approach to develop the framework by tuning the extensive configuration space of gem5 and making modifications for new features. We consider different CPUs, branch predictors, memory modules, and SystemC coupling in gem5. Second, we perform a detailed correlation of the simulator with a real ARM CPU and explain the shortcomings, challenges, and ways to improve the accuracy. We obtain average absolute errors of 13% for the Embench suite Bennett et al. (2019). We also point out large miscorrelations for specific pattern in ALU (50%), Branch (135%) and Memory (35%). Finally, through a case study to design a flexible NVMe SSD controller architecture, we demonstrate the framework’s design space exploration capabilities.

Design of a multi-function high-speed digital baseband data acquisition system

A multi-function digital baseband data acquisition system is designed for the sampling, distribution and recording of wide-band multi-channel astronomical signals. The system hires a SNAP2 board as a digital baseband converter to digitize, channelize and packetize the received signal. It can be configured dynamically from a single channel to eight channels with a maximum bandwidth of 4096 MHz. Eight parallel HASHPIPE instances run on four servers, each carrying two NVMe SSD cards, achieving a total continuous write rate of 8 GB s−1. Data are recorded in the standard VDIF file format. The system is deployed on a 25-meter radio telescope to verify its functionality based on pulsar observations. Our results indicate that during the 30-minute observation period, the system achieved zero data loss at a data recording rate of 1 GB s−1 on a single server. The system will serve as a verification platform for testing the functions of the QTT (QiTai radio Telescope) digital backend system. In addition, it can be used as a baseband/VLBI (Very Long Baseline Interferometry) recorder or D-F-engine of correlator/beamformer as well.

Leveraging NVMe SSDs for Building a Fast, Cost-effective, LSM-tree-based KV Store

Key-value (KV) stores support many crucial applications and services. They perform fast in-memory processing but are still often limited by I/O performance. The recent emergence of high-speed commodity non-volatile memory express solid-state drives (NVMe SSDs) has propelled new KV system designs that take advantage of their ultra-low latency and high bandwidth. Meanwhile, to switch to entirely new data layouts and scale up entire databases to high-end SSDs requires considerable investment. As a compromise, we propose SpanDB, an LSM-tree-based KV store that adapts the popular RocksDB system to utilize selective deployment of high-speed SSDs . SpanDB allows users to host the bulk of their data on cheaper and larger SSDs (and even hard disc drives with certain workloads), while relocating write-ahead logs (WAL) and the top levels of the LSM-tree to a much smaller and faster NVMe SSD. To better utilize this fast disk, SpanDB provides high-speed, parallel WAL writes via SPDK, and enables asynchronous request processing to mitigate inter-thread synchronization overhead and work efficiently with polling-based I/O. To ease the live data migration between fast and slow disks, we introduce TopFS, a stripped-down file system providing familiar file interface wrappers on top of SPDK I/O. Our evaluation shows that SpanDB simultaneously improves RocksDB's throughput by up to 8.8 \times and reduces its latency by 9.5–58.3%. Compared with KVell, a system designed for high-end SSDs, SpanDB achieves 96–140% of its throughput, with a 2.3–21.6 \times lower latency, at a cheaper storage configuration.

Remote Access NVMe SSD via NTB

NVMe SSDs are deployed in data centers for applications with high performance, but its capacity and bandwidth are often underutilized. Remote access NVMe SSD enables flexible scaling and high utilization of Flash capacity and bandwidth within data centers. The current issue of remote access NVMe SSD has significant performance overheads. The research focuses on remote access NVMe SSD via NTB (non-transparent bridge). NTB is a type of PCI-Express; its memory mapping technology can allow to access memory belonging to peer servers. NVMe SSD supports multiple I/O queues to maximize the I/O parallel processing of flash; hence, NVMe SSD can provide significant performance when comparing with traditional hard drives. The research proposes a novel design based on features of NTB memory mapping and NVMe SSD multiple I/O queues. The remote and local servers can access the same NVMe SSD concurrently. The experimental results show the performance of remote access NVMe SSD can approach the local access. It is significantly excellent and proved feasible.

Stabilizing and boosting I/O performance for file systems with journaling on NVMe SSD

Stabilizing and boosting I/O performance for file systems with journaling on NVMe SSD

OurRocks: Offloading Disk Scan Directly to GPU in Write-Optimized Database System

The log structured merge (LSM) tree has been widely adopted by database systems owing to its superior write performance. However, LSM-tree based databases face vulnerabilities when processing analytical queries due to the read amplification caused by its architecture and the limited use of storage devices with high bandwidth. To flexibly handle transactional and analytical workloads, we proposed and implemented OurRocks taking full advantage of NVMe SSD and GPU devices, which improves scan performance. Although the NVMe SSD serves multi GB/s I/O rates, it is necessary to solve the data transfer overhead which limits the benefits of the GPU processing. The primary idea is to offload the scan operation to the GPU with filtering predicate pushdown and resolve the bottleneck from the data transfer between devices with direct memory access (DMA). OurRocks benefits from all the features of write-optimized database systems, in addition to accelerating the analytic queries using the aforementioned idea. Experimental results indicate that OurRocks effectively leverages resources of the NVMe SSD and GPU and significantly improves the execution of queries in the YCSB and TPC-H benchmarks, compared to the conventional write-optimized database. Our research demonstrates that the proposed approach can speed up the handling of the data-intensive workloads.

Live forensics method for acquisition on the Solid State Drive (SSD) NVMe TRIM function

SSD currently has a new storage media technology namely Solid State Drive Non-volatile Memory Express (SSD NVMe). In addition, SSD has a feature called TRIM. The TRIM feature allows the operating system to tell SSDs which blocks are not used. TRIM removes blocks that have been marked for removal by the operating system. However, the TRIM function has a negative effect for the digital forensics specifically related to data recovery. This study aimed to compare the TRIM disable and enable functions to determine the ability of forensics tools and recovery tools to restore digital evidence on the NVMe SSD TRIM function. The operating system used in this study was Windows 10 professional with NTFS file system. Typically, acquisition is conducted by using traditional or static techniques. Therefore, there was a need of a technique to acquire SSD by using the live forensics method without shutting down the running operating system. The live forensics method was applied to acquire SSD NVMe directly to the TRIM disable and enable functions. The tools used for live acquisition and recovery were FTK Imager Portable. The inspection and analysis phases used Sleutkit Autopsy and Belkasoft Evidence Center. This research found that in the recovery process of TRIM disabled and enabled, TRIM disabled could find evidence while maintaining the integrity of evidence. It was indicated by the same hash value of the original file and the recovery file. Conversely, when TRIM is enabled, the files were damaged and could not be recovered. The files were also not identical to the original so the integrity of evidence was not guaranteed.

Live Forensics Method for Acquisition on the Solid State Drive (SSD) NVMe TRIM Function

SSD currently has a new storage media technology namely Solid State Drive Non-volatile Memory Express (SSD NVMe). In addition, SSD has a feature called TRIM. The TRIM feature allows the operating system to tell SSDs which blocks are not used. TRIM removes blocks that have been marked for removal by the operating system. However, the TRIM function has a negative effect for the digital forensics specifically related to data recovery. This study aimed to compare the TRIM disable and enable functions to determine the ability of forensics tools and recovery tools to restore digital evidence on the NVMe SSD TRIM function. The operating system used in this study was Windows 10 professional with NTFS file system. Typically, acquisition is conducted by using traditional or static techniques. Therefore, there was a need of a technique to acquire SSD by using the live forensics method without shutting down the running operating system. The live forensics method was applied to acquire SSD NVMe directly to the TRIM disable and enable functions. The tools used for live acquisition and recovery were FTK Imager Portable. The inspection and analysis phases used Sleutkit Autopsy and Belkasoft Evidence Center. This research found that in the recovery process of TRIM disabled and enabled, TRIM disabled could find evidence while maintaining the integrity of evidence. It was indicated by the same hash value of the original file and the recovery file. Conversely, when TRIM is enabled, the files were damaged and could not be recovered. The files were also not identical to the original so the integrity of evidence was not guaranteed.

NVMe SSD 기반 캐시를 통한 병렬파일시스템 특징 및 성능 분석

NVMe는 PCI-Express 기반의 비휘발성 저장장치로 기존 SATA 방식에 비해 높은 대역폭을 갖는 고속 메모리로의 역할을 수행한다. 최근 대규모 클러스터 환경에서는 병렬파일시스템과 클러스터 노드 사이에 NVMe SSD 기반의 스토리지 계층을 구성하여 고대역폭의 I/O가 요구되는 응용 프로그램의 캐시역할을 수행하도록 하고 있다. 이는 기존 파일시스템에서 발생할 수 있는 클러스터 노드의 규모가 커짐에 따른 I/O 대역폭의 증가를 감내할 수 있고 , I/O 집중형 응용 프로그램이 클러스터 내에 동시에 몰려 특정 순간에 부하가 발생하는 경우의 성능 부하를 분산시키는 역할을 한다.BR 본 연구에서는 병렬파일시스템과 NVMe SSD 기반 캐시 계층의 구성 방법과 특징을 논의하고 벤치마크 도구를 사용하여 각각의 I/O 성능을 파악한다. 이로써 응용 프로그램의 프로세스 단위, 파일크기, 확장성 등에 따른 I/O 성능의 특징을 확인할 수 있으며 I/O 의존성이 높은 사용자의 성능 향상을 제시할 수 있다.

Efficient and Scalable External Sort Framework for NVMe SSD

As the size of data grows in modern applications, the efficient usage of limited resources is becoming crucial. In order to reorganize large data under memory limitations, many data-intensive applications utilize external sort as a critical component. To streamline external sort, a storage framework is especially needed since the entire dataset must be loaded and flushed a couple of times during the sorting process. Most existing frameworks have attempted to simplify the storage access pattern by associating each thread with a separate storage device. This prevents randomized and concurrent I/O requests, which impose a huge overhead for legacy drives in order to enable the parallelism needed for external sort. However, such regulations excessively restrain the capabilities of NVMe-based SSDs that deliver high throughput with abundant parallelism. In this article, we present a new framework for external sort that exploits both external and internal parallelism. Externally, any number of threads are mobilized to parallel external sort in a scalable way, even with one NVMe SSD. Meanwhile, some arbitration schemes, such as adaptive resource allocation and fairness control, are adopted to preserve the internal efficiency of storage devices. Our evaluation results demonstrate that our scheme can greatly improve both the I/O efficiency and scalability compared to the existing frameworks.

Clip: A Disk I/O Focused Parallel Out-of-Core Graph Processing System

Existing parallel out-of-core graph processing systems focus on improving disk I/O locality, which leads to restrictions on their programming models. Although improving the locality, these constraints also restrict the expressiveness and hence only sub-optimal algorithms are supported. These sub-optimal algorithms typically incur sequential, but much larger, amount of disk I/O. In this paper, we explore a fundamentally different tradeoff: less total amount of I/O rather than better locality. We show that out-of-core graph processing systems uniquely provide the opportunities to lift the restrictions of the programming model in a feasible manner. To demonstrate the ideas, we build Clip, which enables more efficient algorithms that require much less amount of total disk I/O. Our experiments show that the algorithms that can be only implemented in Clip are much faster than the original disk-locality-optimized algorithms. We also further extend our technique's scope of application by providing a semi-external mode. Our analysis and evaluation demonstrate that semi-external is not only feasible for many cases, but also be able to deliver a significant speedup for important graph applications. Moreover, we further improve the performance of originally supported applications by designing more optimizations and evaluate our system on NVMe SSD.

VirtIO-trace: 가상화 환경에서 NVMe SSD 입출력 특성 분석을 위한 통합 도구

가상화 기술은 클라우드 컴퓨팅, 데이터 센터 등 다양한 컴퓨팅 환경에서 핵심 기술로 사용되고 있다. CPU 제조사들은 가상화 시스템의 성능 향상을 위하여 적극적으로 하드웨어 기반의 가상화 지원기술을 제공하고 있으며, NIC 제조사들도 네트워크 I/O 성능 향상을 위한 가상화 기술을 제공한다. 하지만 가상화 환경에서 저장장치 I/O는 여전히 성능 저하 문제를 겪고 있으며, 이를 해결하기 위한 많은 연구들이 진행되고 있다. 최근에는 가상화 저장장치로 NVMe(Non-Volatile Memory express) SSD(Solid State Drive)를 적용하여 고성능의 가상화 I/O 시스템을 구축한다. 하지만 기대만큼의 I/O 성능을 보여주지 못하기 때문에 I/O 최적화 또는 개발이 필요하며 이때 I/O 정보를 추적하고 분석할 수 있는 도구를 필요로 한다. 본 논문에서는 가상화 환경에서 NVMe SSD를 저장 장치로 사용할 때 I/O를 효율적으로 추적하고, 수집된 정보를 토대로 I/O 특성을 분석할 수 있는 도구를 제안한다. 제안된 도구는 VirtIO-trace라고 불리며, 기존의 blktrace 도구처럼 I/O 요청 정보와 시간 정보를 기본적으로 제공한다. 한편 이 도구는 기존 blktrace와는 다르게, NVMe SSD 특화 정보 (큐 정보, I/O 제출/완료 정보 등)와 가상화 관련 정보(가상 머신 상에서 I/O 처리시간, 가상화 SW 상에서 I/O 처리시간 등)도 제공한다는 차별성이 있다. 실제 KVM 환경에서 구현하고 실험을 수행한 결과 I/O 정보를 실시간으로 잘 수집하며, 분석 정보는 NVMe SSD에 대한 가상화 I/O 특성 분석 및 성능 향상 기법 개발에 효과적으로 사용할 수 있음을 검증하였다.