Storage Stack Research Articles

Check-QZP: A Lightweight Checkpoint Mechanism for Deep Learning Frameworks

In deep learning (DL) frameworks, a checkpoint operation is widely used to store intermediate variable values (e.g., weights, biases, and gradients) on storage media. This operation helps to reduce the recovery time of running a machine learning (ML) model after sudden power failures or random crashes. However, the checkpoint operation can stall the overall training step of the running model and waste expensive hardware resources by leaving the GPU in idle sleep during the checkpoint operation. In addition, the completion time of the checkpoint operation is unpredictable in cloud server environments (e.g., AWS and Azure) because excessive I/O operations issued by other running applications interfere with the checkpoint operations in the storage stacks. To efficiently address the above two problems, we carefully designed Check-QZP, which reduces the amount of data required for checkpoint operations and parallelizes executions on the CPU and GPU by understanding the internal behaviors of the training step. For the evaluation, we implemented Check-QZP and compared it with the traditional approach in real-world multi-tenant scenarios. In the evaluation, Check-QZP outperformed the baseline in all cases in terms of the overall checkpoint time and the amount of data generated by the checkpoint operations, reducing them by up to 87.5% and 99.8%, respectively. In addition, Check-QZP achieved superior training speeds compared to the baseline.

An Evaluation of DAOS for Simulation and Deep Learning HPCWorkloads

Traditionally, distributed storage systems have relied upon the interfaces provided by OS kernels to interact with storage hardware. However, much research has shown that OSes impose serious overheads on every I/O operation, especially on high-performance storage and networking hardware (e.g., PMEM and 200GBe). Thus, distributed storage stacks are being re-designed to take advantage of this modern hardware by utilizing new hardware interfaces which bypass the kernel entirely. However, the impact of these optimizations have not been well-studied for real HPC workloads on real hardware. In this work, we provide a comprehensive evaluation of DAOS: a state-of-the-art distributed storage system which re-architects the storage stack from scratch for modern hardware.We compare DAOS against traditional storage stacks and demonstrate that by utilizing optimal interfaces to hardware, performance improvements of up to 6x can be observed in real scientific applications.

Role of Correlation among Physical Factors in Probabilistic Simulation of Emissions of Volatile Organic Compounds from Floating Storage and Offloading Vent Stack

This research investigated the roles of correlations among physical factors in the probabilistic simulation of volatile organic compounds (VOCs) emitted from a marine vessel (known as floating storage and offloading, FSO), located in the Gulf of Thailand. The physical factors in this study were wave height, ambient temperature, storage temperature, storage quantity, Reid vapor pressure, and the daily incoming rate. These physical factors were transformed into normally distributed data and a second-order multiple linear regression (MLR) with interaction effects, that were then used to determine the relationship between the transformed physical factors and the VOC venting volume from the FSO. The dataset of relevant predictors (transformed physical factors and interactions) that provided the maximum adjusted coefficient of determination was chosen for inclusion in the MLR. After that, two datasets of 1,000 venting volumes (one with and one without correlations among physical factors) were simulated. In the simulation, 1,000 datasets of six physical factors were generated according to observed averages and standard deviations. Cholesky randomization was used to generate the correlated physical factors for the simulation with correlation among physical factors. The averages of VOC venting volumes calculated from the generated physical factors when correlations among physical factors were and were not applied were 211,610 and 210,906 ft3, respectively (observed average was 210,984 ft3), with standard deviations of 38,828 and 40,787 ft3, respectively (observed standard deviation was 67,961 ft3), and skewness values of 0.74 and 0.51, respectively (observed skewness was 0.71). Therefore, correlation among the physical factors improved the skewness and provided better simulation results for VOC emission.

Index Shipping for Efficient Replication in LSM Key-Value Stores with Hybrid KV Placement

Key-value (KV) stores based on the LSM tree have become a foundational layer in the storage stack of datacenters and cloud services. Current approaches for achieving reliability and availability favor reducing network traffic and send to replicas only new KV pairs. As a result, they perform costly compactions to reorganize data in both the primary and backup nodes, which increases device I/O traffic and CPU overhead, and eventually hurts overall system performance. In this article, we describe Tebis , an efficient LSM-based KV store that reduces I/O amplification and CPU overhead for maintaining the replica index. We use a primary-backup replication scheme that performs compactions only on the primary nodes and sends pre-built indexes to backup nodes, avoiding all compactions in backup nodes. Our approach includes an efficient mechanism to deal with pointer translation across nodes in the pre-built region index. Our results show that Tebis reduces resource utilization on backup nodes compared to performing full compactions: throughput is increased by 1.06 to 2.90×, CPU efficiency is increased by 1.21 to 2.78×, and I/O amplification is reduced by 1.7 to 3.27×, whereas network traffic increases by up to 1.32 to 3.76x.

Caiti: I/O transit caching for persistent memory-based block device

Byte-addressable non-volatile memory (NVM) sitting on the memory bus is employed to make persistent memory (pmem) in general-purpose computing systems and embedded systems for data storage. Researchers develop software drivers such as the block translation table (BTT) to build block devices on pmem, so programmers can keep using mature and reliable conventional storage stack while expecting high performance by exploiting fast pmem. However, our quantitative study shows that BTT underutilizes pmem and yields inferior performance, due to the absence of the imperative in-device cache. We add a conventional I/O staging cache made of DRAM space to BTT. As DRAM and pmem have comparable access latency, I/O staging cache is likely to be fully filled over time. Continual cache evictions and fsyncs thus cause on-demand flushes with severe stalls, such that the I/O staging cache is concretely unappealing for pmem-based block devices. We accordingly propose an algorithm named Caiti with novel I/O transit caching. Caiti eagerly evicts buffered data to pmem through CPU’s multi-cores. It also conditionally bypasses a full cache and directly writes data into pmem to further alleviate I/O stalls. Experiments confirm that Caiti significantly boosts the performance with BTT by up to 3.6×, without loss of block-level write atomicity.

Disaggregating RocksDB: A Production Experience

As in the general industry, there is a trend in Meta's data centers to migrate data from locally attached SSDs to cloud storage. We extended RocksDB [26], a widely used open-source storage engine designed and built for local SSDs, to leverage disaggregated storage. RocksDB's design, such as its data and log files' access patterns, makes an append-only distributed file system a desirable underlying storage. At Meta, we built disaggregated RocksDB using Tectonic File System [35], which so far had mainly been used for our data warehouse and blob storage stacks. We identified that metadata overhead and tail latencies were Tectonic's major performance gaps and addressed them accordingly. We improved the reliability, performance and other requirements with both general and customized optimizations to the core engine in RocksDB. We also took the time to deeply understand the common challenges presented by applications running on RocksDB and implemented enhancements to address them. This architecture enabled RocksDB to adapt to a more distributed architecture for performance enhancements.

Energy, Volume and Cost Analyses of High Temperature Seasonal Thermal Storage for Plus Energy House

The article analyzes the impact of different filling materials for a seasonal heat storage stack that can be used to heat an innovative plus-energy house in Poland. The storage medium is of the sensible heat type. Twelve filling materials and nine thermal insulation materials readily available in the local market were analyzed. Seven variants comprising a combination of the stack’s filling and thermal insulation materials were selected and then grouped into three classes: advanced, medium, and simple technology. Using a mathematical model, energy analysis of the year-round operation of the HVAC installation in the analyzed building was performed for each of the seven variants. The conducted analyses revealed that for each combination of filling and thermal insulation material, there is an optimal maximum temperature of the stack, at which the volume of the stack is the smallest or its costs are the lowest. The obtained results were evaluated to determine the ideal variant combination, and two solutions were recommended: clinker brick and fireplace wool, for which the stack volume is 23 m3 and the total cost is EUR 12,500; and concrete block and glass wool, for which the stack volume is 27 m3 and the total cost is EUR 1700.

A Simulation of a Sustainable Plus-Energy House in Poland Equipped with a Photovoltaic Powered Seasonal Thermal Storage System

This article describes the innovative photovoltaic powered seasonal thermal storage—PVPSTS system. It was used in the design of a plus-energy detached single-family house with a usable area of 98 m2. This area meets the requirements of the latest building regulations in Poland. The building, with the innovative HVAC installation, was subjected to energy analysis, and a numerical model was also developed. The model was tested based on TMY data for the location of Wroclaw, Poland. Analysis of the results allowed the authors to learn the specifics of the operation of the system throughout the year and to also define its efficiency. The required size of the storage stack was determined to be 1.6 × 1.6 × 0.3 m. The photovoltaic installation, which was integrated with the roof, can produce 48 GJ of electricity per year. This is five to six times more than the building’s heating needs, and any excess energy can be exported to the power grid.

Dotori: A Key-Value SSD Based KV Store

Key-value SSDs (KVSSDs) represent a major shift in the storage stack design, with numerous potential benefits. Despite this, their lack of native features critical to operation in real world scenarios hinders their adoption, and these benefits go unrealized. Moreover, simply adapting existing key-value stores to run on KVSSDs proves underwhelming, as KVSSDs operate at lower raw device performance when compared to modern block SSDs. This paper introduces Dotori. Dotori is a KVSSD based key-value store that provides much needed functionality in a KVSSD through an upper layer in the host, and takes advantage of the unique KVSSD interface to enable further gains in functionality and performance. At the core of Dotori is a novel B+tree design that is only practical when the underlying storage device is a KVSSD. We test Dotori with an enterprise grade KVSSD against state-of-the-art block SSD based key-value stores through a range of micro-benchmarks and real world workloads. Despite low KVSSD raw device performance, Dotori achieves superior performance to these block-device based key-value stores while also showing significant gains in other important metrics.

储能堆供电充电桩的研究

储能堆供电充电桩的研究

Coordinated Control of Hybrid DERs Enabled Grid-Interactive Residential PCM With Hybrid Bus Layout

This article proposes a multi-terminal grid interactive power conditioning module (PCM) for integration of hybrid distributed energy resources (DERs) within single phase residential premises, along with the Energy Storage Stacks (ESSs). The PCM also features hybrid bus layouts with medium-low voltage dc (MVdc) and low voltage dc (LVdc), for dc load integrations, and ac bus layout for single phase ac loads. The bi-directional MVdc and LVdc bus facilitate the effective power management and bus voltage support. It operates in all grid feeding, forming and supportive modes. The ESSs includes battery for system energy management and the supercapacitor for dc bus stabilization during transient conditions. The coordinated control of the PCM is capable of interfacing multiple DERs, dc/ac loads, and ESS. The controllers in the PCM are self-adaptive to the frequent islanding situations and ensure the system stability with minimum use of sensory feedbacks. The PCM has loop type dc bus feeder to make residential system operative during transient/faulty conditions. Simulation results verify the feasibility of the proposed system in all active and passive grid feeding; and grid forming modes and experimental setup is used to test the performance of the proposed PCM control through a single microcontroller.

An Enterprise-Grade Open-Source Data Reduction Architecture for All-Flash Storage Systems

Data reduction technologies have proven their effectiveness to decrease the ever-growing demands on storage system capacities, but also introduce new complexity in the system I/O stack that can easily invalidate well-known best practices. In this paper, we conduct an extensive set of experiments on an enterprise all-flash storage (AFS) system equipped with an open-source data reduction module, i.e., RedHat VDO, and reveal novel observations on the performance gap between the state-of-the-art and the optimal AFS stack with integrated data reduction. We then offer cross-layer optimizations to enhance the performance of AFS, which range from deriving new optimal hardware RAID configurations up to modifications of the enterprise storage stack tailored to the major bottlenecks observed. By implementing all proposed optimizations in an enterprise AFS, we show up to 12.5x speedup over the baseline AFS with integrated data reduction, and up to 57x performance/cost improvement over an optimized AFS (with no data reduction) for 100% write, high-reduction workload scenarios.

An Enterprise-Grade Open-Source Data Reduction Architecture for All-Flash Storage Systems

All-flash storage (AFS) systems have become an essential infrastructure component to support enterprise applications, where sub-millisecond latency and very high throughput are required. Nevertheless, the price per capacity ofsolid-state drives (SSDs) is relatively high, which has encouraged system architects to adoptdata reduction techniques, mainlydeduplication andcompression, in enterprise storage solutions. To provide higher reliability and performance, SSDs are typically grouped usingredundant array of independent disk (RAID) configurations. Data reduction on top of RAID arrays, however, adds I/O overheads and also complicates the I/O patterns redirected to the underlying backend SSDs, which invalidates the best-practice configurations used in AFS. Unfortunately, existing works on the performance of data reduction do not consider its interaction and I/O overheads with other enterprise storage components including SSD arrays and RAID controllers. In this paper, using a real setup with enterprise-grade components and based on the open-source data reduction module RedHat VDO, we reveal novel observations on the performance gap between the state-of-the-art and the optimal all-flash storage stack with integrated data reduction. We therefore explore the I/O patterns at the storage entry point and compare them with those at the disk subsystem. Our analysis shows a significant amount of I/O overheads for guaranteeing consistency and avoiding data loss through data journaling, frequent small-sized metadata updates, and duplicate content verification. We accompany these observations with cross-layer optimizations to enhance the performance of AFS, which range from deriving new optimal hardware RAID configurations up to introducing changes to the enterprise storage stack. By analyzing the characteristics of I/O types and their overheads, we propose three techniques: (a) application-aware lazy persistence, (b) a fast, read-only I/O cache for duplicate verification, and (c) disaggregation of block maps and data by offloading block maps to a very fast persistent memory device. By consolidating all proposed optimizations and implementing them in an enterprise AFS, we show 1.3× to 12.5× speedup over the baseline AFS with 90% data reduction, and from 7.8× up to 57× performance/cost improvement over an optimized AFS (with no data reduction) running applications ranging from 100% read-only to 100% write-only accesses.

Concurrent and Robust End-to-End Data Integrity Verification Scheme for Flash-Based Storage Devices

The amount of data generated by scientific applications on high-performance computing systems is growing at an ever-increasing pace. Most of the generated data are transferred to storage in remote systems for various purposes such as backup, replication, or analysis. To detect data corruption caused by network or storage failures during data transfer, the receiver system verifies data integrity by comparing the checksum of the data. However, the internal operation of the storage device is not sufficiently investigated in the existing end-to-end integrity verification techniques. In this paper, we propose a concurrent and reliable end-to-end data integrity verification scheme considering the internal operation of the storage devices for data transfer between high-performance computing systems with flash-based storage devices. To perform data integrity verification including data corruptions that occurred inside the storage devices, we control the order of I/O operations considering the internal operations of the storage devices. Also, to prove the effectiveness of the proposed scheme, we devise a prototype that injects faults on the specific layer of the storage stack and examines detection of faults. We parallelize checksum computation and overlap it with I/O operations to mitigate the overhead caused by I/O reordering. The experimental results show that the proposed scheme reduces the entire data transfer time by up to 62% compared with the existing schemes while ensuring robust data integrity. With the prototype implementation, our scheme detects failures on NAND flash memory inside storage devices that cannot be detected with the existing schemes.

Benchmarking for Observability: The Case of Diagnosing Storage Failures

Diagnosing storage system failures is challenging even for professionals. One recent example is the “When Solid State Drives Are Not That Solid” incident occurred at Algolia data center, where Samsung SSDs were mistakenly blamed for failures caused by a Linux kernel bug. With the system complexity keeps increasing, diagnosing failures will likely become more difficult.To better understand real-world failures and the potential limitations of state-of-the-art tools, we first conduct an empirical study on 277 user-reported storage failures in this paper. We characterize the issues along multiple dimensions (e.g., time to resolve, kernel components involved), which provides a quantitative measurement of the challenge in practice. Moreover, we analyze a set of the storage issues in depth and derive a benchmark suite called BugBenchk. The benchmark suite includes the necessary workloads and software environments to reproduce 9 storage failures, covers 4 different file systems and the block I/O layer of the storage stack, and enables realistic evaluation of diverse kernel-level tools for debugging.To demonstrate the usage, we apply BugBenchk to study two representative tools for debugging. We focus on measuring the observations that the tools enable developers to make (i.e., observability), and derive concrete metrics to measure the observability qualitatively and quantitatively. Our measurement demonstrates the different design tradeoffs in terms of debugging information and overhead. More importantly, we observe that both tools may behave abnormally when applied to diagnose a few tricky cases. Also, we find that neither tool can provide low-level information on how the persistent storage states are changed, which is essential for understanding storage failures. To address the limitation, we develop lightweight extensions to enable such functionality in both tools. We hope that BugBenchk and the enabled measurements will inspire follow-up research in benchmarking and tool support and help address the challenge of failure diagnosis in general.

Detailed black-box monitoring of distributed systems

Modern containerized distributed systems, such as big data storage and processing stacks or micro-service based applications, are inherently hard to monitor and optimize, as resource usage does not directly match hardware resources due to multiple virtualization layers. For instance, interapplication traffic is an important factor in as it directly indicates how components interact, it has not been possible to accurately monitor it in an application independent way and without severe overhead, thus putting it out of reach of cloud platforms. In this paper we present an efficient black-box monitoring approach for gathering detailed structural information of collaborating processes in a distributed system that can be queried for various purposes, as it includes both information about processes, containers, and hosts, as well as resource usage and amount of data exchanged. The key to achieving high detail and low overhead without custom application instrumentation is to use a kernel-aided event driven strategy. We validate a prototype implementation by applying it to multi-platform microservice deployments, evaluate its performance with micro-benchmarks, and demonstrate its usefulness for container placement in a distributed data storage and processing stack (i.e., Cassandra and Spark).

A survey of operating system support for persistent memory

Emerging persistent memory technologies, like PCM and 3D XPoint, offer numerous advantages, such as higher density, larger capacity, and better energy efficiency, compared with the DRAM. However, they also have some drawbacks, e.g., slower access speed, limited write endurance, and unbalanced read/write latency. Persistent memory technologies provide both great opportunities and challenges for operating systems. As a result, a large number of solutions have been proposed. With the increasing number and complexity of problems and approaches, we believe this is the right moment to investigate and analyze these works systematically. To this end, we perform a comprehensive and in-depth study on operating system support for persistent memory within three steps. First, we present an overview of how to build the operating system on persistent memory from three perspectives: system abstraction, crash consistency, and system reliability. Then, we classify the existing research works into three categories: storage stack, memory manager, and OS-bypassing library. For each category, we summarize the major research topics and discuss these topics deeply. Specifically, we present the challenges and opportunities in each topic, describe the contributions and limitations of proposed approaches, and compare these solutions in different dimensions. Finally, we also envision the future operating system based on this study.

Recycling of Critical Raw Materials from Hydrogen Chemical Storage Stacks (PEMWE), Membrane Electrode Assemblies (MEA) and Electrocatalysts

The increase in energy demands for the establishment of a modern digital era has resulted in the significant limitation of the energy sources. The depletion of energy reserves drew attention to alterative renewable energy sources that can satisfy the energy requirements in an environmentally friendly way. Hydrogen is an ideal chemical energy storage. Proton exchange membrane water electrolysis (PEMWE) is a promising technology as a green source of high-purity hydrogen. For that reason, PEMWE devices are already used in aerospace to produce oxygen for the crew on the Space Stations. The use of Critical Raw Materials (CRMs, especially Pt and Ir) and high cost materials in the PEMWE systems compromises their economic feasibility. It is necessary to implement End-of-Life (EoL) strategies that optimise the recovery of CRMs within feasible and environmentally friendly processes. In this paper an overview of the existing technologies for recycling of Membrane Electrode Assembles and electrocatalysts from PEMWE systems are revealed. MONOLITHOS has developed a novel optimized hydrometallurgical method for high PGMs recovery from spent automotive catalysts. This work aims to describe how this method can be applied to the EoL phase of PEMWE to improve techno-environmental and techno-economical performance of CRMs recovery.

Design of Phase‐Change Memory Using Apertureless Scanning Near‐Field Optical Microscopy in the Near‐Infrared Region

A novel optical secondary storage memory operated in the near‐infrared (NIR) region is developed through the integration of an apertureless scanning near‐field optical microscopy probe with the conventional Ge2Sb2Te5 storage stack. The probe composite and the optical coefficients of the storage media stack, particularly for the indium tin oxide capping layer, are optimized according to a previously developed electromagnetic wave model, and a newly established density functional theory model, respectively. The dielectric core and the metal coating medium of the probe are devised to be glass and barium, respectively, to provide high electric field in the NIR region, while the thickness of the ITO capping layer is chosen to be 3 nm here to provide high optical transmittance simultaneously with exemption of complex fabrication process. The recording operation of the aforementioned optimized device is subsequently simulated by a newly established thermo‐optics model, and its feasibility of achieving ≥Tbit in−2, 108 bits per second, and picojoule energy consumption per bit is theoretically demonstrated.

A Novel Global Key-Value Storage System Based on Kinetic Drives

NoSQL databases are flexible and efficient for many data intensive applications, and the key-value store is one of them. In recent years, a new Ethernet accessed disk drive called the “Kinetic Drive” was developed by Seagate. This new Kinetic Drive is specially designed for key-value stores. Users can directly access data with a Kinetic Drive via its IP address without going through a storage server/layer. With this new innovation, the storage stack and architectures of key-value store systems have been greatly changed. In this paper, we propose a novel global key-value store system based on Kinetic Drives. We explore data management issues including data access, key indexing, data backup, and recovery. We offer scalable solutions with small storage overhead. The performance evaluation shows that our location-aware design and backup approach can reduce the average distance traveled for data access requests.